Business Domain Expert — AI Model Evaluation Pilot

Job Type: Independent Contractor
Location: Remote
Schedule: Flexible, short-term project
Compensation: $60–$80/hour (approximately $124,800–$166,400 annualized, based on 40 hours/week)

About the Opportunity

We are seeking experienced professionals in Business Operations, Marketing, Human Resources, and Accounting to participate in a structured AI model evaluation pilot.

You will complete one realistic professional task within your area of expertise and evaluate five AI-generated attempts at the same task. Your professional judgment will help assess how effectively advanced AI models perform real-world professional work.

This is a one-time evaluation study, rather than an ongoing production role.

What You'll Do

• Sign a confidentiality agreement before receiving study materials.

• Complete one realistic, domain-specific task using the provided source materials.

• Review five AI-generated outputs for the same task.

• Rank the outputs from 1–5 with no ties.

• Score each output from 0–100 based on its quality and effectiveness.

• Write clear, evidence-based rationales referencing specific parts of each output.

• Provide structured feedback on the task and study design.

Areas of Expertise

We are looking for professionals with experience in one of these domains:

• Administration / Business Operations

• Marketing

• Human Resources

• Accounting

Time Commitment

The study requires up to 13 hours total:

• 3–10 hours to complete the assigned task.

• Approximately 2 hours to review, rank, and score the five outputs.

• Approximately 1 hour to provide feedback.

Study Requirements

• No rubric or predefined answer will be provided. Your professional judgment is the basis of the evaluation.

• The use of LLMs, AI assistants, or other AI tools is prohibited throughout the study, including while completing the task, evaluating outputs, assigning scores, or writing rationales.

• Evaluations must be based on your own professional knowledge and experience.

Requirements

• 5+ years of hands-on professional experience in at least one of the four target domains.

• Current or recent professional experience in your area of expertise.

• Strong professional judgment and ability to assess the quality of real-world business work.

• Excellent written communication skills.

• Ability to provide specific, clear, and evidence-based reasoning.

• Reliable availability to complete the study within a short turnaround window.

Eligibility Restriction

You are not eligible for this pilot if you have previously worked on Project Alchemy in any capacity, including as a task author, reviewer, domain expert, or anyone who has had access to its source-world data. Prior exposure to the material would compromise the integrity of the study.

Contract Details

• Independent contractor engagement.

• Fully remote with flexible scheduling.

• One-time, short-term evaluation project.

• Projects may be extended, shortened, or concluded early depending on requirements and performance.

• Payments are made weekly based on services rendered.

• H-1B and STEM OPT candidates are not currently supported.

• Work will not require access to confidential or proprietary information belonging to another employer, client, or institution.

Why Participate

• Apply your professional expertise to the evaluation of frontier AI models.

• Help measure how effectively AI performs realistic professional tasks.

• Contribute expert human judgment to the development of next-generation AI systems.

• Participate in emerging AI evaluation work within your area of expertise.

Ideal Candidate: An experienced professional who can independently evaluate business work, identify meaningful strengths and weaknesses, and clearly explain their judgment using specific evidence.