myRemoteJobs
Quality Assurance

AI Evaluation Rubric Reviewer

Mercor

Verified listing
Contract
RemotePosted 5d ago$40–$75 per hour

Overview

Mercor partners with leading AI labs and enterprises to train frontier models. In this role, you'll review AI-drafted evaluation tasks and model attempts, assessing whether failures are fairly graded against their rubrics and whether the rubric criteria themselves are sound.

What You’ll Do

  • Review AI-drafted tasks (prompt and grading rubric) along with model attempts
  • Determine whether each failure is fair by checking if information was available, rubric accuracy, and whether valid answers were marked wrong
  • Fix criteria that aren't fair and flag tasks that cannot be saved
  • Write short notes explaining your assessment on each review call

Requirements

  • Substantial experience writing or reviewing rubrics for LLM evals—list projects in application
  • Ability to read long agent transcripts and identify where failures occurred
  • Precise writing and willingness to verify data instead of trusting reference answers

Who Should Apply

Apply if you have hands-on experience building or auditing LLM evaluation rubrics—not just taking them—and you enjoy the precision work of spotting where a rubric fails to match its prompt or where an AI answer should have passed. Avoid this if you prefer end-to-end product ownership or would find yourself second-guessing edge cases rather than committing to a decision.

Salary Insight

The range of $40–$75/hour is wide and reflects variable project complexity and reviewer experience. This is typical for contract evaluation work, where junior evaluators start lower and senior reviewers with established rubric portfolios command the top end. Clarify where you'd fit before accepting, and ask whether projects include flat rates or time-based billing.

Required Skills

LLM evaluationrubric writingAI gradingagent testing

Compensation

$40–$75 per hour

Location
Remote
Engagement
Contract
Posted
5d ago

Opens Mercor’s listing on MyRemoteJobs — we don’t collect applications ourselves.

About Mercor

A marketplace placing specialists into paid remote contract work with AI companies. One profile and one AI interview, reused across every role you apply to.

Remote policy: Fully remote, worldwide, on hourly contractor agreements rather than employment. But payment eligibility is the thing to check before you invest any time, because it is decided by country and there is no way around it. Mercor pays through Stripe Connect in…

How hiring works at Mercor →

Will your CV get past the filter?

Most applications are rejected by software before a person ever reads them. Scan your CV using our ATS Checker tool, see exactly what's blocking it, then fix it in the CV Builder and apply knowing it'll get through.

Scan my CV

Get roles like this by email

Tell us what you do and we'll email you when a similar remote job is posted. No account needed.

Free. We email you as soon as a role matches — applying early matters. Only genuine matches, and unsubscribe in one click.

See our jobs more often in your search

Add MyRemoteJobs as a Preferred Source on Google. Free, takes seconds, and you stay on this page.

Application Tip

Lead with 2–3 specific LLM eval or grading projects you've shipped: the domain, rubric size, and what you fixed or improved about the original criteria. Mercor will ask for these by name. Rather than listing rubric-writing skills in general, show one example of a criterion you rewrote because the original was ambiguous or unfair—this directly demonstrates the judgment they need. A transcript walkthrough in your cover letter of one real failure you caught will carry more weight than a summary of your approach.

Sourced from MyRemoteJobs · verified against the original posting