Overview
Mercor partners with leading AI labs and enterprises to train frontier models using human expertise. As an AI Evaluation Rubric Auditor, you will audit the work of Project Reviewers by evaluating their grading rubrics and determining whether rubric failures are fair, checking for information availability, rubric accuracy, and structure compliance.
What You’ll Do
- Audit complete tasks including prompts and grading rubrics
- Determine if rubric failures are fair and justified
- Check whether information was available to agents
- Verify rubric accuracy against prompt requirements
- Identify valid answers incorrectly marked as wrong
- Validate rubric structure compliance with guidelines
- Provide feedback to reviewers on prompt and rubric issues
Requirements
- Substantial experience writing or reviewing rubrics for LLM evals (must list projects in application)
- Experience providing feedback to writers/reviewers
- Consistent availability
- Ability to read long agent transcripts and identify failure points
- Precise writing and verification of data against reference answers
Who Should Apply
This role suits someone with proven experience building or auditing evaluation rubrics for language models—you need a track record of actual projects to cite. It's ideal if you have a methodical mind, can read complex transcripts closely to pinpoint where reasoning broke down, and enjoy giving constructive feedback. You'll struggle here if you work intuitively (this role demands verifying data, not trusting summaries) or if you need a fixed schedule or significant human interaction.
Salary Insight
The $60–$85 range reflects the seniority this role demands: it's higher than the reviewer position ($40–$75) because you're auditing their work and need deeper rubric expertise. The wide band likely accounts for variation in project complexity and your experience level. No base commitment is stated, so clarify expected hours per week and whether payment scales with volume or stays hourly.
About Mercor
A marketplace placing specialists into paid remote contract work with AI companies. One profile and one AI interview, reused across every role you apply to.
Remote policy: Fully remote, worldwide, on hourly contractor agreements rather than employment. But payment eligibility is the thing to check before you invest any time, because it is decided by country and there is no way around it. Mercor pays through Stripe Connect in…
How hiring works at Mercor →Required Skills
Compensation
$60–$85 per hour
- Location
- Argentina, Bolivia, Brazil, Canada, Chile, Colombia, Ecuador, Guyana, Paraguay, Peru, United Kingdom, United States, Uruguay
- Engagement
- Contract
- Posted
- 5d ago
Opens Mercor’s listing on MyRemoteJobs — we don’t collect applications ourselves.
About Mercor
A marketplace placing specialists into paid remote contract work with AI companies. One profile and one AI interview, reused across every role you apply to.
Remote policy: Fully remote, worldwide, on hourly contractor agreements rather than employment. But payment eligibility is the thing to check before you invest any time, because it is decided by country and there is no way around it. Mercor pays through Stripe Connect in…
How hiring works at Mercor →Will your CV get past the filter?
Most applications are rejected by software before a person ever reads them. Scan your CV using our ATS Checker tool, see exactly what's blocking it, then fix it in the CV Builder and apply knowing it'll get through.
Scan my CVGet roles like this by email
Tell us what you do and we'll email you when a similar remote job is posted. No account needed.
See our jobs more often in your search
Add MyRemoteJobs as a Preferred Source on Google. Free, takes seconds, and you stay on this page.
Application Tip
Lead with 2–3 specific LLM eval projects you've authored or reviewed rubrics for—name the task type and what you learned about common rubric failure modes. Hiring managers will vet this claim because rubric auditing demands real expertise; a vague mention of "evaluation experience" won't survive scrutiny. Then show an example of feedback you've given that improved rubric clarity or fairness, because half this role is teaching reviewers how to grade better.
Sourced from MyRemoteJobs · verified against the original posting
Similar open positions
Music Expert (Music Teacher / Musician)
micro1
VerifiedVisual Content Evaluator
micro1
VerifiedAI Evaluation Rubric Reviewer
Mercor
Verified