When AI gets the details wrong in your field, real people pay for it. The people who know the work best have to be the ones training these models. You’re not being replaced by AI. You’re making sure it’s used accurately and safely.
How You'll Contribute
- Design challenging, realistic software engineering tasks based on professional experience, including scenarios, specifications, codebases, tests, logs, API contracts, and design notes.
- Run tasks through AI coding agents and evaluate their deliverables, including code diffs, tests, designs, and pull request reviews.
- Compare multiple agent outputs against identical specifications and codebases, determine which is stronger, and document shortcomings.
- Create tests, rubrics, and success criteria that define correct behavior, edge-case handling, test coverage, and adherence to conventions.
- Write clear evidence-based evaluations identifying failures such as brittle changes, missed edge cases, misleading tests, fabricated files, trace thrashing, and off-spec interpretations.
- Contribute expertise across your primary stack and adjacent technologies, and review or refine tasks created by other engineers.
- Propose and build task scenarios from professional engineering experience, including expected model failure modes.
What Makes You a Great Fit
- 2+ years of professional software engineering experience preferred, including shipping and maintaining production code as part of a team.
- In-progress Bachelor's degree or higher.
- Expertise in at least one common programming stack, such as TypeScript, JavaScript, Python, Java, Go, Rust, C++, C#, Ruby, PHP, Swift, or Kotlin.
- Hands-on individual-contributor engineering experience; candidates should currently or recently have performed the work themselves rather than working solely in management.
- Ability to use a terminal, Git, tests, and debuggers, and to quickly understand unfamiliar codebases.
- Strong engineering judgment and the ability to explain in writing what good code and genuine model failures look like.
- Full professional or native-level written and spoken English.
- General familiarity with AI and LLM tools, such as Claude or ChatGPT, and the ability to distinguish well-reasoned answers from plausible but incorrect ones.
- Comfort with cloud file tools, browser profiles, installing desktop applications, and everyday file handling.
- No prior AI or machine learning experience is required.
- Docker and CI experience are a plus.
- A computer science degree is a plus but not required.
Perks & Benefits
- Pay range of $40 to $125 per hour in USD.
- Fully remote contract role with flexible scheduling.
- Minimum commitment of 10 hours per week, with no weekly maximum and full-time hours available.
- Multi-day task timers that allow work to be spread across several days.
- Access to Claude and ChatGPT through the project; no personal subscription is required.
- Regular payment via PayPal.
- Opportunity to help improve frontier AI systems using real-world software engineering expertise.