Home
Jobs
Swoon
AI Domain Evaluation Quality Specialist

AI Domain Evaluation Quality Specialist

Hybrid

San francisco bay, United states

$ 60 /hour

Junior

Freelance

11-03-2026

Share this job:

Score my CV

Skills

Communication Python SQL Monitoring Research Attention to detail

Job Specifications

Contract Opportunity | AI Evaluation Quality Control Specialist

San Francisco, CA (Hybrid) | $50hr-$60hrr | Contract

We're hiring a domain expert to support a high-priority AI evaluation campaign for a leading frontier AI lab in the generative AI industry.

This is a hands-on quality control and process monitoring role -- not a research or engineering position. You'll be the expert eye ensuring that AI evaluation tasks, sample outputs, and scoring rubrics reflect how real, high-quality work is actually done in your field.

What You'll Do

Review incoming eval tasks, deliverables, and rubrics for domain realism, feasibility, and clarity
Identify failure modes: vague requests, unrealistic scope, misaligned success criteria, missing context
Provide clear, actionable feedback to contributors and track issues through resolution
Monitor campaign health -- submission volume, rework rates, bottlenecks, recurring issues
Surface systemic problems and propose concrete fixes (checklist updates, template changes, examples)
Support contributors in understanding what "good" looks like in your domain
Use spreadsheets or light Python/SQL to flag anomalies and support workflow improvements
Draft QC documentation, calibration examples, and audit logs

What We're Looking For

3+ years of professional experience in a specialized domain (e.g., law, medicine, finance, engineering, cybersecurity, or similar)
Strong judgment about what realistic, high-quality professional work looks like
Excellent attention to detail and ability to apply standards consistently at volume
Clear, direct written communication -- you give feedback that actually improves quality
Comfort working in fast-moving, ambiguous environments
Familiarity with spreadsheets; basic SQL or Python is a plus

Details

Location: San Francisco, CA -- Hybrid (some in-office time required)
Rate: $50-$60/hr depending on domain and experience

About the Company

In 2010, Swoon launched an agile, client-focused team that is not only savvy in our core industries but elbow-deep, every day, getting to know the strongest talent in the technology and professional fields. Today, Swoon is one of the fastest-growing talent and consulting firms in the U.S., Canada, Australia, and Mexico with an expanding client roster of Fortune 1000 companies, Best-of Awards, and legions of talent in our network. We are a WBENC-certified company founded by industry leaders with 100+ years of combined exp... Know more