AI Agentic Workflow Reviewer

Scope of the Role:

We're building a dedicated reviewer team to evaluate complex, real-world agentic AI workflows, assessing whether AI agents complete tasks safely, respect user intent and consent, and hold up under close scrutiny. This isn't routine content moderation: reviewers work through ambiguous, multi-step scenarios where judgment calls matter as much as checklists. You'll operate inside isolated test environments, apply structured rubrics, and help calibrate what "safe and policy adherent agent behavior" actually looks like.

What You’ll Own:

Review multi-step agent task trajectories against rubric-based criteria covering task success, safety, and policy adherence

Evaluate whether an agent respected user agency and informed consent — not just whether it followed literal instructions

Apply privacy guardrails when reviewing tasks involving sensitive or personal data

Work inside isolated test environments, including verifying environment resets between test runs to prevent cross-contamination

Flag and escalate safety-relevant or ambiguous findings through defined escalation paths

You’ll Thrive in This Role If You Have:

Bachelor's degree or equivalent practical experience

2–5+ years in quality review, QA/QC, trust & safety, content moderation, data annotation, or a similarly judgment-intensive review role

Strong written communication – you can justify a scoring decision clearly enough for someone else to audit it

Comfort with ambiguity and structured decision-making under a rubric, rather than needing a fixed rulebook for every case

Baseline understanding of AI/ML concepts and how AI agents complete tasks (can be learned through onboarding, but some familiarity helps)

The expected hourly salary range for this position is $50-$55p/hour, based on experience, skills, and qualifications.