AI Operations Specialist
Position Summary
We are seeking a thoughtful, analytical, and technically curious AI Operations Specialist to help scale and operationalize Agent AI across Customer Care. This is not an engineering role and not a traditional operations role. AI Operations Specialists sit at the intersection of AI systems, frontline support expertise, and operational rigor - ensuring our AI agents deliver accurate, high-quality resolutions at scale.
In this role, you will be responsible for defining how Agent AI behaves in real-world scenarios by creating and maintaining AI Operating Procedures (AOPs/SOPs), evaluating AI performance through structured QA, and driving iteration loops with Product, Engineering, and external vendors. You will translate human support judgment into system-level instructions and validation frameworks, directly influencing containment, accuracy, and customer experience.
You will operate as part of a cross-functional task force with Product and Engineering, with daily standups and tight execution loops. This role reports into Care Innovation leadership, which holds final authority on go/no-go decisions for AI releases.
The US base salary range for this full-time position is $85,000 - $100,000 + equity + benefits. Our salary ranges are determined by role, level, and location. This is a hybrid position in Mountain View, CA that requires in-office work 2 days a week.
What You’ll Do
AI Operating Model & SOP Development (35%)
Design and maintain detailed AI Operating Procedures (AOPs/SOPs) that define how Agent AI should behave across supported use cases.
Translate human-agent judgment into structured system logic, mapping actions, edge cases, and expected outcomes.
Inspect and reference APIs, system actions, and integration flows directly within documentation.
Continuously update SOPs as product functionality and use cases evolve.
AI Quality Assurance & Performance Improvement (35%)
Define and execute QA frameworks to evaluate containment, accuracy, and resolution quality across vendor bots, internal bots, and LLM-based systems.
Identify failure modes, escalation drivers, and judgment gaps.
Partner with Product, Engineering, and vendors to iterate on performance and improve customer experience.
Enable and calibrate human QA teams evaluating Agent AI at scale.
AI Analytics & Operational Insights (20%)
Build and maintain dashboards tracking key AI metrics (e.g., containment rate, accuracy, escalation rate).
Analyze trends and surface actionable insights to guide prioritization and iteration.
Use data to inform readiness decisions and highlight performance risks.
Cross-Functional Execution (10%)
Participate in daily standups with Product and Engineering to align on priorities and iteration loops.
Provide structured operational feedback to vendors and internal stakeholders.
Support release readiness discussions and post-launch performance reviews.
What We’re Looking For
A college degree or equivalent qualification.
Three to five years of relevant experience in operations, QA, product operations, support tooling, or adjacent roles.
A curious, proactive, and entrepreneurial mindset with strong problem-solving skills grounded in critical thinking and common sense.
Demonstrated ability to reason through complex systems and translate human judgment into structured processes.
Comfort working with APIs, system actions, and technical documentation (without being an engineer).
Strong analytical skills and experience working with performance metrics and dashboards.
Willingness to roll up sleeves and actively participate in execution-heavy work.
Excellent interpersonal skills and empathy for customers.
Strong organizational skills and comfort leading workstreams involving Product, Engineering, QA, and external partners.
Willingness to travel internationally: This role requires travel up to once per quarter to BPO sites supporting AI QA operations.
#LI-Hybrid