AI Quality Engineer

AI Quality Engineer (LLM Evals & Human Data) About Deccan AI Deccan AI is a GenAI data company that runs the post-training and production layer for the world's leading AI labs and enterprises — from expert feedback and RL environments to managed AI workforces and evaluation infrastructure. Backed by A91 Partners, SIG (Susquehanna International Group) and Prosus Ventures, and having grown 10x last year, Deccan works with a majority of the Magnificent 7 and the top frontier labs globally. Its global network of 1 million+ domain experts powers Helix, its hybrid eval suite, and EnterpriseOS, its scaled ops automation platform. Headquartered in Mountain View, CA. About the role Frontier models are now limited less by compute but more by high quality human data and evaluations behind them. This role sits exactly at that intersection. At Deccan AI, our annotators, raters and QCs produce evaluation and training data for some of the world's leading AI labs - multi-turn assistant evals, agentic tool-use tasks, A/B conversation evals, rubric-based rating and robotics video labeling. You will be the person who makes that data trustworthy at scale. We are looking for an analytical thinker with high attention to detail and an uncompromising approach when it comes to data quality - along with the drive to deliver the best outcome for our customers (clients). You will need to have a strong backbone, often requiring pushing back on defective data outcomes. What you'll own Work with project teams to design and build quality systems, using a combination of agents and deterministic code, right from project’s kickoff to final delivery Understand client guidelines, interpret nuanced requirements and design training programs to ensure effective talent deployment on projects. Analyse datasets and findings to build insight reports for client research teams. Turn quality findings into targeted feedback into example-driven training content on the LMS. What you must bring 2–3 years in the annotation industry - at a data vendor, an AI lab, or an in-house labelling team - working on evals, SFT, multimodal or agentic data. Hands-on quality systems experience - you have designed or run QC workflows, sampling plans, rubrics or audit programmes. Strong Python - you write scripts and small pipelines to process, score and analyse task data without using automations. SQL fluency - comfortable joining messy production tables and building dashboards (Metabase, Superset, Looker or similar). Working ML and LLM knowledge - how models are trained (SFT, RLHF, preference data), how evals work, and what inter-rater agreement, precision/recall and calibration mean in practice. Strong analytical skills - you can break down large, messy datasets, spot patterns and root causes, and turn them into clear, defensible insights. Ownership under pressure - you can run several projects at once on tight timelines and still hold the highest quality bar. Great if you also have Built agents or pipelines with LLM APIs, LangChain/LangGraph, or similar tooling. Prepared golden-sets, calibrated LLM judges and hillclimbed judge performance. Worked on agentic or tool-use evals, coding / SWE-bench-style tasks, or robotics and video labeling. Experience with statistics for audits: confidence intervals, stratified sampling, Cohen's kappa / Krippendorff's alpha. Why this role Frontier-lab exposure from day one - your work goes straight into how leading AI models are trained and evaluated. Build, don't maintain - many of our quality systems and auto-QC pipelines are being built now; you'll shape them, not inherit them. Real autonomy - you own projects, setting quality across accounts. The skill set the market is short of - ML + code + evals + human-data judgement is one of the hardest profiles to hire in AI; this role builds all four at once.