AI Evaluation Infrastructure Engineer

Location: United States – Remote Clearance: Ability to obtain and maintain a Public Trust LTS is seeking a Senior AI Platform and Harness Engineer to develop and maintain the infrastructure, tooling, and evaluation frameworks that power enterprise AI solutions. This role is responsible for building the AI platform and reusable "AI harnesses" that enable Large Language Models (LLMs), AI agents, Retrieval-Augmented Generation (RAG), and Generative AI applications to be securely developed, tested, evaluated, monitored, and deployed at scale. The ideal candidate has experience with AI platforms, LLMOps, software engineering, cloud-native technologies, and backend systems, along with a passion for building reliable, observable, and production-ready AI solutions. You will work closely with AI architects, software engineers, data scientists, and product teams to ensure AI solutions are scalable, secure, cost-effective, and continuously improving. What You'll Do: Design, build, and maintain enterprise AI platform capabilities supporting Large Language Models (LLMs), AI agents, RAG, and Generative AI applications. Lead data preparation, training and fine-tuning of LLMs Develop reusable AI harnesses to automate testing, prompt evaluation, model benchmarking, regression testing, and quality assurance. Build AI evaluation frameworks to measure model accuracy, retrieval quality, hallucination detection, latency, throughput, cost, and overall application performance. Implement observability and monitoring solutions for AI applications, including telemetry, tracing, logging, dashboards, and operational metrics. Build and maintain LLMOps pipelines supporting model deployment, versioning, evaluation, experimentation, rollback, and continuous improvement. Design automated workflows for prompt testing, retrieval evaluation, AI system validation, and performance benchmarking. Develop internal tools for prompt management, model experimentation, AI performance optimization, and developer

About the Team pAGI Infra team builds and operates the systems that make large-scale model training and evaluation reliable, efficient, and easy to run. Our work spans distributed training infrastructure, inference and grading platforms, compute scheduling, and research tooling. We partner closely with researchers and engineering teams to turn new research needs into dependable infrastructure, improve GPU efficiency, and shorten the path from an experiment to a validated model. About the Role We’re looking for an AI Systems Engineer to help scale the infrastructure behind our training and evaluation workflows. You’ll own projects from identifying bottlenecks and designing solutions through deployment and operation. The work combines distributed systems engineering, performance optimization, and close collaboration with researchers. You might build a shared grading service, improve resource allocation across workloads, or bring a new training stack into production — directly improving how quickly and reliably research moves forward. In this role, you will: Build and operate infrastructure for large-scale training and evaluation, improving reliability, throughput, and resource efficiency. Develop shared inference and grading platforms with automated capacity management, health monitoring, and visibility into performance. Improve compute scheduling and resource allocation to reduce idle GPU time and help workloads recover quickly from failures. Diagnose bottlenecks across training, inference, and orchestration, and work across teams to improve end-to-end performance. Build self-service tools, automated validation, and observability that help researchers launch experiments, diagnose issues, and compare results with less manual intervention. You might thrive in this role if you: Are excited about the potential of personal AGI and want to build the infrastructure that enables it. Have strong software engineering fundamentals and experience building or operating large-scal

Who We Are At OKX, we believe that the future will be reshaped by crypto, and ultimately contribute to every individual's freedom. OKX is a leading crypto exchange, and the developer of OKX Wallet, giving millions access to crypto trading and decentralized crypto applications (dApps). OKX is also a trusted brand by hundreds of large institutions seeking access to crypto markets. We are safe and reliable, backed by our Proof of Reserves. Across our multiple offices globally, we are united by our core principles: We Before Me , Do the Right Thing , and Get Things Done . These shared values drive our culture, shape our processes, and foster a friendly, rewarding, and diverse environment for every OK-er. OKX is part of OKG, a group that brings the value of Blockchain to users around the world, through our leading products OKX, OKX Wallet, OKLink and more. About the Opportunity OKX’s Internal Audit function has an early but working AI-native capability: a multi-agent platform (Hive Mind), agentic workflows, data pipelines, and AI-enabled tools that let a very small team punch far above its weight. Your job is not to rebuild it at today’s maturity. Your job is to take it to a regulator-grade production standard and well beyond , and to drive AI Enablement across the department. You own the foundation, the agent runtime, and the harness: a resilient cloud platform, the agentic runtime and evaluation harnesses that make agents trustworthy in a regulated setting, and the governed data and AI infrastructure everything else depends on. We hire on demonstrated building , not on claims or credentials. We expect you to be more capable than the hiring manager in your domain: you will co-own and challenge the tooling strategy, not just implement it. This is a two-person engineering team: you deploy, debug, and hotfix your counterpart’s stack when needed. Environment Cloud on AWS or GCP . Claude as the starting point in a deliberately multi-model architecture (Anthropic

SpaceX was founded under the belief that a future where humanity is out exploring the stars is fundamentally more exciting than one where we are not. Today, SpaceX is actively developing the technologies to make this possible, with the ultimate goal of making life multiplanetary and expanding consciousness to the stars. SpaceXSI is building Super Intelligence to help humanity understand the universe. Our teams are building the full Super Intelligence stack: from our Colossus supercomputers, to Grok models, to our suite of applications, including Grok Bot, Grok Build, Grok Imagine, Grok Voice, and X. We're creating a powerful flywheel where more compute trains better models, better models power more useful products, and what we learn from people using those products shapes what we build next. About the Role As a Software Engineer on the Agent Quality team at SpaceXAI, you’ll build the measurement, evaluation, and feedback-loop infrastructure that makes the Cursor core agent reliably better over time. This role sits at the intersection of product, data, and engineering: you’ll instrument what matters, help define how we judge quality, build pipelines and tooling to analyze agent behavior at scale, and partner closely with research, product, and infrastructure teams to turn insights into improvements. Your impact will compound across every Cursor product built on the shared harness—and across high-stakes decisions around model choice, quality, and cost. What you’ll work on Designing and building best-in-class AI evaluation system: curated datasets, offline replay, scorers / judges, regression alerts, and dashboards. Designing feedback loops from real usage: collecting, cleaning, and interpreting user signals to inform model and harness changes. Developing analysis tooling and workflows for debugging agent behavior: deep dives on failure modes, clustering themes, and surfacing actionable insights. Improving reliability and guardrails by making quality measurable and ope

About the Team Our Robotics team is focused on unlocking general-purpose robotics and pushing towards AGI-level intelligence in dynamic, real-world settings. Working across the entire model stack, we integrate cutting-edge hardware and software to explore a broad range of robotic form factors. We strive to seamlessly blend high-level AI capabilities with the constraints of physical systems to improve peoples’ lives. About the Role We are hiring a Sim Infrastructure Engineer to turn simulation systems into reliable, automated, production-quality pipelines that power model training, evaluation, and hardware-in-the-loop validation. This role owns the automation, orchestration, and tool integration that apply simulation to concrete robotics tasks: building CI/CD for SIL/HIL, presubmit checks, automatic model evaluation, metric computation and reporting, and the runtime infrastructure to run simulations at scale. You will collaborate closely with Sim Realism, Sim Environments, Research, and Ops to make simulation an integrated, reproducible, and measurable part of our ML and robotics workflows. This role is based in San Francisco, CA, and requires in-person 4 days a week. In this role, you will: Build and maintain presubmit checks, continuous integration and deployment pipelines for simulation code, environments, and tasks so simulation artifacts are testable, versioned, and reproducible. Implement end-to-end automation to run model evaluation in sim (SIL) and orchestrate HIL runs; compute realism and task metrics, generate dashboards and alerts, and ensure evaluation is repeatable and auditable. Create robust APIs and connectors so research, training, and data-collection systems can schedule, seed, and evaluate batches of simulations; support RL rollouts, imitation-data collection, and presubmit model checks. Build scheduling, batching and orchestration for running very large numbers of concurrent rollouts (target tens of thousands of rollouts / large RL workloads), sol

About the Team DoorDash's MDX team sits within Machine Learning Platform and builds the shared foundation that ML engineers across DoorDash, Wolt, and Deliveroo rely on to build and run ML-powered products. We work across the full ML lifecycle: feature engineering, training, evaluation, deployment, serving, and monitoring. Our mission is to increase the velocity and reliability of ML-driven business impact across the company. A central pillar of that work is agentic infrastructure. We are building systems that let agents accelerate and operate the ML workflow itself, including auto research, where agents explore and iterate on modeling ideas, and self-healing systems that detect, diagnose, and remediate failures in ML pipelines and production models with minimal human intervention. About the Role You will join a small, high-leverage team building the agentic layer of DoorDash's ML infrastructure. You'll design and build the systems that let agents accelerate how ML engineers across DoorDash, Wolt, and Deliveroo develop, ship, and operate models. That includes auto research systems that explore and iterate on modeling ideas, and agentic developer support that helps ML engineers move through training, evaluation, deployment, and debugging faster. You'll own well-scoped components end to end, from design through production, working across agent orchestration, tooling and integrations with our ML platform, evaluation of agent behavior, and observability. This role is ideal for an engineer with a strong backend foundation who is excited to apply AI agents to real engineering workflows, learns quickly, and wants to grow into technical leadership in a fast-moving area where model capabilities, tooling, and best practices change month to month. You're excited about this opportunity because you will… Build agentic systems that increase the velocity of ML development across DoorDash, Wolt, and Deliveroo, helping ML engineers get from idea to production model faster. Design an