Test Safety 201 in United States

About the Team We’re hiring a Developer Productivity engineer to support OpenAI’s Inference Runtime teams. These teams own the systems responsible for serving models reliably, efficiently, and safely across Codex, ChatGPT, API, and internal research workloads. We’re hiring a Developer Productivity Engineer to help scale the engineering systems, safeguards, and developer workflows that enable our teams to move quickly without compromising reliability or performance. This role sits at the intersection of developer experience, CI/CD infrastructure, release engineering, production readiness, and inference systems reliability. You’ll work on the tooling and operational foundations that support model launches, inference optimizations, cloud provider integrations, and large-scale deployments across a rapidly evolving inference stack. About the Role We’re looking for an autonomous, high-ownership engineer who cares deeply about making other engineers faster, safer, and more confident. A major focus of this role will be improving the tooling and infrastructure around deploy gates for inference engine images. These systems help ensure that every image released to production and research is correct, numerically sound, free of regressions, and performant across key metrics like time-to-first-token (TTFT) and time-between-tokens (TBT). You’ll help harden the systems that catch issues before they reach production, reduce noise from flaky or infrastructure-related test failures, and improve automation around triage, ownership, debugging, and escalation when failures occur. You’ll also work on improving observability, rollout safety, release automation, and developer self-service tooling across a rapidly evolving inference stack. This is not generic internal tools work. The systems you build directly impact OpenAI’s ability to support new model launches, safely ship inference optimizations to the world, onboard new infrastructure providers, and operate one of the largest and most p

About the Team The Personal AGI team is responsible for training and improving pre-trained models to be deployed into ChatGPT, the API, and potential future products. The team partners closely with research and product teams across the company, and conducts research as a final step to prepare for real world deployment to millions of users, ensuring that our models are safe, efficient, and reliable. About the Role As a Research Engineer / Scientist, you will research and develop improvements to our models. Our team works in research areas combining reinforcement learning and products. We're looking for individuals with strong ML engineering skills and research experience, especially with novel and highly capable models. An ideal candidate is passionate about product-driven research. This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees. In this role, you will: Own and pursue a research agenda to improve model capability and performance. Collaborate closely with the other research and product teams, allowing customers to optimize their own models. Build robust evaluations for tracking modeling improvements. Design, implement, test, and debug code across our research stack. You might thrive in this role if you: Have a deep understanding of machine learning and machine learning applications. Have a working knowledge of relevant models, and building evaluations for model capability improvement. Are comfortable diving into a large ML codebase to debug. Thrive in a dynamic and technically complex environment. About OpenAI OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. We push the boundaries of the capabilities of AI systems and seek to safely deploy them to the world through our products. AI is an extremely powerful tool that must be created with safety and human needs at its core, and

About the Team The RL and Reasoning team drives the core reasoning paradigm and has created groundbreaking innovations such as o1 and o3. They focus on pushing the boundaries of reinforcement learning research, building next-generation generative models, and deploying them at scale. About the Role As a Research Engineer/Research Scientist at OpenAI, you will advance the frontier of AI alignment and capabilities through cutting-edge RL methods. Your work will sit at the heart of training intelligent, aligned, and general-purpose agents, including the systems that power various models. We’re looking for people who have a background in reinforcement learning research, are able to iterate quickly, and are proficient at coding. This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees. You might thrive in this role if: You love being on the cutting edge of RL and language model research. You’re a self-starter who takes initiative and ownership of ideas, driving them to completion. You value principled approaches, simple experiments in tightly-controlled settings, and reaching trustworthy conclusions which stand the test of time. You thrive in a fast-paced, dynamic, and technically complex environment where rapid iteration is key. You’re comfortable diving into a large ML codebase to debug and improve it. You have a deep understanding of machine learning and machine learning applications. About OpenAI OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. We push the boundaries of the capabilities of AI systems and seek to safely deploy them to the world through our products. AI is an extremely powerful tool that must be created with safety and human needs at its core, and to achieve our mission, we must encompass and value the many different perspectives, voices, and experiences that form the ful

About the Team The Future of Computing Research team is an applied research team within the Consumer Devices group focused on developing new methods, models, and evaluation frameworks that support our vision for the future of computing. We work at the frontier of multimodal AI, helping turn emerging model capabilities into product experiences that are useful, delightful, and worthy of long-term trust. Our work explores a new class of AI systems that can learn over time, adapt to individuals, and support people in the flow of daily life. This includes long-term memory, user modeling, and personalization systems that are aligned not just with immediate satisfaction, but with a person’s broader goals, values, and well-being. We work closely across research, engineering, design, product, and safety to define what it means to build AI systems that know you over time, act at the right moment, and help in ways that are context-aware, respectful, and demonstrably beneficial. About the Role We are looking for a Research Engineer / Scientist to join the Future of Computing Research team to work on RLHF and post-training for personalized, multimodal AI systems. This role will focus on building the learning and evaluation foundations that help models become more context-aware, adaptive, and useful over time. You will work on problems such as reward modeling, preference learning, long-horizon evaluation, and policy improvement for systems that must make high-quality behavioral decisions in realistic user settings. The work is deeply product-grounded: success is not just higher benchmark performance, but better model behavior in real-world use. The ideal candidate is excited about pushing beyond one-turn assistant behavior toward systems that improve through feedback, learn from richer signals, and are trained against meaningful notions of user value. Internally, that maps closely to the need for careful reward design, feedback loops, and evaluation frameworks that test whether i

The Engineering Lead Analyst – Test Automation Platform Engineering is a senior-level technical leadership role responsible for driving the architecture, implementation, management, operational support, and continuous enhancement of enterprise test management and automation platforms. In this role, you will lead efforts to modernize test automation capabilities across the global technology ecosystem. You will architect end-to-end integration workflows, embed automated quality gates into enterprise CI/CD pipelines, and administer as well as operationally support both vendor and internally developed enterprise platforms (e.g., Core Performance Engineering / Performance Center, ALM-Quality Center, Zephyr Enterprise, CSDP / Octane). Additionally, you will play a critical role in production operations—delivering tier-3 platform support to rapidly and safely troubleshoot, triage, and remediate performance and availability issues in complex, distributed production environments. The ideal candidate blends deep hands-on expertise in software testing frameworks, modern DevOps pipelines, containerized infrastructure, message-driven integration, and robust operational resilience practices with strong governance, compliance, and stakeholder leadership skills. Key Responsibilities 1. Platform Engineering & Operational Support Install, configure, upgrade, administer, and support enterprise test management and performance engineering toolsets (e.g., Core Performance Engineering / Performance Center, ALM-Quality Center, Zephyr Enterprise, Core Software Development Platform [CSDP] / Octane). Provide end-to-end operational support for

Flight Test Engineering Manager Company: The Boeing Company We are seeking a Flight Test Engineering Manager to join the Boeing Test & Evaluation (BT&E) team supporting the Mobility Surveillance and Bombers (MS&B) portfolio in Oklahoma City, OK . In this role, you will lead the team responsible for planning, integrating, and executing flight test and system test activities for Boeing’s Airborne Warning and Control System (AWACS) programs. This role combines technical leadership, program execution, and people development, requiring a leader who can build high-performing teams, foster collaboration across engineering disciplines, and drive disciplined test execution in a dynamic environment. You will partner closely with internal and external stakeholders to deliver safe, efficient, and high-quality test outcomes while developing the next generation of flight test engineers. Why Join Us: Lead a high-impact Flight Test Engineering team supporting AWACS programs, influencing program test strategy and execution on mission-critical platforms Join a close-knit Flight Test community where teamwork, trust, and collaboration are essential to safely executing complex test programs Partner with cross-functional engineering, operations, program management, and customer teams to solve complex technical challenges and deliver safe, efficient, and high-quality test outcomes Position Responsibilities: Develop and coach a diverse team of engineers and mission systems operators by providing technical guidance, performance management, mentoring, and career development Lead the planning integration, and execution of lab, ground, and flight tests while ensur

About the Team Codex is OpenAI’s first-party developer product focused on agentic software engineering. We’re building tools that help engineers design, write, test, and ship code faster—safely and at scale. We partner tightly with research and product to translate model advances into tangible developer productivity. About the Role As a Data Scientist on Codex, you will measure and accelerate product-market fit for AI developer tools. You’ll define what “developer productivity” means for our product, run experiments on new coding models and UX, and pinpoint where the model helps or hurts across languages and tasks. Your insights will directly shape how an entire industry builds software. This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees. In this role, you will Embed with the Codex product team to discover opportunities that improve developer outcomes and growth Design and interpret A/B tests and staged rollouts of new coding models and product features Define and operationalize metrics such as suggestion acceptance, edit distance, compile/test pass rates, task completion, latency, and session productivity Build dashboards and analyses that help the team self-serve answers to product questions (by language, framework, repo size, task type) Diagnose failure modes and partner with Research on targeted improvements (model quality signals, user feedback, evals) You might thrive in this role if you have 5+ years in a quantitative role at a developer-facing or high-growth product Fluency in SQL and Python; comfort with experiment design and causal inference Experience defining product metrics tied to user value Ability to communicate clearly with PM, Eng, and Design—and to influence product direction You could be an especially great fit if you have Strong programming background; ability to prototype, run simulations, and reason about code quality Familiarity with IDE/extensi

What we’re doing isn’t easy, but nothing worth doing ever is. Diligent builds helpful robots that work safely and autonomously in real world environments. We move quickly, solve messy problems, and care deeply about reliability at scale. We’re hiring a Manufacturing Reliability Engineer to own production test for our robots at our contract manufacturer: you’ll design and run robust end-to-end test protocols, provision fleets of robots for production, and own the KPIs that define production quality. This role is based in Austin, TX. However, the position will require 50% travel to the Milwaukee, WI area and requires close collaboration across software, hardware, operations, and product engineering teams. Key Responsibilities End-to-end test process ownership. Create, validate, and maintain production test protocols and gating criteria from incoming inspection through final test and shipment. Provisioning of bots. Design and operate provisioning flows (imaging, firmware deployment, configuration, validation) and the tooling/fixtures needed to provision and handoff robots for production. KPIs and continuous improvement. Own key production metrics — First Pass Yield (FPY), cycle time, and test coverage — and drive continuous improvements to meet throughput and quality targets. Test automation & infrastructure. Architect, implement, and maintain automated test frameworks, harnesses, and test rigs used at the CM site. Ensure tests are stable, fast, and provide actionable failure data. Cross-functional escalation & RCA. Lead root-cause analysis for field and production failures; coordinate corrective actions with design, firmware, and CM engineering to close quality loops. On-site production leadership. Be the onsite technical authority at the contract manufacturer: train operators, debug failures on the line, and continuously refine processes with CM partners. What Success Looks Like Improved FPY and reduced rework rates across production builds. Reduced per

Aerodynamics Engineer (Aero-Stability & Control Analyst) Company: The Boeing Company Boeing Defense, Space & Security (BDS) is hiring an Experienced or Senior Aerodynamics Engineer to perform propulsion system qualification support and analysis in the St. Louis, MO region ! Position Responsibilities: Performs aerodynamic/aerothermodynamics design and analysis of flight vehicles to ensure that the vehicle meets requirements Key contributor to developing new and improving existing technologies, tools and processes to enhance technology readiness, improve vehicle performance and reduce cycle time and cost Conducts, coordinates and/or develops test requirements to meet test objectives using established processes Assesses the impact of proposed regulations to determine the effect on cost, performance, producibility, maintainability and safety Supports certification and qualification processes of the aerodynamic/ aero thermal characteristics of flight vehicles Assists in training others Works under general supervision Basic Qualifications (Required Skills and Experience): Bachelor of Science degree in Engineering, Engineering Technology (including Manufacturing Technology), Computer Science, Data Science, Mathematics, Physics, Chemistry or non-US equivalent qualifications directly related to the work statement 5+ years work-related experience with a Bachelors degree OR an equivalent combination of work-related experience and technical education. Preferred Qualifications (Desired Skills and Experience): (Level 4) 9+ years work-related experience w

About Sentry Software runs the world and the pace is faster than ever. Sentry helps developers fix errors and performance issues before users notice, so teams can spend less time firefighting and more time building. Trusted by 200,000+ organizations, Sentry is today’s application monitoring standard and our team is building its AI-native future. About the role As a Senior Software Engineer on Sentry’s AI/ML team, you’ll be responsible for building the evaluation infrastructure that measures the accuracy, reliability, and real-world performance of our AI systems. This role is critical to ensuring that our debugging agents and AI-powered features behave correctly, safely, and predictably as they scale. You’ll design datasets, benchmarks, and test harnesses that turn ambiguous AI behavior into measurable signals, helping the team ship AI with confidence. In this role you will Design and build robust evaluation frameworks to measure accuracy, reliability, regressions, and edge cases in AI systems Create and curate high-quality datasets, golden test cases, and benchmarks grounded in real production data Build automated test harnesses and metrics pipelines to continuously evaluate models, prompts, and agentic workflows Partner closely with applied AI engineers and product leaders to define what “good” looks like and translate it into measurable criteria Own the evaluation lifecycle for major AI initiatives, from early experimentation through production monitoring You’ll love this job if you Care deeply about correctness, rigor, and measurement in AI systems Enjoy turning fuzzy product goals and model behavior into concrete tests and metrics Like building foundational infrastructure that unlocks faster iteration and higher confidence for the entire AI team Thrive in cross-functional environments and enjoy influencing model design through better evaluation Qualifications Minimum 5+ years of professional experience with a Bachelor’s degree in computer science, machine learni

About the Team OpenAI’s Industrial Compute organization is building the infrastructure required to support the next generation of frontier AI systems. Through a combination of strategic partnerships and self-built data center campuses, we are scaling the power, cooling, electrical, mechanical, and controls infrastructure needed to deliver compute at unprecedented scale. The Commissioning organization is responsible for ensuring this infrastructure is safely tested, validated, integrated, and transitioned into reliable operations. For our self-build campuses, the team operates through a hybrid delivery model: OpenAI provides commissioning leadership, discipline ownership, governance, and project integration, while commissioning partners provide field and test engineering capacity to support inspections, startup, testing, and turnover. About the Role We are seeking a Commissioning Project Lead to own the commissioning strategy and execution for a large-scale, self-build data center project. You will lead the overall commissioning program from early construction planning through startup, functional testing, integrated systems testing, and final turnover. You will establish the commissioning execution plan, integrate commissioning activities into the master project schedule, coordinate multidisciplinary readiness, and lead the vendor commissioning partners providing field and test engineering capacity. This role serves as the primary commissioning interface to project leadership, construction management, contractors, equipment vendors, operations, and commissioning partners. You will be responsible for creating clarity across organizations, identifying readiness and schedule risks early, and ensuring the facility progresses through testing and turnover against clearly defined acceptance criteria. The role will initially support planning and coordination in a hybrid capacity and transition to full-time onsite presence as construction, inspections, startup, testing, and t

At Vanta, our mission is to help businesses earn and prove trust. We believe that security should be monitored and verified continuously, and we empower companies to practice better security and prove it with ease. Vanta has a kind and talented team, and while some have prior security experience, many have been successful at Vanta without it. Vanta’s Developer Experience team builds the tools engineers use every day to bring ideas to production rapidly and reliably. You’ll empower other Vanta engineers to leverage cutting-edge technologies and best practices to make Vanta more performant and scalable on a platform level. Example projects include modernizing our CI/CD pipelines, introducing new test frameworks, launching AI-powered dev tools, and scaling developer environments to support a growing engineering team. This team has a wide breadth of impact across all of product engineering. The work we do compounds in value by making it easier for engineers to diagnose and solve bugs, streamline workflows, and ship value to our customers quickly and safely. Vanta engineers design and develop new product functionality and infrastructure leveraging modern frameworks and tooling, including TypeScript, React, Node.js, MongoDB, Github Actions, and various AWS services such as Fargate and ECS. Vanta has a kind and talented team, and while some have prior security experience, many have been successful at Vanta without it. We’d love for you to join us! You will: Set direction for critical dev infrastructure, enabling us to stay ahead of continued rapid growth Design and build CI and build systems that ensure Vanta engineers can develop and ship robust products quickly and confidently Improve the efficiency and reliability of our deployment workflows, including tools for hotfixes, rollbacks, and incident mitigation Lead development of tools that accelerate feedback loops — from typechecking and linting to running tests and deploying changes Build and maintain scalable developmen

About the Team The Cybersecurity Products team builds products at the frontier of AI and cybersecurity. Our work includes Codex Security and related cyber products that turn advances in model capability into dependable tools for defenders. We help teams find, validate, and remediate vulnerabilities, continuously improve the security of software, and test AI-powered applications before they reach production. About the Role As a Full Stack Software Engineer, you will build the product experiences and systems that make AI-powered security useful in real engineering environments. You will work across web surfaces, APIs, orchestration, data models, and integrations to help security and engineering teams move from a codebase or application to evidence-backed findings, prioritized remediation, and revalidation. You will collaborate closely with product engineers, security researchers, and customer-facing teams. The work spans fast-moving product development and hard systems problems: long-running workflows, large repositories, sensitive data, reliability, observability, and a high bar for earning user trust. This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees. In this role, you will: Build end-to-end workflows for vulnerability discovery, security scanning, red teaming, findings review, remediation, and reruns. Design and operate backend services for long-running security work, including APIs, asynchronous orchestration, durable state, and integrations with developer workflows. Make complex security results actionable through clear product surfaces, strong evidence, thoughtful prioritization, and reliable reporting. Partner with security researchers, product teams, and users to evaluate quality, reduce noise, improve coverage, and ship safely. You might thrive in this role if you: Have experience shipping production full-stack products across modern web frontends and backend s

About the Team The Release Engineer team is responsible for building and maintaining the systems that power software delivery—from CI/CD pipelines and artifact management to release automation and fleet telemetry. We ensure software across bootloaders, firmware, operating systems, and cloud services is built reproducibly, validated rigorously, and released safely at scale. About the Role As a Release Engineer, you’ll design, build, and operate release infrastructure that enables reliable, secure, and traceable software delivery across complex multi-component systems. You’ll partner closely with embedded, cloud, and QA teams to ensure that every build—from development to OTA deployment—is fast, verifiable, and production-ready. We’re looking for engineers who take pride in automation, build reproducibility, and system reliability—and who enjoy building the connective tissue that allows hardware and software to ship together seamlessly. This role is based in San Francisco, CA. We use a hybrid work model of four days in the office per week and offer relocation assistance to new employees. In this role, you will: Design and operate CI/CD pipelines for multi-component builds (bootloader, firmware, OS images, backend, companion apps) using hermetic toolchains. Define versioning and branching strategies; automate promotions, changelogs, and artifact retention. Integrate unit, integration, and hardware-in-the-loop (HIL) test results; quarantine flaky tests, auto-bisect failures, and block unsafe promotions. Build A/B OTA update flows with verity and health checks; run staged rollouts and canaries; implement safe rollback and roll-forward strategies. Implement code signing for binaries and firmware, generate SBOMs, run vulnerability scanning, and attach build attestations and provenance. Manage dashboards and alerts for build health, promotion latency, failure rates, and fleet update telemetry. You might thrive in this role if you: Have experience building and operating buil

About the Team Security is at the foundation of OpenAI’s mission to ensure that artificial general intelligence benefits all of humanity. The Security team protects OpenAI’s technology, people, and products. We are technical in what we build but are operational in how we do our work, and are committed to supporting all products and research at OpenAI. Our Security team tenets include: prioritizing for impact, enabling researchers, preparing for future transformative technologies, and engaging a robust security culture. About the Role We’re seeking an exceptional Staff - Principal level offensive security domain expert to build agents that continuously identify and coordinate remediation of vulnerabilities across OpenAI’s infrastructure and applications. You will be the technical owner of this effort, combining deep offensive security judgment with agent engineering to build a production system that can operate safely and reliably at scale. As OpenAI increasingly uses automation throughout the company, we believe our security testing must become increasingly automated as well. Advances in model capabilities create an opportunity to test more of our attack surface than would be possible through human effort alone and a need to ensure that we remain ahead of those same capabilities as they become available to attackers. In this role, you’ll build a portfolio of specialized agents that develop a deep understanding of OpenAI’s infrastructure, applications, processes, and security boundaries. These agents will combine internal context with feedback from running systems to explore our cloud environments, Kubernetes clusters, web applications, endpoints, external attack surface, and other high-value targets. The goal is for agents to not only discover vulnerabilities, but also to validate exploitability, document impact, drive remediation, and verify fixes. Success will be measured through outcomes like vulnerabilities fixed, attack surface covered, and performance on evals