Reliability Engineer Jobs

Scale AI is seeking a highly skilled and motivated Software Engineer, ARC (Architecture, Reliability, & Compute) to join our dynamic Public Sector Engineering team. As a part of this team, you will define how the company ships software, establishing the patterns for deploying into complex government and high-security environments, rather than just running Terraform scripts. You will build and maintain internal CLIs/tools that standardize testing, deployment, environment management and are tools that engineering relies on to prevent downstream breakages. You will execute on automated deployment efforts to pay down tech debt, creating fully functional staging/testing environments, and defining the company's standard for safe deployments. You will: Design and implement secure scalable backend systems for Public Sector customers, leveraging Scale's modern and cloud-native AI infrastructure. Own services or systems and define their long-term health goals, while also improving the health of surrounding components. Re-architect the stack to run in compliant or restrictive environments. This requires designing swappable components (auth, storage, logging) to meet government/security mandates without breaking the product. Collaborate with cross-functional teams to define and execute the vision for backend solutions, ensuring they meet the unique needs of government agencies operating in secure environments. Participate actively in customer engagements, working closely with stakeholders to understand requirements and deliver innovative solutions. Contribute to the platform roadmap and product strategy for Scale AI's Public Sector business, playing a key role in shaping the future direction of our offerings. Must have: At least an active secret clearance and the ability & willingness to up level to TS/SCI with CI Poly. This is a requirement and candidates will not be considered who do not hold at least a secret clearance Ideally you'd have: Full Stack Development: Prof

Opportunity Overview: We’re looking for a senior-level automation engineer who will help raise the bar on release quality, environment reliability, and change safety across Cohere’s platform. You’ll partner closely with Product, Engineering, Platform, and SRE to build scalable automation, guardrails, and validation systems that reduce production risk while increasing delivery velocity. This is not a “test scripts only” role. You’ll shape automation strategy, embed quality into the SDLC, and help define how changes move safely from dev → staging → UAT → prod in a fast-moving healthcare platform. You’ll help define how quality scales as Cohere grows. This role has real influence over release safety, platform reliability, and how engineering teams ship software in a regulated, high-impact domain. You won’t just test features — you’ll shape how Cohere delivers them safely to production. What you’ll do: Own and evolve Cohere’s end-to-end test automation strategy across UI, API, config changes, and critical workflows Design and maintain scalable E2E automation frameworks for multi-tenant, payer-specific workflows Build automated validation for deployment guardrails, release readiness, and production change safety Partner with Platform/DevOps to integrate automation into CI/CD pipelines and deployment workflows Create automated coverage for high-risk paths (authorization flows, partner integrations, file pipelines, feature flags, config changes) Drive test reliability, flake reduction, and actionable failure signals Define and enforce quality gates for prod releases, blue/green and canary deployments, and config changes Collaborate with Product and Engineering to ensure business outcomes are testable, measurable, and observable Improve test data management and environment stability to enable reliable automation at scale Mentor engineers on testability, automation best practices, and quality-first development Partner with SRE and Security to ensure production readines

Opportunity Overview: We’re looking for a senior-level automation engineer who will help raise the bar on release quality, environment reliability, and change safety across Cohere’s platform. You’ll partner closely with Product, Engineering, Platform, and SRE to build scalable automation, guardrails, and validation systems that reduce production risk while increasing delivery velocity. This is not a “test scripts only” role. You’ll shape automation strategy, embed quality into the SDLC, and help define how changes move safely from dev → staging → UAT → prod in a fast-moving healthcare platform. You’ll help define how quality scales as Cohere grows. This role has real influence over release safety, platform reliability, and how engineering teams ship software in a regulated, high-impact domain. You won’t just test features — you’ll shape how Cohere delivers them safely to production. What you’ll do: Own and evolve Cohere’s end-to-end test automation strategy across UI, API, config changes, and critical workflows Design and maintain scalable E2E automation frameworks for multi-tenant, payer-specific workflows Build automated validation for deployment guardrails, release readiness, and production change safety Partner with Platform/DevOps to integrate automation into CI/CD pipelines and deployment workflows Create automated coverage for high-risk paths (authorization flows, partner integrations, file pipelines, feature flags, config changes) Drive test reliability, flake reduction, and actionable failure signals Define and enforce quality gates for prod releases, blue/green and canary deployments, and config changes Collaborate with Product and Engineering to ensure business outcomes are testable, measurable, and observable Improve test data management and environment stability to enable reliable automation at scale Mentor engineers on testability, automation best practices, and quality-first development Partner with SRE and Security to ensure production readines

About the Team The Spark Platform team owns and operates DoorDash's Apache Spark ecosystem — the execution runtime, remote shuffle service, cluster scheduler, and reliability tooling that powers the company's data, analytics, and ML workloads. We run Spark across the company at significant scale and continue to expand the workloads, capabilities, and consumer base we serve. Orchestrating and operating thousands of Spark cluster deployments is a complex distributed system problem which the team invests heavily in runtime optimization, systems architecture, multi-tenant scheduling, and end-user tooling. About the Role As a Software Engineer on Spark Platform, you will execute across the surfaces of our in-house Spark deployment that serves the entire company. The work spans Spark runtime upgrades and performance, multi-tenant scheduling and executor bin-packing on Kubernetes, cluster lifecycle automation, and the observability and incident automation that keep the platform sustainable. You will move between layers as the work demands — picking up the next high-leverage problem regardless of where it sits — and partner closely with the rest of the team and with platform consumers across the company. You must be located in San Francisco, Sunnyvale, Seattle, or New York City for this hybrid position. You will report into the Engineering Manager on our Spark Platform team. You're excited about this opportunity because you will… Build and operate an in-house Spark platform that runs at company-wide scale, spanning runtime, scheduler, reliability, and user-facing tooling. Drive multi-tenant scheduling, executor bin-packing, and cost-aware placement that let a small team serve dozens of consumer teams. Own pieces of cluster lifecycle automation — provisioning, upgrades, capacity changes, and node-failure handling — at a scale where these stop being manual events. Build the observability and incident automation that make the platform debuggable end-to-end and keep on-call sus

About the Team API Frontiers turns OpenAI’s frontier models into production APIs that developers can use to build reliable products and agents. We own the core path connecting models to developers through the Responses API, with a focus on safety, reliability, and speed. Working closely with Research, Safety, Codex, and other API teams, we bring new model capabilities into production and improve them through developer feedback. About the Role We are looking for a backend software engineer to build and operate the services behind the Responses API. You will shape API behavior, bring new capabilities from research into production, and make long-running agent workflows dependable and fast. The work combines distributed systems engineering with product judgment: designing useful developer interfaces, managing staged rollouts, and following production issues through to durable fixes. In this role, you will: Design, build, and operate APIs and backend services that bring frontier model capabilities to developers. Partner with Research, Safety, Codex, and API teams to define API behavior and deliver safe, staged launches. Build API capabilities for agent workflows, including task delegation, context sharing, and parallel execution. Strengthen long-running request reliability across timeouts, cancellation, streaming, and background execution. Improve request-processing performance and tail latency through profiling, efficient systems code, and persistent connections. Turn developer feedback and production failures into better observability, diagnostics, and lasting product improvements. Your background might look something like: 5+ years of experience building and operating backend services or developer-facing APIs in production. Strong software engineering fundamentals, with practical knowledge of distributed systems, concurrency, and asynchronous execution. Ability to diagnose production failures and performance bottlenecks using observability data and profiling. Product

Join the engineering teams that bring OpenAI’s ideas safely to the world! The Applied Engineering team works across research, engineering, product, and design to bring OpenAI’s technology to consumers and businesses. We seek to learn from deployment and distribute the benefits of AI, while ensuring that this powerful tool is used responsibly and safely. Safety is more important to us than unfettered growth. About the Role As OpenAI continues to grow, we are looking for experienced, problem-solving engineers to ensure our systems scale. Our success depends on our ability to quickly iterate on products while also ensuring that they are performant and reliable. You will work in a deeply iterative, collaborative, fast-paced environment to bring our technology to millions of users around the world, and ensure it’s delivered with safety and reliability in mind. Successful candidates will play a crucial role in ensuring the reliability, scalability, and performance of our systems as we continue to expand. As a reliability expert, you will be at the forefront of maintaining and enhancing the stability, scalability, and performance of our rapidly evolving infrastructure. You will work closely with cross-functional teams, including software engineers, product managers, and data scientists, to build and maintain resilient systems that can handle our growing user base and workload. In this role, you will: Design and implement solutions to ensure the scalability of our infrastructure to meet rapidly increasing demands. Build and maintain the load, chaos and synthetic-testing software leveraged by development teams to make the systems they design and operate more reliable. Build and maintain automation tools to streamline repetitive tasks and improve system reliability. Build and maintain the platform for CPU, storage, GPU, and network lifecycle management to drive efficiency, accountability and dynamic optimization of our resources. Implement fault-tolerant and resilient design

ABOUT BASETEN Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F , led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to to ship AI products. THE ROLE Baseten’s Inference Stack team builds the distributed runtime that powers large-scale LLM inference across our platform. We operate at the intersection of distributed systems, model performance, infrastructure, and developer experience. We enable customers to deploy and operate cutting-edge LLM models with industry-leading performance, scalability, reliability, and ease of use. As a Software Engineer on the Inference Stack team, you’ll work across the stack - from the developer experience customers use to deploy models, the libraries used for features like tool calling and reasoning, all the way down to the systems we use to orchestrate deployments in Kubernetes and route traffic efficiently. This is an ideal role for engineers who enjoy owning systems in production, solving hard integration problems, and making complex infrastructure simple and reliable for users. EXAMPLE INITIATIVES Blog Posts https://www.baseten.co/blog/nvidia-dynamo-day-baseten-inference-stack/ https://www.baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo/ https://www.baseten.co/blog/how-baseten-multi-cloud-capacity-management-mcm-powers-cloud-self-hosted-and-hybr/#comparing-deployment-options-cloud-vs-self-hosted-vs-hybrid RESPONSIBILITIES Develop infrastructure and orchestration systems for deploying and managing large-scale distributed LLM inference Work across the stack, from customer-facing features to low-le

The Consumption Based Billing & Licensing (CBBL) team builds the systems that decide what customers can access, what they've purchased, and what they've used. We own the infrastructure behind licensing, entitlements, and usage-based billing for Asana's newer products and add-ons — making sure customers get the right paid features, usage is tracked accurately, and admins have clear ways to understand and manage their accounts. As Asana expands its product surface and pricing models, the systems we build directly shape how the business grows and how customers experience the value of what they've bought. This role is based in our Vancouver office with an office-centric hybrid schedule. The standard in-office days are Monday, Tuesday, and Thursday. Most Asanas have the option to work from home on Wednesdays. Working from home on Fridays depends on the type of work you do and the teams with which you partner. If you're interviewing for this role, your recruiter will share more about the in-office requirements. What you’ll achieve: Lead and grow a team of engineers working across backend systems, platform services, and customer-facing experiences. Set technical direction and help shape the roadmap for licensing and billing capabilities. Partner closely with Product, Design, Data, Finance, Sales, and other engineering teams to turn ambiguous business needs into shipped solutions. Drive execution on projects such as: licensing and access control usage tracking and billing admin dashboards, alerts, and controls bulk and automated license assignment trial and add-on provisioning flows Make thoughtful trade-offs across scalability, reliability, correctness, and speed of delivery. Coach engineers, support their growth, and build a high-performing team culture. Communicate clearly with senior stakeholders and explain technical decisions in business-friendly terms. About you: Strong engineering management experience, ideally at the level of leading a team through complex prod

About the Team OpenAI's mission is to ensure that AGI benefits all of humanity. The Business Systems team helps make that mission possible by building the internal products and platforms that allow OpenAI to operate with speed, reliability, and care. We build internal applications and workflows for Finance and Supply Chain. Our work spans product discovery, React and TypeScript interfaces, Python services and APIs, data models, workflow orchestration, enterprise integrations, and the systems that connect people to systems of record. We work directly with the people who use these products and care about correctness, permissions, auditability, and production reliability. Examples of our work include building an integration platform for supply chain integrations, integrations with Oracle Fusion and Zip, contract intelligence applied to B2B revenue recognition, and Temporal-based agentic workflows for credit checks, duplicate bank detection, and invoice triaging. We turn these efforts into reusable patterns that can support many workflows, rather than one-off automations. About the Role We are looking for Product Engineers to build internal applications end to end. This role spans product discovery, user experience, frontend, backend services, data models, workflow orchestration, and integrations with order management, fulfillment, and supply chain systems. You will take a problem from a first conversation with a Finance or Supply Chain partner through design, implementation, rollout, and production support. Strong candidates combine product judgment with engineering depth. You should be comfortable moving between a React interface, a Python API, a durable workflow, and an integration with an enterprise system. You should be able to ship a useful first version quickly while building the foundations for reuse, security, and long-term maintainability. Direct AI experience is helpful, but the core requirement is strong product engineering judgment and reliable execution. I

About the Team OpenAI’s Infrastructure Operations team is responsible for the availability, reliability, and operational excellence of one of the world’s largest AI infrastructure networks. The team owns day-to-day operations of production AI networks across Industrial Compute's data centers, working with colocation providers, deployment teams, and hardware vendors to deliver highly available GPU infrastructure for AI training and inference workloads. About the Role We are seeking an Infrastructure Operations Engineer to operate and improve the large-scale Ethernet fabrics that support GPU clusters, storage systems, and management infrastructure. This role combines hands-on production operations with automation, observability, and incident response across a global AI network. The ideal candidate has experience operating high-availability data center, cloud, AI, or HPC networks and can move comfortably from physical-layer troubleshooting to routing and fabric behavior, change execution, and root-cause analysis. You will partner closely with network architecture, systems engineering, GPU engineering, storage engineering, security, deployment, site operations, service providers, colocation partners, and hardware vendors to raise reliability and reduce operational toil. Key Responsibilities Own the operational health, availability, and reliability of production AI network infrastructure across Industrial Compute's data centers. Monitor, troubleshoot, and resolve network incidents while meeting service-level objectives (SLOs), reducing Mean Time to Detect (MTTD), and minimizing Mean Time to Recovery (MTTR). Operate and maintain large-scale Ethernet fabrics supporting GPU compute, storage, and management networks. Execute production network changes, maintenance windows, and capacity expansions with minimal customer impact. Manage the hardware lifecycle, including switch and optics replacements, RMA coordination, software upgrades, and preventive maintenance. Support new A

About the Team The Cloud Agents team builds product infrastructure for long-running agents in the cloud: orchestration, sandboxing and isolation, secure environment connectivity, secrets and identity, observability, reliability, and cost controls. These agents securely connect to diverse developer and customer environments and use tools to accomplish goals. We partner closely with product, research, and infrastructure teams to turn agentic capabilities into dependable platforms for OpenAI products and developers building on OpenAI. About the Role We are looking for an experienced software engineer to help build and scale our cloud agent platform. You will design and operate systems for orchestrating agents at scale. You will work closely with product engineers on ChatGPT, API, and Codex to define the right abstractions and enable them to ship products quickly. Strong backend or infrastructure experience is important; experience with Python, Rust, distributed systems, cloud infrastructure, or product platforms is especially helpful. In this role, you will: Design and scale the orchestration, sandboxing and storage systems that run agentic workloads for Codex, ChatGPT, and the OpenAI API. Partner with product engineers to build a platform that enables them to ship quickly and turn feedback into robust abstractions. Improve reliability, security, performance, and cost efficiency for long-running agents. Deploy services that can operate across different environments and clouds. Your background might look something like: 9+ years of professional engineering experience, excluding internships, in relevant roles at technology and product-driven companies. Experience leading large-scale backend, platform, or infrastructure projects from ambiguous problem statements to production systems. Proficiency in one or more backend languages such as Python, Go, Rust, TypeScript, or similar, and the ability to move across service, platform, and product boundaries. Strong understanding

About the Team: The Database Systems team specializes in high-performance distributed databases. Our team built Rockset, the real-time search, analytics, and vector database that powers all vector search and retrieval augmented generation (RAG) at OpenAI. In addition to retrieval, as an online database, Rockset powers core functionality across all of OpenAI's product lines and many critical internal use cases. About the Role : We are looking for engineers passionate about distributed systems, close-to-the-metal performance optimization (our core engine is written in C++), and building scalable database infrastructure from the ground up. As an engineer on the Database Systems team, you'll contribute to the core database engine, driving improvements across ingestion, query execution, indexing, and storage. You'll partner with teams across OpenAI to unlock new product capabilities and help scale online database reliability and throughput as usage grows by orders of magnitude. In this role you will: Design, build, and operate high-performance distributed systems Identify and resolve performance bottlenecks to scale infrastructure to the next order of magnitude Define long-term technical direction and guide system evolution Collaborate with product, engineering, and research teams to deliver scalable and reliable infrastructure Dig deep into complex production issues across the stack Contribute to incident response, postmortems, and best practices for system reliability You might thrive in this role if you: Have significant experience building, scaling, and optimizing distributed systems at scale Are curious about database internals, storage engines, or low-latency query systems Enjoy debugging challenging performance issues in complex, high-throughput systems Have experience operating production clusters at scale (e.g., Kubernetes or other orchestration systems) Think rigorously about scalability, correctness, and reliability Thrive in fast-paced environments with high

About the Team The Safety Systems team is dedicated to ensuring the safety, robustness, and reliability of AI models and their deployment in the real world. Learn more about OpenAI’s approach to safety. Building on the many years of our practical alignment work and applied safety efforts, Safety Systems addresses emerging safety issues and develops new fundamental solutions to enable the safe deployment of our most advanced models and future AGI, to make AI that is beneficial and trustworthy. About the Role At OpenAI, we're dedicated to advancing artificial intelligence, and we know that creating a secure and reliable platform is vital to our mission. That's why we're seeking a software engineer to help us build out our trust and safety capabilities. In this role, you'll work with our entire engineering team to design and implement systems that detect and prevent abuse, promote user safety, and reduce risk across our platform. You'll be at the forefront of our efforts to ensure that the immense potential of AI is harnessed in a responsible and sustainable manner. Your Responsibilities: Architect, build, and maintain anti-abuse and content moderation infrastructure designed to protect us and end users from unwanted behavior. Work closely with our other engineers and researchers to utilize both industry standard and novel AI techniques to measure, monitor and improve AI models’ alignment to human values. . Diagnose and remediate active incidents on the platform and build new tooling and infrastructure that address the root causes of system failure. You might thrive in this role if: You have built and run production services in a high growth, rapidly scaling environment. You can debug live issues and restore systems quickly. You have worked on content safety, fraud, or abuse, or are motivated and excited to work on present-day (“now-term”) AI safety. You have experience with Python or with modern languages such as C++, Rust, or Go, and are able to quickly ramp up on Py

About the Team The Safety Systems team is dedicated to ensuring the safety, robustness, and reliability of AI models and their deployment in the real world. Building on the many years of our practical alignment work and applied safety efforts, Safety Systems addresses emerging safety issues and develops new fundamental solutions to enable the safe deployment of our most advanced models and future AGI, to make AI that is beneficial and trustworthy. Learn more about OpenAI’s approach to safety About the Role As an Analytics Engineer in Safety Systems, you will play a pivotal role in building a data-centric culture, enhancing decision-making processes, and driving strategic initiatives through analytics. You will partner closely with Engineering, Research, and Data Science to develop and maintain canonical data sources and source-of-truth dashboards that enable both people and AI agents across the organization to derive trustworthy, actionable insights. You will own the consumption layer for safety metrics: defining intuitive, reliable ways for stakeholders across Safety Systems, partner teams, and leadership to understand the safety of our products, answer safety-related questions independently, and inform product decisions and company strategy. Most importantly, you will be a core member of the Safety Systems team, collaborating with researchers and engineers to advance our goals of safe, robust, and reliable AI. This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees. In this role, you will: Design and maintain canonical datasets that serve as sources of truth for safety metrics. Develop and refine data products such as dashboards, reports, agent-enabled workflows, and machine-readable interfaces that empower stakeholders to extract and analyze data independently. Work closely with stakeholders in Engineering, Research, and Data Science to understand their decision-making n

About the Team Revenue Platform sits at the intersection of customer experience, financial precision, and enterprise-grade reliability. We build both end-user experiences and the underlying platform capabilities that power invoicing, billing, payments, and revenue recognition across OpenAI. Our work spans high-leverage customer surfaces and deep, reusable platform primitives used by multiple teams. These are foundational systems that will support OpenAI’s growth for years to come, and we’re looking for engineers who care deeply about craftsmanship, correctness, and building platforms and experiences that scale gracefully and are a joy to build on. OpenAI’s mission is to ensure that artificial general intelligence benefits all of humanity. Achieving that mission requires more than great models, it demands a world-class commercial platform that enterprises can trust. The Revenue Platform team enables organizations to confidently adopt, scale, and manage their usage of OpenAI products by providing reliable, extensible financial infrastructure and shared services that power the broader ecosystem. We’re looking for a full-stack engineer to take end-to-end ownership of critical revenue workflows and the platform abstractions behind them. You’ll work across APIs, data models, shared services, and user interfaces to solve hard problems in scale, correctness, and usability. Partnering closely with Product, Finance, Sales, Operations, and other engineering teams, you’ll turn complex business requirements into durable platform capabilities, raising the bar on engineering quality and developer experience as our products and enterprise footprint grow. About the Role As a Full Stack Engineer on the Revenue Platform team, you will design, build, and operate platform services and user-facing interfaces that form the backbone of OpenAI’s commercial engine. You’ll collaborate with product, design, finance, and engineering partners to deliver intuitive customer experiences while also