Recovery Operations Senior Analyst in San Francisco

About the Team The Infrastructure Engineering function sits within IT and is responsible for reliably building, deploying, and operating critical on prem and hybrid environments that power internal services and critical R&D environments. This is an early, high-leverage technical role focused on applying strong Site Reliability Engineering discipline to environments where uptime, safety, recoverability, and security are non-negotiable. This person helps replace bespoke, one-off infrastructure with standardized infrastructure-as-code building blocks that compound reliability and operational leverage as OpenAI scales. About the Role We are looking for an experienced Site Reliability Engineer working on security infrastructure to design, build, and operate reliable, secure, and scalable infrastructure that underpins identity, access, endpoint, and shared platform services across the company. In this role, you will be a senior technical owner for infrastructure and identity systems end to end, from architecture and implementation through policy enforcement, upgrades, recovery, and day-two operations. You will build durable, production-grade platforms that remove operational friction, enforce security by default, and enable teams to move faster with confidence. This role is well suited for a hands-on senior engineer who thrives in ambiguity, enjoys owning complex systems end to end, and raises the reliability and security bar by replacing fragile implementations with standardized, repeatable infrastructure. This role is based in our San Francisco HQ and requires in-office presence. In this role, you will: Design, build, and operate reliable infrastructure across on-prem, hybrid, shared, and product adjacent environments. Establish standardized infrastructure patterns that replace bespoke implementations with repeatable, auditable, secure-by-default systems. Own the lifecycle of critical infrastructure platforms, including provisioning, deployment, upgrades, patching,

ABOUT BASETEN Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F , led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to to ship AI products. THE ROLE As an OS / K8s Systems Engineer at Baseten, you’ll build the automation and systems that turn raw GPU hardware into production-ready compute. From provisioning to orchestration, you’ll own the software layer that makes our infrastructure reproducible, scalable, and reliable across data centers. This is a senior, hands-on role focused on building systems not operating them. You’ll work close to the metal designing OS images, building provisioning pipelines, and automating cluster bring-up from scratch. Your work will define how quickly we can turn new capacity into usable compute. EXAMPLE INITIATIVES Zero-to-cluster automation Build workflows that take new hardware from unprovisioned to fully operational cluster. Provisioning systems Design PXE-based or equivalent systems for imaging and lifecycle management. Reproducible infrastructure — Ensure clusters deploy consistently across data centers. RESPONSIBILITIES Own the end-to-end automation of cluster bring-up and lifecycle management. Build and maintain OS images, provisioning systems, and configuration pipelines. Deploy and operate cluster orchestration platforms (Kubernetes, Slurm, or similar). Design systems for reproducibility across sites and hardware generations. Automate upgrades, rollouts, and failure recovery. Optimize system performance, including GPU utilization and networking. Partner with hardware and network teams to validate and improve system b

About the Team OpenAI’s Infrastructure Operations team is responsible for the availability, reliability, and operational excellence of one of the world’s largest AI infrastructure networks. The team owns day-to-day operations of production AI networks across Industrial Compute's data centers, working with colocation providers, deployment teams, and hardware vendors to deliver highly available GPU infrastructure for AI training and inference workloads. About the Role We are seeking an Infrastructure Operations Engineer to operate and improve the large-scale Ethernet fabrics that support GPU clusters, storage systems, and management infrastructure. This role combines hands-on production operations with automation, observability, and incident response across a global AI network. The ideal candidate has experience operating high-availability data center, cloud, AI, or HPC networks and can move comfortably from physical-layer troubleshooting to routing and fabric behavior, change execution, and root-cause analysis. You will partner closely with network architecture, systems engineering, GPU engineering, storage engineering, security, deployment, site operations, service providers, colocation partners, and hardware vendors to raise reliability and reduce operational toil. Key Responsibilities Own the operational health, availability, and reliability of production AI network infrastructure across Industrial Compute's data centers. Monitor, troubleshoot, and resolve network incidents while meeting service-level objectives (SLOs), reducing Mean Time to Detect (MTTD), and minimizing Mean Time to Recovery (MTTR). Operate and maintain large-scale Ethernet fabrics supporting GPU compute, storage, and management networks. Execute production network changes, maintenance windows, and capacity expansions with minimal customer impact. Manage the hardware lifecycle, including switch and optics replacements, RMA coordination, software upgrades, and preventive maintenance. Support new A

Who We Are Notion is the collaborative AI workspace where teams and agents think together . We're building one place where your knowledge, projects, meetings, and AI tools live side by side, so work is faster, clearer, and less fragmented. Millions of individuals, small teams, and large companies run their work on Notion. Notinos (our employees) are customer zero in bringing this future of work to life. We care about craft, building things that last, and the belief that great work is still fundamentally human. Our goal isn’t to ship the next feature. Each and every team of Notinos is working to set the standard for how humans work together in the AI era. From building a business’s system of record to making and managing AI agents to automating away the busy work, we care deeply about giving our customers more time for their life’s work. About the Role: We’re seeking an experienced UX Researcher to define and scale how we evaluate Notion’s AI-powered experiences—focusing on what “good” looks like not only for model output quality, but for the end-to-end product experience where people discover, set goals, delegate work, review results, and build trust over time with AI. This role sits at the intersection of research craft and evaluation operations: you’ll run studies that uncover user mental models, expectations, and failure/recovery behaviors, then translate those insights into reusable rubrics, workflows, and measurement approaches that product, design, engineering, and data science can apply consistently. This role can be based in either San Francisco or New York City. We work from our offices on Mondays, Tuesdays and Thursdays (our Anchor Days) because we do our best thinking and building together in person. We’re looking for someone who’s excited to work alongside the team during those days. What You'll Achieve: Define what “good” looks like (frameworks & rubrics): Establish clear, reusable evaluation criteria that reflect real user expectations—helpfulness,

We are hiring a Security Software Engineer to design and implement the hardware-backed security foundations used across OpenAI’s device ecosystem. A central focus of this role is hardening the boundary between our policy systems and the HSMs that protect sensitive cryptographic keys. This boundary determines which operations may be performed, what may be signed, which policies must be satisfied, and how changes to trusted software and policy are authorized. You will develop security-critical software and firmware within, or immediately adjacent to, an HSM trust boundary. Depending on your background, this may include HSM trusted applications, firmware services, cryptographic mechanisms, device drivers, PKCS#11 components, secure-provisioning protocols, or signing-policy enforcement systems. This is a hands-on software-engineering role. You will be expected to design systems, write and review production code, debug across hardware and software boundaries, and carry projects from initial requirements through deployment. It is not an HSM administration, PKI operations, compliance, or architecture-only position. In This Role, You Will Design and implement security-critical software and firmware for HSMs, secure elements, trusted execution environments, and hardware roots of trust. Build and harden the policy-to-HSM boundary responsible for authorizing certificate issuance and cryptographic signing operations. Develop HSM trusted applications, firmware components, host interfaces, device drivers, SDKs, or cryptographic service integrations. Implement or extend cryptographic interfaces such as PKCS#11, OpenSSL providers or engines, platform key-storage APIs, or comparable hardware-security interfaces. Build firmware and software that cryptographically enforces key generation, provisioning, usage, rotation, recovery, and destruction policies. Design and implement HSM-backed certificate authority, code-signing, key-management, and device-identity systems. Develop end-to-end

About the Team OpenAI’s Hardware organization develops AI-native silicon and system-level solutions for the unique demands of advanced AI workloads. Building on efforts like Jalapeño, the team is developing future generations of AI-native silicon and tightly integrated systems to power the next generation of frontier models. By co-designing chips, systems, tools, and methodologies, the team helps deliver faster, more efficient, and production-ready hardware for OpenAI’s supercomputing platform. About the Role As a Supply Chain Program Manager, you will own material readiness and supply chain execution for critical hardware programs spanning custom silicon, systems, memory, storage, networking, and rack infrastructure. You will work cross-functionally with Engineering, Strategic Sourcing, Manufacturing Operations, Finance, Planning, Quality, and external suppliers to develop and execute scalable supply strategies that support aggressive product development and deployment timelines. This role requires deep understanding of hardware supply chains, material planning, NPI execution, supplier management, and operational scaling in constrained and rapidly evolving environments. In this role you will: Material Readiness & Supply Planning - Own end-to-end material readiness across NPI and production phases, including building the necessary framework and processes for enablement. Drive supply planning and execution for long lead-time and constrained commodities including ASICs, HBM, DDR, SSDs, networking, optics, power, thermal, and mechanicals. Build and manage material readiness plans aligned to proto/pre-EVT, EVT, DVT, PVT, and mass production schedules. Monitor supply health, lead times, inventory positions, allocation risk, and capacity constraints. Drive shortage management, allocation mitigation, and recovery planning. Coordinate supply commits, forecast alignment, and supply continuity planning with suppliers and manufacturing partners. Cross-Functional Program Ma

About the Team The OpenAI Robotics team is focused on unlocking general-purpose robotics and pushing towards AGI-level intelligence in dynamic, real-world settings. Working across the entire model stack, we integrate cutting-edge hardware and software to explore a broad range of robotic form factors. We strive to seamlessly blend high-level AI capabilities with the constraints of physical systems to improve peoples’ lives. About the Role OpenAI Robotics depends on high-quality real-world robot data from a large, live operational environment. That environment has many deployed workcells, changing configurations, and little tolerance for downtime. We are seeking a Field Engineer to help keep that environment running well. You will own the day-to-day technical health of robotic workcells used in ongoing data acquisition operations, diagnose and resolve failures across hardware and software, and build the practical tools, documentation, and support workflows that make operators and technicians more effective. The work is close to the floor, close to the failure modes, and close to the research impact. This role sits at the intersection of software, robotics hardware, and live operations. The best people in it are practical, technically sharp, calm under pressure, and motivated by making real systems work reliably at scale. This role will be based in San Francisco, CA 5 days per week and offer relocation assistance to new employees. In this role, you will: Serve as an engineering owner for keeping a fleet of robotic workcells online in daily operation. Diagnose, fix, mitigate, or escalate issues spanning mechanics, electronics, controls, software interfaces, logs, and configuration. Partner closely with technicians, operators, and engineering teams to improve scalable fleet support processes. Build or spec lightweight hardware and software tools that improve monitoring, diagnostics, recovery, and handoffs. Create documentation, SOPs, and diagnostic playbooks that turn en

About the Team OpenAI’s Hardware organization develops AI-native silicon and system-level solutions for the unique demands of advanced AI workloads. Building on efforts like Jalapeño, the team is developing future generations of AI-native silicon and tightly integrated systems to power the next generation of frontier models. By co-designing chips, systems, tools, and methodologies, the team helps deliver faster, more efficient, and production-ready hardware for OpenAI’s supercomputing platform. About the Role We're seeking a Security Engineer to join our First-Party Hardware team. In this role, you will own the end-to-end security foundation for OpenAI's first-party AI hardware systems, working across hardware security, embedded security, system security, and practical deployment at data center scale. You will partner with silicon, hardware, firmware, infrastructure, manufacturing, operations, and security teams to define and deliver system-level device trust. This includes boot integrity, device identity, provisioning, attestation, management-plane security, storage encryption, debug controls, firmware update and recovery, RMA, and decommissioning. You will be accountable for turning threat models into requirements, requirements into implementation, and implementation into validation evidence that can support launch decisions. Location: San Francisco, CA (Hybrid: 3 days/week onsite) Relocation assistance available. In this role, you will: Own security requirements, threat models, validation strategy, and launch-readiness evidence for first-party hardware platforms from early design through production deployment. Design and review secure boot, measured boot, roots of trust, platform firmware resilience, firmware signing, recovery, and anti-rollback strategies across heterogeneous devices. Own device identity, provisioning, enrollment, attestation, certificate lifecycle, and key-management requirements across manufacturing and data center bring-up. Harden management

About the Team Compute Foundations builds the software that manages OpenAI’s GPU compute infrastructure across sites, data centers, and infrastructure providers, supporting model training and inference. Our systems turn large, heterogeneous fleets of machines into dependable compute for research and products. We build Kubernetes-based control planes, controllers, services, and APIs that coordinate the lifecycle of machines and clusters. We connect global infrastructure management with the realities of bare-metal systems, giving clients consistent interfaces across differences in hardware, topology, and provider behavior. About the Role You will build distributed systems that provision, configure, and manage compute throughout its lifecycle. Your work will connect global services and Kubernetes controllers with the systems that bring machines online, update them safely, and recover them when something goes wrong. This role combines software architecture with an understanding of how machines and data centers work. You might design a lifecycle API, improve controller performance under high concurrency and provider rate limits, or trace a provisioning failure from an API through reconciliation to network boot or host configuration. You will help these systems remain reliable as the fleet expands across sites and generations of GPU hardware. We value depth in relevant systems and the ability to connect layers. You do not need to arrive as an expert in every component of the stack. In this role, you will: Design, build, and operate Kubernetes-based controllers and distributed services that coordinate infrastructure across sites, isolate failures, and scale as GPU capacity grows. Define APIs and resource models that let clients request and track lifecycle operations through consistent interfaces across hardware platforms and providers. Build provisioning and configuration services that coordinate network boot, hardware management interfaces, and the deployment of firmware,

Who Are We? Postman is the world’s leading API platform, used by more than 45 million+ developers and 500,000 organizations, including 98% of the Fortune 500. Postman is helping developers and professionals across the globe build the API-first world by simplifying each step of the API lifecycle and streamlining collaboration—enabling users to create better APIs, faster. The company is headquartered in San Francisco and has offices in Boston, New York, Austin, Tokyo, London, and Bangalore - where Postman was founded. Postman is privately held, with funding from Battery Ventures, BOND, Coatue, CRV, Insight Partners, and Nexus Venture Partners. Learn more at postman.com or connect with Postman on X via @getpostman. P.S: We highly recommend reading The "API-First World" graphic novel to understand the bigger picture and our vision at Postman. The Opportunity Postman is seeking an experienced AI Systems Reliability Engineer to help define, build, and maintain the infrastructure and processes that ensure the reliability, scalability, and performance of Postman’s AI-powered API and agentic systems in production. This role focuses on monitoring, availability, incident response, and automation to support AI services and tools trusted by millions of developers globally. What You’ll Do Develop and manage reliability metrics (SLOs) for AI-driven API services and agentic AI platform features Implement comprehensive observability and monitoring systems for real-time performance and fault detection Design and drive automated failover, recovery, and incident response strategies for high-availability AI infrastructure Optimize resource utilization, particularly GPU/accelerator efficiency, ensuring cost-effective AI system operation Collaborate closely with engineering, platform, and product teams to align reliability efforts with broader organizational goals Lead efforts to build internal tooling and automation focused on AI system stability and operational excellence Drive continuo

About the Team Frontier Systems Foundations, part of Compute Foundations at OpenAI, builds the systems software foundation that turns new compute infrastructure into reliable, usable capacity for frontier model training. Our mission is to make some of the world's largest GPU clusters work reliably for frontier training. We bring new platforms and clusters online, safely maintain installed fleets, and partner with hardware, infrastructure, and research teams to resolve the system-level issues that keep jobs from running. That means building and maintaining the software closest to the machine: Linux and Ubuntu operating-system images, kernels and modules, drivers, packages and repositories, disks and boot configuration, firmware integration, provisioning, and system-level validation. We make these components reproducible, compatible, and safe to operate across heterogeneous fleets. About the Role We are looking for systems software engineers with deep Linux and host-systems experience to build, qualify, and maintain the operating-system foundation for OpenAI's frontier compute fleet. Relevant backgrounds include kernel and module development, Linux distribution or image engineering, package management, firmware and driver integration, disks and boot, and bare-metal provisioning. You'll work closely with hardware engineers, vendors, and infrastructure teams to bring up new platforms, integrate system components, and debug failures across firmware, disks, boot, operating systems, kernels, drivers, and workload interactions. Your work will directly influence how quickly new capacity becomes usable and how reliably large GPU fleets operate. You should be comfortable writing and maintaining production-quality systems software and automation, but we do not expect expertise across every layer. This is an opportunity to go deep on challenging systems problems while building the image, package, qualification, and recovery paths that power the next generation of frontier models

About the Team pAGI Infra team builds and operates the systems that make large-scale model training and evaluation reliable, efficient, and easy to run. Our work spans distributed training infrastructure, inference and grading platforms, compute scheduling, and research tooling. We partner closely with researchers and engineering teams to turn new research needs into dependable infrastructure, improve GPU efficiency, and shorten the path from an experiment to a validated model. About the Role We’re looking for an AI Systems Engineer to help scale the infrastructure behind our training and evaluation workflows. You’ll own projects from identifying bottlenecks and designing solutions through deployment and operation. The work combines distributed systems engineering, performance optimization, and close collaboration with researchers. You might build a shared grading service, improve resource allocation across workloads, or bring a new training stack into production — directly improving how quickly and reliably research moves forward. In this role, you will: Build and operate infrastructure for large-scale training and evaluation, improving reliability, throughput, and resource efficiency. Develop shared inference and grading platforms with automated capacity management, health monitoring, and visibility into performance. Improve compute scheduling and resource allocation to reduce idle GPU time and help workloads recover quickly from failures. Diagnose bottlenecks across training, inference, and orchestration, and work across teams to improve end-to-end performance. Build self-service tools, automated validation, and observability that help researchers launch experiments, diagnose issues, and compare results with less manual intervention. You might thrive in this role if you: Are excited about the potential of personal AGI and want to build the infrastructure that enables it. Have strong software engineering fundamentals and experience building or operating large-scal

About the Team OpenAI’s 1P Hardware Systems team operates at the intersection of Hardware Engineering, Infrastructure, Supply Chain, Manufacturing, Deployment, and Finance to translate infrastructure demand into an executable systems plan. The Planning Lead connects system demand and deployment timing to site and configuration requirements, power availability, XPU needs, hardware supply commitments, manufacturing capacity, and site readiness. About the Role OpenAI is seeking a 1P Hardware Systems Planning Lead to own the integrated demand, supply, and deployment plan for our 1P AI infrastructure systems. This role will connect infrastructure demand and site power availability to system configurations, XPU requirements, and hardware supply commitments—creating a single, actionable view of whether our deployment plan can be met. You will establish the planning mechanisms that allow teams to see what changed, understand the impact, and act quickly when demand, supply, configuration, site readiness, or power timing moves. You will work across Hardware Engineering, Infrastructure, Supply Chain, Manufacturing, Deployment, Finance, and external partners to identify gaps early and drive recovery plans to closure. This is a highly cross-functional role for someone who combines systems-level thinking, strong analytical judgment, and rigorous program execution. The ideal candidate can turn complex and changing inputs into a clear operating plan, surface the decisions that matter, and drive accountability across teams without relying on formal authority. Success also requires strong communication judgment: the ability to align cross-functional teams, brief leadership at a concise and decision-oriented level, and go deep into the underlying assumptions, dependencies, risks, and recovery plans when needed. In this role, you will: Own the integrated planning process for 1P hardware systems, connecting system demand, deployment timing, site and configuration requirements, power ava

About the Team The Ona team at OpenAI is helping build the software factory for the enterprise. We build infrastructure that enables AI agents to work in secure, customer-controlled cloud environments, with the context, tools, and controls they need to make progress across the software lifecycle—beyond a single developer’s laptop or active session. Our focus is helping enterprises move from experimenting with agents to using them reliably in production. That means solving challenging problems in cloud environments, orchestration, security, and collaboration, while making the experience straightforward for the people directing and reviewing the work. We’re a team that values initiative, close relationships with customers, and exceptional engineering craft. We take ownership, learn quickly, and communicate directly and kindly. About the Role We’re hiring backend-focused Product Engineers across our platform and security product teams. You’ll build infrastructure and customer-facing workflows that let developers and AI agents work reliably in parallel. You’ll work primarily in Go on APIs, complex networking, development environments, and orchestration for long-running tasks. You’ll own outcomes from understanding a user’s problem and choosing an approach through shipping, operating, and improving the solution, working closely with frontend, infrastructure, and security engineers. In this role, you will: Work directly with customers to build developer and security workflows, from getting a project running to investigating findings, reviewing agent-generated changes, and verifying fixes. Build Go services and APIs for provisioning cloud environments, running agents in customer infrastructure, and integrating with source control, CI, and other developer tools. Design reliable orchestration for long-running, parallel work, including durable state, retries, cancellation, and recovery. Build security into execution workflows through clear permissions, credential handling, is

About the Team Training Runtime builds the distributed systems that power OpenAI's largest model training runs - most recently GPT-5.5! The Data Movement area owns the infrastructure that keeps training jobs supplied with the right data at the right time, and keeps model state moving safely and efficiently across large clusters. Our work spans machine learning systems, distributed storage, high-throughput data loading, reliability engineering, and developer experience. Success means researchers can move quickly while training runs remain fast, reproducible, debuggable, and resilient at scale. About the Role We are looking for a deeply hands-on Technical Lead Manager to own datasets throughout our training infrastructure. This person will set the direction for how training jobs read data: the APIs, storage contracts, versioning model, benchmarks, debugging tools, and reliability guarantees that make data access consistent across current and future training frameworks. You will begin as the primary technical owner for dataset reads, working directly in the code while aligning researchers, training framework owners, storage teams, and infrastructure partners around a durable platform. The problem is deceptively hard at frontier scale: make enormous, heterogeneous datasets easy to consume, correct across distributed workers, observable when something goes wrong, and flexible enough to support pretraining, reinforcement learning, and multimodal training. In this role, you will Design and build a unified dataset read platform for multiple current and future training frameworks. Define dataset APIs, storage-format expectations, registration/versioning, and migration paths that make data access reproducible and maintainable. Build reliability into the read path, including stateful iteration, caching, fast restart, recovery, and clear operational contracts. Build terminal and web-based visualizers that let teams inspect text, multimodal, and reinforcement learning data late