About the Team OpenAI, in close collaboration with our capital partners, is building the world’s most advanced AI infrastructure ecosystem. Our Industrial Compute organization develops and deploys large-scale AI campuses designed to support the next generation of frontier model training and inference workloads. The Hardware Operations team is responsible for ensuring the reliability, availability, and lifecycle health of OpenAI’s compute infrastructure. We partner closely with Data Center Operations, Fleet Health Engineering, Manufacturing, Network Infrastructure, Capacity Planning, and our infrastructure partners to maintain world-class operational performance across rapidly expanding AI environments. As we scale globally, we are building the operational frameworks, reliability standards, and sustaining engineering practices required to support thousands of GPUs and servers across multiple campuses. About the Role We are seeking a Datacenter Hardware Technician Lead to serve as the senior on-site technical authority for hardware reliability and fleet health at one of OpenAI’s flagship AI campuses. This role operates at the intersection of hardware operations, sustaining engineering, and fleet reliability. You will partner closely with Cloud Service Provider operations teams, OpenAI fleet-health engineers, hardware engineering teams, and OEM vendors to identify, diagnose, and resolve hardware issues affecting production systems. Beyond day-to-day operational support, you will drive root cause investigations, reliability improvement initiatives, lifecycle management programs, and operational readiness efforts. You will help establish hardware maintenance standards, operational procedures, and best practices that scale across future OpenAI infrastructure deployments. The ideal candidate combines deep hands-on datacenter hardware expertise with strong troubleshooting, failure analysis, and cross-functional leadership skills. Candidates must be able to sit onsite at our
Jobs in United States
Training Performance Engineer in United States
1,453 active opportunities · Updated October 2026
Showing
15 jobs
Explore current training performance engineer jobs across United States. Filter by work mode, employment type, experience, department, date posted and distance.
About the Team We’re hiring software engineers to make OpenAI’s networking teams more productive. These teams build and operate the high-performance networking systems that support OpenAI’s training and inference infrastructure at frontier scale. About the Role We’re looking for someone who cares deeply about the developer experience of engineers working on complex infrastructure systems — especially around build systems, test architecture, release pipelines, and reliable development workflows. This role will be embedded with OpenAI’s networking team: making it faster, safer, and easier for engineers to build, test, validate, and ship changes across multi-server, networked, and hardware-adjacent environments. In this role you will: Improve development workflows for engineers building and operating OpenAI’s networking systems Design and improve continuous deployment, release, and validation pipelines Build and maintain test harnesses for multi-server, networked, and hardware-backed environments Improve iteration speed across C++, Python, and build-system-heavy codebases Partner with engineers to identify friction in CI, testing, debugging, and deployment workflows Drive testing and reliability strategy for infrastructure components that support large-scale training and inference workloads Work closely with centralized developer experience teams while staying deeply embedded with the networking engineers closest to the systems You might thrive in this role if: You are motivated by helping other engineers move faster and with more confidence You have experience with CI/CD, release pipelines, testing infrastructure, or build systems You are comfortable moving between C++, Python, and build systems such as CMake, Bazel, or Blaze You enjoy building test harnesses, automation, and workflow improvements for complex systems You do not need to be a networking expert, but you are excited to learn enough about the domain to make the team meaningfully more effective When you see
Abbott is a global healthcare leader that helps people live more fully at all stages of life. Our portfolio of life-changing technologies spans the spectrum of healthcare, with leading businesses and products in diagnostics, medical devices, nutritionals and branded generic medicines. Our 122,000 colleagues serve people in more than 160 countries. JOB DESCRIPTION: Position Overview The AI Platform Engineer builds and operates the machine learning and generative AI platform used by teams across Abbott Cancer Diagnostics. You'll own the full model lifecycle in production — data and feature pipelines, training and experimentation, evaluation and promotion, serving, and monitoring — along with the platform services, compute and tooling underneath it. This is hands-on infrastructure work backed by solid platform engineering practice: making inference fast and cheap, making the path from experiment to production repeatable and auditable, and shipping interfaces other engineers can build on — in support of software that ultimately reaches patients. Essential Duties Include, but are not limited to, the following: Build and maintain data, feature, and training pipelines for ML and LLM workloads — ingestion, transformation, fine-tuning, distributed training, and reproducible experiment execution with lineage tracked from dataset and code to resulting model. Implement automated evaluation and promotion gates — performance benchmarks, regression checks, and validation criteria that determine whether a model advances toward production. Automate the model lifecycle end to end through CI/CD and GitOps: packaging, promotion across environments, progressive rollout, and rollback. Build and operate production model-serving infrastructure for LLMs and predictive models, including inference optimization, autoscaling,
Associate Systems Engineering Support Analyst Company: The Boeing Company Boeing Defense, Space & Security (BDS has an exciting opportunity for an Associate Systems Engineering Analyst (Level 2) for Flight Simulation Labs , located in Berkeley, MO . The Flight Simulation Labs organization has expertise supporting marketing, design, development, integration, test and training objectives for Boeing military aircraft. As a Systems Engineering Analyst, you will be responsible for researching, analyzing, and compiling technical data to optimize system performance throughout the program lifecycle. You will apply systems engineering processes, methodologies, and tools to develop and refine system architectures, functional requirements, and product designs for programs such as T-7, F-15, F-18, and MQ-25. Position Responsibilities Apply systems engineering methodologies to design, develop, and maintain systems, tools, and new product solutions Serves as system administrator for Engineering tools in the Flight Simulation Labs organization Identifies opportunities and implements data analysis and automation including dashboard and reporting for the organization Derives and develops basic architecture, functional requirements, and product designs, ensuring alignment with project objectives Collaborates with cross functional project teams including engineering, IT, cyber, networking, software, customers, and suppliers to achieve organizational goals Engages in continuous improvement initiatives for all Flight Simulation Labs processes and tools This position requires the ability to obtain a U.S. Security Clearance for which the U.S. Government re
NVIDIA is leading the way in groundbreaking developments in Artificial Intelligence, High Performance Computing and Visualization. The GPU, our invention, serves as the visual cortex of modern computers and is at the heart of our products and services. Our work opens up new universes to explore, enables amazing creativity and discovery, and powers what were once science fiction inventions from artificial intelligence to autonomous cars. We are looking for a motivated Deep Learning engineer to bring advanced communication technologies into AI stacks, including PyTorch, TRT-LLM, vLLM, SGLang, JAX, etc. You will be working with the team that created communication libraries like NCCL, NVSHMEM & technology like GPUDirect -- for scaling Deep Learning and HPC applications. Your customers will have diverse multi-GPU demands, ranging from training on scales up to 100K GPUs to inference down at microsecond latency. Communication performance between the GPUs has a direct impact on AI applications. Your work in AI toolkits will make all of those easier for the community. This is an outstanding opportunity for someone with an AI background to advance the state of the art in this space. Are you ready to contribute to the development of innovative technologies and help realize NVIDIA's vision? What you will be doing: Integrate new communication libraries features in AI frameworks: from PoC to performance analysis to production Perform deep analysis of AI workloads and frameworks to identify multi-GPU communication requirements and opportunities. Collaborate hands-on with teams working on the latest AI models. Improve AI compilers to hide communications or perform automatic fusion. Conduct in-depth AI workload performance characterization on multi-GPU clusters. Design fault-tolerant and elastic solutions for large-scale or dynamic AI workloads. Author
Who are we? Cohere is the leading security-first enterprise AI company. We build cutting-edge foundation AI models and end-to-end products that are designed to solve real-world business problems. We’re training and deploying frontier models for enterprises who are building AI systems. We believe that our work is instrumental to the widespread adoption of AI and we are looking for folks that want to be part of that. We obsess over what we build. Each one of us is responsible for contributing to increasing the capabilities of our models and the value they drive for our customers. Cohere is a team of researchers, engineers, designers, and more, who are all passionate about their craft. We are a global technology company headquartered in Toronto with key offices in London, New York City, San Francisco, Montreal, Paris, Berlin and Seoul. Join us! Why this role? Large Language Models (LLMs) continue to push the boundaries of what AI systems can do — but inference is still the bottleneck. The Model Efficiency team is responsible for pushing the limits of LLM inference efficiency across our foundation models. We explore and ship breakthroughs across the model execution stack, including: model architecture and MoE routing optimization decoding and inference-time algorithm improvements software/hardware co-design for GPU acceleration performance optimization without compromising model quality Please Note: We have offices in Toronto, Montreal, San Francisco, New York, Paris, Seoul and London. We embrace a remote-friendly environment, and as part of this approach, we strategically distribute teams based on interests, expertise, and time zones to promote collaboration and flexibility. You'll find the Model Efficiency team concentrated in the EST and PST time zones, these are our preferred locations. As a Staff Research Engineer, you will develop, prototype, and deploy techniques that materially improve how fast and efficiently our models run in production. You may be a good fit
Who are we? Cohere is the leading security-first enterprise AI company. We build cutting-edge foundation AI models and end-to-end products that are designed to solve real-world business problems. We’re training and deploying frontier models for enterprises who are building AI systems. We believe that our work is instrumental to the widespread adoption of AI and we are looking for folks that want to be part of that. We obsess over what we build. Each one of us is responsible for contributing to increasing the capabilities of our models and the value they drive for our customers. Cohere is a team of researchers, engineers, designers, and more, who are all passionate about their craft. We are a global technology company headquartered in Toronto with key offices in London, New York City, San Francisco, Montreal, Paris, Berlin and Seoul. Join us! Why this role? Are you energized by building high-performance, scalable and reliable machine learning systems? Do you want to help define and build the next generation of AI platforms powering advanced NLP applications? We are looking for Members of Technical Staff to join the Model Serving team at Cohere. The team is responsible for developing, deploying, and operating the AI platform delivering Cohere's large language models through easy to use API endpoints. In this role, you will work closely with many teams to deploy optimized NLP models to production in low latency, high throughput, and high availability environments. You will also get the opportunity to interface with customers and create customized deployments to meet their specific needs. You may be a good fit if you have: 5+ years of engineering experience running production infrastructure at a large scale Experience designing large, highly available distributed systems with Kubernetes, and GPU workloads on those clusters Experience with Kubernetes dev and production coding and support Experience with GCP, Azure, AWS, OCI, multi-cloud on-prem / hybrid serving Experienc
What you’ll do Partner with medical image reconstruction scientists / engineers to build ML components that improve reconstruction quality, speed, robustness, or quantitative accuracy. Define training/evaluation pipelines, datasets, and metrics that map to user needs and design requirements. Productionize models: inference performance, reproducibility, monitoring for drift/regressions, and safe fallbacks. Collaborate on hybrid algorithms, incorporating physics and learned priors, denoisers, learned regularizers, and quality estimation. Help build tooling for rapid experimentation as well as rigorous verification of algorithm changes. What we’re looking for Strong applied ML experience plus comfort with signal processing / imaging or adjacent domains. Ability to move fluidly between research prototypes and production-quality systems. Strong evaluation discipline: metrics, ablations, data leakage avoidance, and reproducibility. A demonstrated track record of applying ML to physics-based or inverse problems (i.e., shipped projects, a portfolio, or publications.) Useful experience ML for imaging/inverse problems (or adjacent) with strong evaluation discipline and comfort with GPU performance constraints. Pragmatic production mindset: reproducible training/inference, regression testing, and safe deployment in high-stakes contexts. A background in computational physics or scientific computing. Leverage ML-based methods such as PiNNs and Neural Operators to solve partial differential equations arising in ultrasound simulation and imaging. Experience in Agentic-SciML is a plus. Hands-on experience with data curation for ML: building datasets from messy, real-world sources, defining ground truth, and managing labeling or simulation pipelines. Background in data assimilation: combining observations with physics-based models (Kalman filtering, variational methods, ensemble approaches, or learned variants).
Overview: The Data Acquisition team within the Foundations organization at OpenAI is responsible for all aspects of data collection to support our model training operations. Our team manages web crawling and GPTBot services and works closely with Data Processing, Architecture, and Scaling teams. We are looking for a skilled Software Engineer to join our Data Acquisition team. Responsibilities: Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search. Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability. Work closely with the legal team to handle any compliance or data privacy-related matters. Develop and deploy highly scalable distributed systems capable of handling petabytes of data. Architect and implement algorithms for data indexing and search capabilities. Build and maintain backend services for data storage, including work with key-value databases and synchronization. Deploy solutions in a Kubernetes Infrastructure-as-Code environment and perform routine system checks. Conduct and analyze experiments on data to provide insights into system performance. Qualifications: BS/MS/PhD in Computer Science or a related field. 4+ years of industry experience in software development. Experience with large web crawlers a plus Strong expertise in large stateful distributed systems and data processing. Proficiency in Kubernetes, and Infrastructure-as-Code concepts. Willingness and enthusiasm for trying new approaches and technologies. Ability to handle multiple tasks and adapt to changing priorities. Strong communication skills, both written and verbal. About OpenAI OpenAI is an AI research and deployment company dedicated to ensuring that general-purpose artificial intelligence benefits all of humanity. We push the boundaries of the capabilities of AI systems and seek to safely deploy them to the world through our products. AI is an
This role will support the fleet infrastructure team at OpenAI. The fleet team focuses on running the world’s largest, most reliable, and frictionless GPU fleet to support OpenAI’s general purpose model training and deployment. Work on this team ranges from Maximizing GPUs doing useful work by building user-friendly scheduling and quota systems Running a reliable and low maintenance platform by building push-button automation for kubernetes cluster provisioning and upgrades Supporting research workflows with service frameworks and deployment systems Ensuring fast model startup times though high performance snapshot delivery across blob storage down to hardware caching Much more! About the Role As an engineer within Fleet infrastructure, you will design, write, deploy, and operate infrastructure systems for model deployment and training on one of the world’s largest GPU fleet. The scale is immense, the timelines are tight, and the organization is moving fast; this is an opportunity to shape a critical system in support of OpenAI's mission to advance AI capabilities responsibly. This role is based in San Francisco, CA. We use a hybrid work model of 3 days in the office per week and offer relocation assistance to new employees. In this role, you will: Design, implement and operate components of our compute fleet including job scheduling, cluster management, snapshot delivery, and CI/CD systems. Interface with researchers and product teams to understand workload requirements Collaborate with hardware, infrastructure, and business teams to provide a high utilization and high reliability service You might thrive in this role if you: Have experience with hyperscale compute systems Possess strong programming skills Have experience working in public clouds (especially Azure) Have experience working in Kubernetes Execution focused mentality paired with a rigorous focus on user requirements As a bonus, have an understanding of AI/ML workloads About OpenAI OpenAI is an AI resea
Our vision is to transform how the world uses information to enrich life for all . Micron Technology is a world leader in innovating memory and storage solutions that accelerate the transformation of information into intelligence, inspiring the world to learn, communicate and advance faster than ever. This Notice is provided as a result of the filing of applications for permanent alien labor certification for the positions noted below. Any person may provide documentary evidence bearing on an application to the Certifying Officer, U.S. Department of Labor, Employment and Training Administration, Office of Foreign Labor Certification, 200 Constitution Avenue NW, Room N–5311, Washington, DC 20210. Employer: Micron Technology, Inc. Job Title: Firmware Engineer Job Requisition: JR42883 Job Location: Longmont, Colorado Job Type: Full Time Rate of Pay: $100,000 to $254,000 Duties: Develop high-performance controller firmware for innovative volatile and non-volatile memory systems. Assist in evaluating, designing, building, bench testing, debugging, and failure analyzing firmware for new high-performance memory controllers and Solid State Drives that will increase performance, while reducing power, latency and SoC (System on Chip) complexity for the target markets. Work closely with system test engineers, system failure analysis engineers, firmware engineers, firmware test engineers, system hardware design engineers, product engineers, memory design engineers, firmware tools developers, and production engineers to solve cross-functional product development issues. Validate customer requirements and perform failure analysis on FW related issues. Develop structure
Aerodynamics Engineer (Aero-Stability & Control Analyst) Company: The Boeing Company Boeing Defense, Space & Security (BDS) is hiring an Experienced or Senior Aerodynamics Engineer to perform propulsion system qualification support and analysis in the St. Louis, MO region ! Position Responsibilities: Performs aerodynamic/aerothermodynamics design and analysis of flight vehicles to ensure that the vehicle meets requirements Key contributor to developing new and improving existing technologies, tools and processes to enhance technology readiness, improve vehicle performance and reduce cycle time and cost Conducts, coordinates and/or develops test requirements to meet test objectives using established processes Assesses the impact of proposed regulations to determine the effect on cost, performance, producibility, maintainability and safety Supports certification and qualification processes of the aerodynamic/ aero thermal characteristics of flight vehicles Assists in training others Works under general supervision Basic Qualifications (Required Skills and Experience): Bachelor of Science degree in Engineering, Engineering Technology (including Manufacturing Technology), Computer Science, Data Science, Mathematics, Physics, Chemistry or non-US equivalent qualifications directly related to the work statement 5+ years work-related experience with a Bachelors degree OR an equivalent combination of work-related experience and technical education. Preferred Qualifications (Desired Skills and Experience): (Level 4) 9+ years work-related experience w
NVIDIA is a global leader in high-speed computer vision, artificial intelligence (AI), and deep learning. Our team develops data engineering solutions that empower AI developers in autonomous vehicle (AV) domains to innovate quickly and effectively at scale. Are you ready to take on a senior technical role in building high-performance AI data pipelines? We seek an exceptional individual to design and optimize microservices and data pipelines to process massive volumes of AV data and enable seamless data mining and AI training. The ideal candidate will bring expertise in big data processing and distributed computing to create efficient solutions and overarching architectures for challenges such as video data curation, behavioral search, and AI dataset management. What you'll be doing: Scope and build tools, microservices, workflows, and distributed applications to accelerate data mining and AI training. Design and implement solutions for streaming, resilience, logging, security, authentication, workflow orchestration, and data management. Deploy AI models. Design and develop Retrieval-Augmented Generation (RAG) workflows enabling hybrid and agentic patterns. Analyze and operationalize complex distributed systems for speed-of-light performance. What we need to see: Experience developing high-performance, scalable software systems. MS with 6+ years, or BS (or equivalent experience) with 8+ years of relevant experience in Computer Science, Computer Engineering, or a related technical field. Strong programming skills in Python or Golang Proficiency in key technologies like Kubernetes, Helm, Hive, Parquet, SQL, vector databases, e.g., Milvus. Strong architectural skills with a proactive, problem-solving mentality. Experience in data mi
We are looking for a Senior System Software Engineer, Software Defined Networking to design, build, and operate highly performant and scalable SDN solutions for NVIDIA's AI Clouds hosting GPU-accelerated workloads — including hyperscale multi-node training, inference, cloud gaming, and cloud functions. This role spans the full lifecycle of our SDN stack — from designing and developing new control and data plane software to ensuring operational excellence in production through reliability engineering, CI/CD, observability, and incident response. What you'll be doing: Design and develop next-generation multi-tenant cloud SDN control and data plane software (OVS, OVN, OpenFlow) Build Infrastructure-as-a-Service virtual network orchestration and services using gRPC and REST to support tenant workload security and performance SLAs for BMaaS, VMaaS, and Kubernetes Drive upstream contributions to OVN-Kubernetes and related open-source projects Develop software for network observability — monitoring, telemetry, intelligent metering, and performance analysis Operate and support OVS-OVN based SDN solutions in large-scale NVIDIA AI Cloud environments Own end-to-end observability for the SDN stack — build and maintain monitoring, alerting, distributed tracing, and dashboarding to ensure real-time insight into network health, performance, and tenant SLAs Design, enhance, and maintain CI/CD pipelines (GitLab) across Linux host networking, OVS, OVN, and Kubernetes CNIs Implement GitOps approaches or related experience for secure, seamless integration with cloud infrastructure Drive reliability through incident management, resource monitoring, and performance tuning<
NVIDIA is hiring an NCX Senior Engineer who is passionate about NVIDIA Cloud Partner (NCP) infrastructure operations to join our DSX team. This role involves working closely with strategic NVIDIA Cloud Partners to build and improve the operational capabilities essential for running large-scale NVIDIA accelerated infrastructure reliably in production. Your role involves guiding partners beyond the initial cluster deployment and validation phase into advanced Day 2 operations. These operations cover ongoing infrastructure health, observability, lifecycle management, quick remediation, performance validation, and operational readiness. You will engage directly with partner engineering and operations teams to develop consistent approaches that support NVIDIA workloads and the broader external customer environments of the partners. This is a highly technical, hands-on role at the intersection of NVIDIA accelerated computing, cloud infrastructure, distributed systems, and production operations. What you'll be doing: Lead NCP Day 2 operational readiness efforts. Collaborate directly with NVIDIA Cloud Partners to set up the systems, procedures, automation, and operational methods necessary to consistently manage NVIDIA accelerated infrastructure following initial deployment and activation. Build continuous infrastructure validation. Develop and implement methods to continuously validate GPU, CPU, storage, and network health. Do this across large-scale AI clusters to identify degraded infrastructure before it impacts critical training or inference workloads. Establish observability and operational telemetry. Help NCPs implement comprehensive telemetry, monitoring, alerting, dashboards, and operational signals across compute, GPU, InfiniBand/RoCE networking, storage, Kubernetes, and AI workloads. Devel
Other cities to consider
More places hiring for this role
Get new training performance engineer jobs in United States by email
Daily job updates · Unsubscribe anytime