Cluster Lead Facilities Services Jobs

NVIDIA is seeking a Senior System Architect: Heterogeneous EDA Systems to solve a complex challenge in accelerated computing: Failure Attribution at Scale. As EDA or equivalent experience workloads scale across thousands of heterogeneous nodes, a single failure can cause massive resource waste. We need an engineer to develop and build an automated framework. This framework will ingest telemetry from CPU and GPU clusters to identify the root cause of job failures in real-time. It will distinguish between hardware faults, infrastructure instability, and software defects. What you'll be doing: Architect Failure Attribution Frameworks: Build a scalable "flight recorder" for EDA jobs that captures high-fidelity state across the CPU, GPU, and Fabric at the moment of failure. Build automated diagnostics that correlate GPU XID errors, PCIe bus failures, and CUDA memory exceptions. Connect these errors with system-level events such as OOM kills or NUMA-related hangs. Distributed Logging & Tracing: Implement low-overhead tracing mechanisms (using tracing tools or custom agents) that provide access to job execution across multi-node Slurm or Kubernetes clusters. Root Cause Automation: Develop heuristics and models based on machine learning to classify failures as "Hardware Fault," "Software Bug," or "Environment Issue." This reduces the Mean Time to Identify (MTTI) for R&D teams. Resiliency Engineering: Work closely with hardware and infrastructure teams to define "signals of impending failure," enabling proactive job migration or check-pointing before a crash occurs. What we need to see: Distributed Systems Mastery: BS, MS, or PhD in Computer Science or Electrical Engineering (or equivalent experience) with 6+ years in systems programming. Experience building automated

Key Responsibilities : Primary responsibilities :- Installation and configuration of MySQL instances on single or multiple ports. Hands-on experience of working with MysQL 5.7 and MySQL 8. Clear understanding of MysQL Replication process flows , threads , setting up multi node clusters and basic troubleshooting. Understanding of at least one of the backup and recovery methods for MySQL . Strong fundamentals of SQL and able to understand and tune complex SQL queries when needed. Strong fundamentals on the linux system side and monitoring tools like top , iostats , sar etc. At Least couple of years of production hands on experience on medium to big sized MySQL databases. Setting up and maintaining users and privileges management system and troubleshooting relevant access issues. Some exposure to external tools like Percona , ProxySQL , HAP etc. Understand the transaction flows and ACID compliance. Basic understanding of networking concepts . Performing on-call support and should be able to provide the first level support . Excellent verbal and written communication skills. Strong shell scripting skills . Good to have Python . Secondary responsibilities. :- Able to configure and setup NOSQL databases like Mongodb and Cassandra. Ability to learn new technologies along with a team and a positive outlook to understand problems from the business point of view. Qualifications: Proficiency in database management systems such as , MySQL or NoSQL databases. SQL programming and database design skills. Knowledge of database performance tuning and optimization techniques. Familiarity with database security best practices. Scripting and automation skills (Good to have- Python). Good problem-solving and analytical skills. Excellent communication and teamwork skills.

Key Responsibilities : Primary responsibilities :- Installation and configuration of MySQL instances on single or multiple ports. Hands-on experience of working with MysQL 5.7 and MySQL 8. Clear understanding of MysQL Replication process flows , threads , setting up multi node clusters and basic troubleshooting. Understanding of at least one of the backup and recovery methods for MySQL . Strong fundamentals of SQL and able to understand and tune complex SQL queries when needed. Strong fundamentals on the linux system side and monitoring tools like top , iostats , sar etc. At Least couple of years of production hands on experience on medium to big sized MySQL databases. Setting up and maintaining users and privileges management system and troubleshooting relevant access issues. Some exposure to external tools like Percona , ProxySQL , HAP etc. Understand the transaction flows and ACID compliance. Basic understanding of networking concepts . Performing on-call support and should be able to provide the first level support . Excellent verbal and written communication skills. Strong shell scripting skills . Good to have Python . Secondary responsibilities. :- Able to configure and setup NOSQL databases like Mongodb and Cassandra. Ability to learn new technologies along with a team and a positive outlook to understand problems from the business point of view. Qualifications: Bachelor's degree in Computer Science, Information Technology, or a related field (or equivalent experience). Proficiency in database management systems such as , MySQL or NoSQL databases. SQL programming and database design skills. Knowledge of database performance tuning and optimization techniques. Familiarity with database security best practices. Scripting and automation skills (Good to have- Python). Good problem-solving and analytical skills. Excellent communication and teamwork ski

About the Role At Together AI, you’ll build and operate one of the world’s largest GPU fleets used for frontier model training and inference. This isn’t a traditional infrastructure role—we’re looking for engineers who love building systems, automating everything, and solving problems at massive scale. If you enjoy writing software more than clicking dashboards, obsess over eliminating manual work, and want to build infrastructure that manages tens of thousands of GPUs autonomously, we’d love to talk. Responsibilities Design and build fleet automation systems that provision, validate, deploy, upgrade, repair, and retire GPU clusters with minimal human intervention. Build AI Infrastructure Agents that automate deployment, root-cause failures, incident triage, and autonomous remediation. Develop Fleet Intelligence platforms that continuously monitor hardware health, firmware, networking, storage, thermals, and workload performance to predict failures before they impact customers. Build software that maximizes GPU availability, utilization, performance, and reliability across thousands of accelerators. Create automated validation systems for GPUs, InfiniBand/RoCE fabrics, NVLink/NVSwitch, storage, and distributed AI workloads. Build internal platforms and developer tools that allow infrastructure to be managed through software—not manual operations. Continuously improve deployment velocity, reliability, and operational efficiency through automation. Partner closely with hardware, networking, platform, and AI teams to push the limits of AI infrastructure. Requirements 3+ years building distributed systems, infrastructure platforms, or large-scale backend software. Strong software engineering skills in Python, Go, or Rust . Experience building platforms, automation systems, or developer infrastructure. Experience with Linux, Kubernetes, Terraform, Ansible, or similar infrastructure technologies. Strong systems thinking with the ability to understand problems across hardw

Tenstorrent is leading the industry on cutting-edge AI technology, revolutionizing performance expectations, ease of use, and cost efficiency. With AI redefining the computing paradigm, solutions must evolve to unify innovations in software models, compilers, platforms, networking, and semiconductors. Our diverse team of technologists have developed a high performance RISC-V CPU from scratch, and share a passion for AI and a deep desire to build the best AI platform possible. We value collaboration, curiosity, and a commitment to solving hard problems. We are growing our team and looking for contributors of all seniorities. Tenstorrent is building the world’s fastest, most efficient AI compute clusters. TT-Fabric is the high-performance nervous system of this platform: the low-level networking layer that lets thousands of RISC-V and AI processors snap together into a single, massively parallel distributed supercomputer. If you love squeezing nanoseconds out of hot paths, designing protocols that move data at absurd scale, and turning messy hardware constraints into elegant distributed systems, this is an opportunity to shape the fabric that future AI models will run on This role is hybrid based out of Santa Clara, CA; Austin, TX; or Toronto, ON. We welcome candidates at various experience levels for this role. During the interview process, candidates will be assessed for the appropriate level, and offers will align with that level, which may differ from the one in this posting. Who We Are Strong systems engineer with deep C or C++ experience and comfort working in low-level or bare-metal environments. Passionate about hardware-software interaction, performance tuning, and eliminating inefficiencies at the protocol level. Curious about networking, synchronization, and communication across large clusters. Comfortable reasoning from first principles and challenging industry conventions. Motivated by building infrastructure that directly impacts large-scale

About the Team The Storage organization builds and operates the online stateful systems and abstractions that DoorDash Engineering depends on: reliable, efficient, secure, and easy to use. Within Storage, the Distributed Caching team owns every caching offering at DoorDash end to end, including ElastiCache (Redis/Valkey), Boulder (our KVRocks-based key-value store for high-QPS feature serving), Entity Cache (read Bill Shen’s engineering blog post, “ High-Performance Proxy Cache for DoorDash Services ”), and the Distributed Lock Service, plus the smart clients (asgard-redis, valkey-go) that sit in front of them. These systems back critical product surfaces across DoorDash, Wolt, and Deliveroo: the team runs roughly 400 ElastiCache clusters serving hundreds of millions of GET requests per second in aggregate, and Boulder, our offline-to-online feature store, serves billions of feature lookups per second at peak. About the Role The team owns provisioning of clusters and the smart clients that sit in front of them, baking in sensible defaults so that other engineering teams get a turnkey caching solution instead of having to run their own. You'll help drive Boulder's evolution to scale further, improve cost efficiency, enhance performance, and support real-time updates; re-platform the Distributed Lock Service onto a strongly consistent backend; and build the self-serve tooling and recommendation engine that let customers describe a workload (QPS, TTL, payload size, latency profile) and get the right backend without talking to a human. You'll go deep on cache invalidation, replication, sharding, compaction, and failover, while shipping the guardrails, automation, and observability that keep this scale operable by a small team. You must be located in San Francisco, Seattle, or the New York Metro Area for this hybrid position. You will report to the Engineering Manager on the Distributed Caching team within the Storage organization. You’re excited about this opportunity b

About the Team The DoorDash Research Fellowship is a 3-month program (extendable to 6 months) looking for Summer and Fall 2026 cohorts, for researchers and engineers who want to work on the hardest applied ML and AI problems in local commerce. Fellows are given the resources, autonomy, and access to real-world operational data needed to pursue ambitious research directions — with the goal of producing work that influences both the field and how DoorDash operates at scale. This program is modeled on the best external research fellowships: fellows are treated as independent researchers, not as junior employees on a product team. You pick the problem (within a set of priority areas), you own the direction, and you publish or ship the outcome. You’re excited about this opportunity because you will receive… Dedicated compute allocation sized to the research agenda — GPU clusters for training and inference budgets for experimentation Full access to DoorDash's research infrastructure — our internal RL stack, training and evaluation pipelines, RL environments built on real operational systems, agent evaluation harnesses, and the tooling our own research teams use day-to-day. Fellows are first-class users, not sandboxed visitors. Access to DoorDash operational data — real-world datasets spanning logistics, merchant operations, consumer behavior, and marketplace dynamics, under appropriate data governance Research mentorship from senior researchers and engineering leaders at DoorDash, plus a named research sponsor for each fellow who meets with you weekly and is accountable for unblocking your work Speaker series featuring leading researchers and practitioners from academia and industry — faculty from top ML programs, research leads from frontier AI labs, and senior operators from across tech. Fellows get dedicated 1:1 time with speakers when possible. A cohort of fellows working alongside you — a small, tight-knit group of researchers tackling different problems but sharing

About the Team Compute Foundations builds the software that manages OpenAI’s GPU compute infrastructure across sites, data centers, and infrastructure providers, supporting model training and inference. Our systems turn large, heterogeneous fleets of machines into dependable compute for research and products. We build Kubernetes-based control planes, controllers, services, and APIs that coordinate the lifecycle of machines and clusters. We connect global infrastructure management with the realities of bare-metal systems, giving clients consistent interfaces across differences in hardware, topology, and provider behavior. About the Role You will build distributed systems that provision, configure, and manage compute throughout its lifecycle. Your work will connect global services and Kubernetes controllers with the systems that bring machines online, update them safely, and recover them when something goes wrong. This role combines software architecture with an understanding of how machines and data centers work. You might design a lifecycle API, improve controller performance under high concurrency and provider rate limits, or trace a provisioning failure from an API through reconciliation to network boot or host configuration. You will help these systems remain reliable as the fleet expands across sites and generations of GPU hardware. We value depth in relevant systems and the ability to connect layers. You do not need to arrive as an expert in every component of the stack. In this role, you will: Design, build, and operate Kubernetes-based controllers and distributed services that coordinate infrastructure across sites, isolate failures, and scale as GPU capacity grows. Define APIs and resource models that let clients request and track lifecycle operations through consistent interfaces across hardware platforms and providers. Build provisioning and configuration services that coordinate network boot, hardware management interfaces, and the deployment of firmware,

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. Workforce Identity Cloud Okta Workforce Identity Cloud (WIC) provides easy, secure access for your workforce so you can focus on other strategic priorities—like reducing costs, and doing more for your customers. If you like to be challenged and have a passion for solving large-scale automation, testing, and tuning problems, we would love to hear from you. The ideal candidate is someone who exemplifies the ethics of, “If you have to do something more than once, automate it” and who can rapidly self-educate on new concepts and tools. Position Overview: The Site Reliability Engineer (SRE) will play a key role in building and managing Kubernetes platforms that support cloud-native applications and services. This position focuses on architecting and managing reliable, scalable, and secure Kubernetes-based platforms on AWS, ensuring high availability and performance while optimizing costs and automation. The ideal candidate will have hands-on experience with AWS infrastructure, Kubernetes platform creation, Helm charts, Karpenter scaling, and Istio service mesh. Key Responsibilities: Kubernetes Platform Creation: Design, implement, and maintain highly available, scalable, and fault-tolerant Kubernetes platforms. Ensure clusters are optimized for production workloads, providing high resilience and operational efficiency. AWS Infrastructure Management: Build, manage, and optimize AWS cloud infrastructure, including EKS,ECS, S3, VPCs, RDS, IAM, and more. Implement b

NVIDIA is looking for Senior Networking (ETH/IB) Solutions Architect to join its NVIDIA Infrastructure Specialist Team. Academic and commercial groups around the world are using NVIDIA products to revolutionize deep learning and data analytics, and to power data centers. Join the team building many of the largest and fastest AI/HPC systems in the world! We are looking for someone with the ability to work on a dynamic customer focused team that requires excellent interpersonal skills. This role will be interacting with customers, partners and internal teams, to analyze, define and implement large scale Networking projects. The scope of these efforts includes a combination of Networking, System Design and Automation and being the face to the customer! What you'll be doing: Primary responsibilities will include building AI/HPC infrastructure for new and existing customers. Support operational and reliability aspects of large-scale AI clusters, focusing on performance at scale, real-time monitoring, logging, and alerting. Engage in and improve the whole lifecycle of services—from inception and design through deployment, operation, and refinement. Maintain services once they are live by measuring and monitoring availability, latency, and overall system health. Provide feedback to internal teams such as opening bugs, documenting workarounds, and suggesting improvements. What we need to see: BS/MS/PhD or equivalent experience in Computer Science, Electrical/Computer Engineering, Physics, Mathematics, or related fields. At least 5+ years of professional experience in networking fundamentals, Ethernet or InfiniBand World. Hands-on experience with network switch/router platforms like Cumulus Linux, SONiC, IOS, JunosOS, and EOS, etc. Possess solid working knowl

We are developing advanced multi-rack, multi-tenant AI/ML datacenters with NVIDIA GB200, and upcoming GB300 GPUs. NVIDIA seeks a Senior Software Engineer for our CSP (Cloud Service Provider) Engagements team to focus on the cloud-native stack for datacenter products like GB200. In this role, You will define customer workflows, prototype stack enhancements, and debug the toughest Kubernetes + Slurm issues in multi-rack, multi-tenant AI datacenters. You'll tackle complex scheduling challenges across racks, tenants, and clouds as part of the CSP engagements team. What you’ll be doing: Perform deep-dive debugging of multi-rack, multi-tenant clusters: scheduler behavior, container runtime issues, device-plugin crashes, RDMA/IB fabric anomalies, etc. Gather customer requirements and prototype feature extensions for Kubernetes operators, Slurm plugins, and custom micro-services that expose new GPU capabilities. Drive joint architecture reviews and “whiteboard” sessions with CSP and internal platform teams; convert findings into RFCs and upstream pull requests. Create reproducible testbeds (Helm/Ansible/Terraform) that mirror customer environments; automate validation and benchmark suites. Deliver technical collateral-design docs, how-to guides, demo scripts-and present at customer on-sites, KubeCon, and SlurmUG. Collaborate with AE, FAE, and Solution Architect teams to deliver integrated customer solutions and technical documentation. What we need to see: Strong source-level expertise in Kubernetes internals (scheduler, CRI/CNI/CSI, operators) and Slurm (federation, power-save, plugins). Hands-on experience integrating next-gen GPUs (Blackwell/GB200/GB300) or comparable accelerators into containerized clusters. Proven track record debugging large-scale, cloud-native stacks across ne

We are looking for a Senior Software Engineer to become part of our storage management plane team. The management plane is a web-based application crafted to provide our storage customers the capabilities to handle and supervise our distributed storage infrastructure. Our team is continually dedicated to acquiring and implementing ground breaking technologies to overcome obstacles and innovate solutions for improving our ability to handle large clusters of machines efficiently. What You Will Be Doing: Maintain and develop Kubernetes operators and our Container Storage Interface (CSI) plugin. Develop a web-based solution that manages, operates and monitors our distributed storage. Work closely with other teams to define and implement new APIs. What We Need to See: B.Sc., M.Sc. or Ph.D. in Computer Science, or related discipline, or equivalent experience. 8+ years of experience in web development ( both client and server ) Proven experience with Kubernetes (K8s), including developing or maintaining operators and/or CSI plugins. Experience scripting with Python, Bash or similar. Experience with nodejs is a must At least 5 years of experience working in a Linux OS environment You’re smart and a quick learner You do what it takes to get the job done Passionate about coding and big challenges Ways to stand out from the crowd: NodeJS for the server side: dominant modules are async & express . Kafka, MongoDB, K8s JavaScript frameworks: React, jQuery, c3j

Biohub is the first large-scale initiative bringing frontier AI models, massive compute, and frontier experimental capabilities under one roof. We're building a general-purpose system to accelerate scientific discovery, integrating frontier AI models, biological foundation models, and lab capabilities, with the ultimate goal of curing disease. Our technology powers scientists around the world, translating AI capabilities into tools that accelerate research everywhere. Biohub operates one of the largest AI compute clusters dedicated to biology, spanning three frontier research institutes with some of the world's leading biologists. We're not a startup trying to find product-market fit, and we're not a pharma company optimizing a pipeline. We're building frontier AI for fundamental science, as open science, at a scale no one else is doing. This is a unique moment for scientific acceleration. The problems are among the hardest and most impactful problems you can choose to work on, and we move at a pace that meets this moment. Our research spans: Frontier molecular modeling, from protein language models (e.g., ESM) to structure prediction (e.g., ESMFold) and beyond. Scaled biological foundation models trained on some of the largest GPU clusters dedicated to science Imaging foundation models trained across the world's largest microscopy datasets Reasoning and agentic systems that connect frontier LLMs with biological foundation models Mechanistic interpretability of biological foundation models: extracting new biological knowledge directly from model weights Scientific data at unprecedented scale: AI systems to collect, curate, and learn from some of the richest biological datasets ever assembled Join Our Team! As a Research Engineer, you'll build the models and systems that define what AI can do in biology: foundation models, reasoning, reinforcement learning, and multi-agent systems at frontier scale. What You'll Do Build on and advance the AI systems at th

Biohub is the first large-scale initiative bringing frontier AI models, massive compute, and frontier experimental capabilities under one roof. We're building a general-purpose system to accelerate scientific discovery, integrating frontier AI models, biological foundation models, and lab capabilities, with the ultimate goal of curing disease. Our technology powers scientists around the world, translating AI capabilities into tools that accelerate research everywhere. Biohub operates one of the largest AI compute clusters dedicated to biology, spanning three frontier research institutes with some of the world's leading biologists. We're not a startup trying to find product-market fit, and we're not a pharma company optimizing a pipeline. We're building frontier AI for fundamental science, as open science, at a scale no one else is doing. This is a unique moment for scientific acceleration. The problems are among the hardest and most impactful problems you can choose to work on, and we move at a pace that meets this moment. Our research spans: Frontier molecular modeling, from protein language models (e.g., ESM) to structure prediction (e.g., ESMFold) and beyond. Scaled biological foundation models trained on some of the largest GPU clusters dedicated to science Imaging foundation models trained across the world's largest microscopy datasets Reasoning and agentic systems that connect frontier LLMs with biological foundation models Mechanistic interpretability of biological foundation models: extracting new biological knowledge directly from model weights Scientific data at unprecedented scale: AI systems to collect, curate, and learn from some of the richest biological datasets ever assembled Join Our Team! As a Research Scientist, you'll build the models and systems that define what AI can do in biology: foundation models, reasoning, reinforcement learning, and multi-agent systems at frontier scale. What You'll Do Build on and advance the AI systems at t

Biohub is the first large-scale initiative bringing frontier AI models, massive compute, and frontier experimental capabilities under one roof. We're building a general-purpose system to accelerate scientific discovery, integrating frontier AI models, biological foundation models, and lab capabilities, with the ultimate goal of curing disease. Our technology powers scientists around the world, translating AI capabilities into tools that accelerate research everywhere. Biohub operates one of the largest AI compute clusters dedicated to biology, spanning three frontier research institutes with some of the world's leading biologists. We're not a startup trying to find product-market fit, and we're not a pharma company optimizing a pipeline. We're building frontier AI for fundamental science, as open science, at a scale no one else is doing. This is a unique moment for scientific acceleration. The problems are among the hardest and most impactful problems you can choose to work on, and we move at a pace that meets this moment. Our research spans: Frontier molecular modeling, from protein language models (e.g., ESM) to structure prediction (e.g., ESMFold) and beyond. Scaled biological foundation models trained on some of the largest GPU clusters dedicated to science Imaging foundation models trained across the world's largest microscopy datasets Reasoning and agentic systems that connect frontier LLMs with biological foundation models Mechanistic interpretability of biological foundation models: extracting new biological knowledge directly from model weights Scientific data at unprecedented scale: AI systems to collect, curate, and learn from some of the richest biological datasets ever assembled Join Our Team! As a Research Engineer, you'll build the models and systems that define what AI can do in biology: foundation models, reasoning, reinforcement learning, and multi-agent systems at frontier scale. What You'll Do Build on and advance the AI systems at th