Performance Engineer, Inference Engine

ABOUT BASETEN Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F , led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to ship AI products. THE ROLE We're looking for inference performance engineers who want to make the world's most demanding AI workloads run faster and more efficiently. You'll work across the stack, from the inference engine and runtime through scheduling, serving, and routing. Along the way you'll apply techniques like prefill/decode disaggregation, speculative decoding, and KV-cache management. You'll reason from first principles about where time and memory go, find what's holding performance back, and close the gap. Your work directly impacts how fast our customers' models run and how efficiently we serve them. This role is ideal for someone who thrives in a fast-paced startup environment and is eager to make significant contributions to the exciting field of LLM inference. EXAMPLE INITIATIVES You'll get to work on these types of projects as an Inference Performance engineer: Agentic Kernels in Production How we built the new fastest API for GLM-5.2 Live draft model training for speculative decoding Making Kimi K3 Tokenization 18x faster The Baseten Inference Stack Driving model performance optimization RESPONSIBILITIES Implement and productionize cutting-edge inference techniques, working deep in runtime internals. That includes quantization, speculative decoding, KV-cache reuse, chunked prefill, LoRA, guided generation for structured outputs, and custom scheduling and routing algorithms. Profile and optimize inference end to end, from

About the Team We’re hiring a Developer Productivity engineer to support OpenAI’s Inference Runtime teams. These teams own the systems responsible for serving models reliably, efficiently, and safely across Codex, ChatGPT, API, and internal research workloads. We’re hiring a Developer Productivity Engineer to help scale the engineering systems, safeguards, and developer workflows that enable our teams to move quickly without compromising reliability or performance. This role sits at the intersection of developer experience, CI/CD infrastructure, release engineering, production readiness, and inference systems reliability. You’ll work on the tooling and operational foundations that support model launches, inference optimizations, cloud provider integrations, and large-scale deployments across a rapidly evolving inference stack. About the Role We’re looking for an autonomous, high-ownership engineer who cares deeply about making other engineers faster, safer, and more confident. A major focus of this role will be improving the tooling and infrastructure around deploy gates for inference engine images. These systems help ensure that every image released to production and research is correct, numerically sound, free of regressions, and performant across key metrics like time-to-first-token (TTFT) and time-between-tokens (TBT). You’ll help harden the systems that catch issues before they reach production, reduce noise from flaky or infrastructure-related test failures, and improve automation around triage, ownership, debugging, and escalation when failures occur. You’ll also work on improving observability, rollout safety, release automation, and developer self-service tooling across a rapidly evolving inference stack. This is not generic internal tools work. The systems you build directly impact OpenAI’s ability to support new model launches, safely ship inference optimizations to the world, onboard new infrastructure providers, and operate one of the largest and most p

We are looking for an AI Inference Engineer to join our growing team. We build and run the inference engine behind every Perplexity query and deploy dozens of model architectures at scale with tight latency and cost budgets. Our stack is Rust, Python, CUDA, and CuTe DSL. Responsibilities: New models support. Support transformer-based retrieval, text-generation, and multimodal models in our inference infrastructure, from weight loading, request scheduling and KV-cache management to support in API Gateway. GPU kernels migration to CuTe DSL. Port our in-house CUDA kernels to NVIDIA's CuTe DSL so they run on GB200 today and are portable to Vera Rubin racks tomorrow. Rust-native serving runtime. Develop our internal Rust-based inference server to solve all Python pains and keep up with rapidly growing traffic. Performance optimisation. Profile and fix bottlenecks from network ingress through continuous batching and GPU kernels interleaving. Reliability and observability. Build dashboards, alerts, and automated remediation so we catch regressions before users do. Respond to and learn from production incidents. Who we're looking for: Deep experience with GPU programming and performance work (CUDA, Triton, CUTLASS, or similar). Any other deep systems programming experience is a plus. You understand modern LLM architectures and are able to bring them up reliably in a production environment. You've built and operated production distributed systems under real load - ideally performance-critical ones. Comfortable working across languages and layers: Rust for the serving runtime, Python for model code, CUDA/CuteDSL for kernels. You own problems end-to-end. You can read a research paper on Monday, write a kernel on Wednesday, and debug a production incident on Friday. Self-directed. You do well in fast-moving environments where the path forward isn't laid out for you. Nice-to-have: ML compilers and framework internals: PyTorch internals, torch.compile, custom operators. Distribut

ABOUT BASETEN Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F , led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to ship AI products. THE ROLE We're looking for distributed systems engineers and product-minded generalists to build the distributed runtime that powers large-scale LLM inference on Baseten. Our inference platform empowers customers to deploy and operate cutting-edge models with industry-leading performance, scalability, and reliability. It also powers Model APIs, our hosted endpoints for the latest open-source models. You'll work across the stack, from the developer experience customers use to deploy models, through the libraries behind features like tool calling and reasoning, down to the systems that orchestrate deployments on Kubernetes and route traffic efficiently. Your job is to make sure every model on our platform is fast, reliable, and cost-efficient. You'll join a small, high-impact team at the intersection of distributed systems, model performance, infrastructure, and product, helping define how developers use AI models at scale. This role is ideal for engineers who enjoy owning systems in production, solving hard integration problems, and making complex infrastructure simple and reliable for users. EXAMPLE INITIATIVES You'll get to work on these types of projects on our Inference Platform: The Baseten Inference Stack at NVIDIA Dynamo Day 2x faster inference with KV cache-aware routing How Baseten multi-cloud capacity management (MCM) unifies deployments RESPONSIBILITIES Build the infrastructure and orchestration systems that de

About the Job Turn complex AI workload signals into faster, more efficient systems at data center scale. As a Senior AI Performance Engineer, you will analyze and optimize AI training and inference workloads across large-scale distributed systems. Your work will connect compute, memory, communication and software behavior. You will own complex performance investigations from problem definition through validation. You will turn profiling, benchmarking and modeling results into improvements engineers can build from. You will design benchmarks, investigate system bottlenecks and optimize distributed communication software across technologies such as MPI, NCCL, UCX and RDMA. Your work will help Graphcore deliver efficient, reliable AI systems for large-scale deployment. This role gives you rare scope across hardware, software, networking and system architecture. It is based in Austin, Texas. The team and culture The System Engineering Performance team architects, evaluates and optimizes high-performance infrastructure for large-scale data center deployments. The team works across the computing stack to understand real system behavior. Work moves through evidence, ownership and clear technical judgment. You will define investigations, coordinate across teams and validate impact with reliable performance data. Decisions are shaped through benchmarks, models, simulations and practical engineering tradeoffs. The team values engineers who think big, act fast, take responsibility, speak up and lead beyond their own area. What we’re looking for · Strong experience profiling and optimizing AI, machine learning or high-performance computing workloads. · Experience with distributed systems and communication libraries such as MPI, NCCL, UCX or libfabric. · Strong C++ and Python skills, including reliable tools or performance-sensitive software. · Deep understanding of compute, memory and communication behavior in large-scale systems. · Ability to own complex technical work and coo

About the Job Set the performance direction behind large-scale AI systems built for real deployment. As a Principal AI Performance Engineer, you will define how Graphcore analyzes and optimizes AI training and inference workloads. Your work will shape decisions across hardware, software, networking and system architecture. You will lead the hardest performance investigations, from model behavior to multi-node scaling. You will turn profiling, benchmarking and modeling evidence into technical roadmaps and engineering priorities. You will set measurement standards, guide distributed communication strategy and review critical C++ and Python tools. Your judgment will help teams make better decisions when data is incomplete. This role gives you rare influence across AI infrastructure at data center scale. It is based in Austin, Texas. The team and culture The System Engineering Performance team architects, evaluates and optimizes high-performance infrastructure for large-scale data center deployments. The team works across the computing stack to understand real system behavior. Work happens through evidence, technical ownership and direct cross-team influence. You will guide strategy, resolve complex tradeoffs and raise performance-engineering standards across the organization. Decisions are shaped by benchmarks, simulations, models and clear technical debate. The team values engineers who think big, act fast, take responsibility, speak up and lead beyond their own area. What we’re looking for · Deep experience profiling and optimizing AI, machine learning or high-performance computing workloads at scale. · Proven technical leadership across complex performance-engineering or system-architecture initiatives. · Expert C++ and Python skills, including reliable tools or performance-sensitive software. · Expert understanding of compute, memory, communication and scaling behavior in distributed systems. · Ability to define benchmarks, measurement practices or optimization sta