Ai Runtime Engineer Jobs

ABOUT BASETEN Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F , led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to to ship AI products. THE ROLE Baseten’s Inference Stack team builds the distributed runtime that powers large-scale LLM inference across our platform. We operate at the intersection of distributed systems, model performance, infrastructure, and developer experience. We enable customers to deploy and operate cutting-edge LLM models with industry-leading performance, scalability, reliability, and ease of use. As a Software Engineer on the Inference Stack team, you’ll work across the stack - from the developer experience customers use to deploy models, the libraries used for features like tool calling and reasoning, all the way down to the systems we use to orchestrate deployments in Kubernetes and route traffic efficiently. This is an ideal role for engineers who enjoy owning systems in production, solving hard integration problems, and making complex infrastructure simple and reliable for users. EXAMPLE INITIATIVES Blog Posts https://www.baseten.co/blog/nvidia-dynamo-day-baseten-inference-stack/ https://www.baseten.co/blog/how-baseten-achieved-2x-faster-inference-with-nvidia-dynamo/ https://www.baseten.co/blog/how-baseten-multi-cloud-capacity-management-mcm-powers-cloud-self-hosted-and-hybr/#comparing-deployment-options-cloud-vs-self-hosted-vs-hybrid RESPONSIBILITIES Develop infrastructure and orchestration systems for deploying and managing large-scale distributed LLM inference Work across the stack, from customer-facing features to low-le

About the Team Training Runtime designs the core distributed machine-learning training runtime that powers everything from early research experiments to frontier-scale model runs. With a dual mandate to accelerate researchers and enable frontier scale, we’re building a unified, modular runtime that meets researchers where they are and moves with them up the scaling curve. Our work focuses on three pillars: high-performance, asynchronous, zero-copy tensor and optimizer-state-aware data movement; performant, high-uptime, fault-tolerant training frameworks (training loop, state management, resilient checkpointing, deterministic orchestration, and observability); and distributed process management for long-lived, job-specific and user-provided processes. We integrate proven large-scale capabilities into a composable, developer-facing runtime so teams can iterate quickly and run reliably at any scale, partnering closely with model-stack, research, and platform teams. Success for us is measured by raising both training throughput (how fast models train) and researcher throughput (how fast ideas become experiments and products). About the Role As a Training: ML Framework Engineer, you will work on improving the training throughput for our internal training framework, while enabling researchers to experiment with new ideas. This requires good engineering (for example designing, implementing, and optimizing state-of-the-art AI models), writing bug-free machine learning code (surprisingly difficult!), and acquiring deep knowledge of the performance of supercomputers. In all the projects this role pursues, the ultimate goal is to push the field forward. We’re looking for people who love optimizing performance, understanding distributed systems, and who cannot stand having bugs in their code. Since our training framework is used for large runs with massive numbers of GPUs, performance improvements here will have a large impact. This role is based in San Francisco, CA. We use a

Coordination Systems provides foundational distributed systems building blocks for internal Datadog platforms. Our services cover sharding, consensus, resource protection, configuration distribution, and much more. We are looking for a manager to lead the Coordination Systems - Storage team. This team provides essential configuration storage and distribution systems that are depended upon by almost every service and pod at Datadog. We power critical runtime configuration (e.g. feature flags), complex control planes (e.g. dynamic sharding configuration), and much more. Storage is one of four subteams within Coordination Systems. If successful, the candidate will have opportunities to lead other growing and impactful areas such as Resource Protection. At Datadog, we place value in our office culture - the relationships that it builds, the creativity it brings to the table, and the collaboration of being together. We operate as a hybrid workplace to ensure our employees can create a work-life harmony that best fits them. What You’ll Do: (Describe role responsibilities here/max 6 bullets) Lead a core team of 5 engineers (distributed, with majority in NYC) Lead ceremonies, prioritize and delegate project Stay hands-on with the code, e.g. isolated features, small remediations, investigation follow ups Stay actively involved in operations, incidents, root cause analysis, etc. Constantly promote a culture of operational excellence, organizing gamedays, conducting operational reviews, staying proactive with reliability Who You Are: (Describe role qualifications here/max 6 bullets) Strong distributed systems skills, able to understand and account for a variety of failure modes, well-versed in end-to-end o11y, validation testing, simulation setup, etc. Worked on platform teams before, providing critical infrastructure to internal stakeholders Experienced in handling significant incidents, both as a responder and follow-up ow

Datadog's Cloud FinOps team answers a question the business runs on: what does each product and each customer actually cost us to serve? Finance uses this team's output to close Datadog's books, Product uses it to compare cost to serve against what a customer pays, and engineering leaders use it to catch cost movements before they compound. As the Engineering Manager for this team, you'll own delivery across the runtime pipeline, coverage onboarding, and cost-to-customer identity mapping. You'll own the contracts that Finance and platform teams depend on. The core engineering problem is reproducibility: every dollar of cloud cost is placed against the product and customer that caused it, with lineage on every result and closed periods that replay exactly. This role starts by closing a coverage gap and grows into owning the model every platform team at Datadog onboards through. At Datadog, we place value in our office culture - the relationships that it builds, the creativity it brings to the table, and the collaboration of being together. We operate as a hybrid workplace to ensure our employees can create a work-life harmony that best fits them. What You'll Do: Lead and grow a team of engineers building the platform that allocates Datadog's cloud cost to products and customers, setting the operating rhythm across runtime and release, coverage onboarding, and identity and mapping. Own the publication contract with Finance: versioned metrics with effective dates, backfill, and reproducible history, so any closed period can be replayed on demand and trusted for close. Own the metric contracts with platform teams across Engineering, shifting cost facts to the teams that generate the cost so coverage growth stops scaling linearly with your headcount. Partner with Revenue Engineering on the cost-to-revenue identity join that turns cost data into margin per product and per customer. Drive continuous improvement in product and customer cost coverage and accuracy, and hold d

About Prophecy The leader in AI-native data preparation and analysis, Prophecy is revolutionizing how the world's top enterprises turn data chaos into reliable insights. We introduce the AI-native data lifecycle (generate, refine, deploy) where our industry leading AI agents and humans work hand-in-hand in visual and document interfaces to analyze, transform and prepare data, to ship trusted insights at enterprise scale. To learn more, visit us on LinkedIn. Position Summary This is an exceptional opportunity to be a technical leader on the team that powers the heart of Prophecy: the Runtime, our orchestration and execution service. Every data workflow , whether generated by our AI data agents or built by a human in the visual editor , is planned, scheduled, and run at enterprise scale by Runtime, directly against the customer's own platform (Databricks, Snowflake, BigQuery, and more). When a business user asks an agent a question and sees trusted results seconds later, Runtime is what made it happen. As a Staff Engineer, you will set the technical direction for this service: the distributed execution engine, the task coordinator, the multi-engine connectors, and the fast interactive previews that make the AI-native experience feel instant. You will own hard, ambiguous, high-leverage problems end-to-end , from architecture to production reliability , and raise the bar for the engineers around you. We're seeking A players to work with A players. We're a high-growth company on a once-in-a-lifetime journey to revolutionize the way data is prepared and analyzed. Be a part of it. The impact you will have Own the engine behind the product. Architect and evolve the runtime service that turns agent- and human-authored data workflows into high-performance jobs that run reliably in production and interactively during authoring. This is deep-tech at the center of the platform. Make it fast and make it scale. Design the distributed execution and task-coordination layer , schedul

About the Role We are looking for a Principal Software Engineer to shape how our AWS-based SaaS products are built, deployed, and operated. This is a hands-on technical leadership role for an engineer with prior software development experience and deep expertise in AWS, infrastructure as code, CI/CD, automation, and observability. You will partner with application, platform, product, and security teams to improve the full software delivery lifecycle. You will modernize existing applications by reducing unnecessary coupling and enabling components to be built, tested, released, and deployed independently where the architecture supports it. You will also set technical direction across teams, lead complex modernization initiatives, and mentor engineers in the Bangalore organization. What You Will Do DevOps Strategy and Technical Leadership Define DevOps and platform engineering standards, reference architectures, delivery patterns, and operational practices. Lead cross-team initiatives that improve delivery speed, reliability, security, and maintainability. Make and communicate technical decisions that balance modernization, customer impact, and service continuity. Mentor engineers, lead knowledge-sharing sessions, and contribute to hiring and technical interviews. Application Modernization and Software Engineering Apply software development experience to understand application architecture, dependencies, build systems, runtime behavior, and operational risks. Assess existing applications and reduce tight coupling between components, services, shared libraries, and deployment units where practical. Enable components to be built, tested, versioned, and deployed independently Improve configuration, secrets handling, health checks, logging, metrics, and graceful startup and shutdown behavior. Build automation and engineering tools using a programming or scripting language such as Python, Go, Groovy, or shell. CI/CD and Developer Experience Design, implement, and ope

ABOUT BASETEN Baseten powers mission-critical inference for the world's most dynamic AI companies, like Cursor, Notion, OpenEvidence, Abridge, Clay, Gamma, and Writer. By uniting applied AI research, flexible infrastructure, and seamless developer tooling, we enable companies operating at the frontier of AI to bring cutting-edge models into production. We're growing quickly and recently raised our $1.5B Series F , led by Altimeter Capital, Conviction Partners, and Spark Capital. Join us and help build the platform engineers turn to ship AI products. THE ROLE We're looking for distributed systems engineers and product-minded generalists to build the distributed runtime that powers large-scale LLM inference on Baseten. Our inference platform empowers customers to deploy and operate cutting-edge models with industry-leading performance, scalability, and reliability. It also powers Model APIs, our hosted endpoints for the latest open-source models. You'll work across the stack, from the developer experience customers use to deploy models, through the libraries behind features like tool calling and reasoning, down to the systems that orchestrate deployments on Kubernetes and route traffic efficiently. Your job is to make sure every model on our platform is fast, reliable, and cost-efficient. You'll join a small, high-impact team at the intersection of distributed systems, model performance, infrastructure, and product, helping define how developers use AI models at scale. This role is ideal for engineers who enjoy owning systems in production, solving hard integration problems, and making complex infrastructure simple and reliable for users. EXAMPLE INITIATIVES You'll get to work on these types of projects on our Inference Platform: The Baseten Inference Stack at NVIDIA Dynamo Day 2x faster inference with KV cache-aware routing How Baseten multi-cloud capacity management (MCM) unifies deployments RESPONSIBILITIES Build the infrastructure and orchestration systems that de

Staff Software Engineer - Frontend - New York City Datadog’s Staff Engineers are our technical leaders operating at the forefront of technology, building solutions that take us through at least our next five years of growth. They do this in three major ways: As individual contributors, they bring world-class technical abilities to deliver industry-leading systems in areas such as data visualization, virtual runtime profiling, and planet-scale streaming. As technical leaders, they bring experienced technical breadth and communication skills to tackle design and architectural problems spanning the organization, charting the right course, and then leading delivery. In both roles, they participate in the staff engineering community and help us learn from what the industry is doing and what we've built before, and so improving company-wide standards around software and systems engineering. At Datadog, we place value in our office culture - the relationships and collaboration it builds and the creativity it brings to the table. We operate as a hybrid workplace to ensure our Datadogs can create a work-life harmony that best fits them. What You’ll Do: Use your expertise to design and deliver technology that will power the next generation of products Coach and develop engineers at various levels, from new university graduates to senior engineers with years of industry experience Leading redesigns of significant parts of the Datadog product and website Owning core customer experiences, including the first-time user experience. Who You Are: 10+ years of relevant engineering experience 4+ years of technical leadership experience, including both effective communication of complex concepts and experience coordinating across teams to ship critical products Running a high volume of experimentation and iteration cycles (A/B testing). Strong UI/UX intuition, adept at working closely with product design and

This role is part of Datadog’s Security Agent team, which powers critical security capabilities across Workload Protection, Vulnerability Management, Cloud Security products, and other emerging security offerings. As a Staff Software Engineer, you will lead the design and development of low-level Linux instrumentation and runtime security technologies that help customers detect threats, monitor system activity, and protect cloud-native workloads at scale. You will work on complex technical challenges involving eBPF, Linux kernel internals, performance-sensitive systems, and large-scale data collection while influencing technical direction across multiple product teams. This role offers significant ownership, broad organizational impact, and the opportunity to shape the future of Datadog’s security platform. At Datadog, we place value in our office culture - the relationships and collaboration it builds and the creativity it brings to the table. We operate as a hybrid workplace to ensure our Datadogs can create a work-life harmony that best fits them. What You’ll Do: Lead the architecture and development of security agent capabilities that power runtime threat detection and workload protection across Datadog Security products. Design and build reusable eBPF-based monitoring functionality for process, file, and network visibility within Linux environments. Drive end-to-end delivery of new features, from technical strategy and design through implementation, testing, and rollout. Establish and evolve testing methodologies that improve platform coverage, detection quality, reliability, and performance. Partner with product, security, infrastructure, and engineering teams to deliver shared platform capabilities used across multiple Datadog products. Provide technical leadership by influencing engineering direction, mentoring peers, and helping resolve complex cross-functional challenges. Who You Are: You have significant experience building software in Linux environments,

This role is part of Datadog’s Security Agent team, which powers critical security capabilities across Workload Protection, Vulnerability Management, Cloud Security products, and other emerging security offerings. As a Staff Software Engineer, you will lead the design and development of low-level Linux instrumentation and runtime security technologies that help customers detect threats, monitor system activity, and protect cloud-native workloads at scale. You will work on complex technical challenges involving eBPF, Linux kernel internals, performance-sensitive systems, and large-scale data collection while influencing technical direction across multiple product teams. This role offers significant ownership, broad organizational impact, and the opportunity to shape the future of Datadog’s security platform. At Datadog, we place value in our office culture - the relationships and collaboration it builds and the creativity it brings to the table. We operate as a hybrid workplace to ensure our Datadogs can create a work-life harmony that best fits them. What You’ll Do: Lead the architecture and development of security agent capabilities that power runtime threat detection and workload protection across Datadog Security products. Design and build reusable eBPF-based monitoring functionality for process, file, and network visibility within Linux environments. Drive end-to-end delivery of new features, from technical strategy and design through implementation, testing, and rollout. Establish and evolve testing methodologies that improve platform coverage, detection quality, reliability, and performance. Partner with product, security, infrastructure, and engineering teams to deliver shared platform capabilities used across multiple Datadog products. Provide technical leadership by influencing engineering direction, mentoring peers, and helping resolve complex cross-functional challenges. Who You Are: You have significant experience building software in Linux environments,

This role is part of Datadog’s Security Agent team, which powers critical security capabilities across Workload Protection, Vulnerability Management, Cloud Security products, and other emerging security offerings. As a Staff Software Engineer, you will lead the design and development of low-level Linux instrumentation and runtime security technologies that help customers detect threats, monitor system activity, and protect cloud-native workloads at scale. You will work on complex technical challenges involving eBPF, Linux kernel internals, performance-sensitive systems, and large-scale data collection while influencing technical direction across multiple product teams. This role offers significant ownership, broad organizational impact, and the opportunity to shape the future of Datadog’s security platform. At Datadog, we place value in our office culture - the relationships and collaboration it builds and the creativity it brings to the table. We operate as a hybrid workplace to ensure our Datadogs can create a work-life harmony that best fits them. What You’ll Do: Lead the architecture and development of security agent capabilities that power runtime threat detection and workload protection across Datadog Security products. Design and build reusable eBPF-based monitoring functionality for process, file, and network visibility within Linux environments. Drive end-to-end delivery of new features, from technical strategy and design through implementation, testing, and rollout. Establish and evolve testing methodologies that improve platform coverage, detection quality, reliability, and performance. Partner with product, security, infrastructure, and engineering teams to deliver shared platform capabilities used across multiple Datadog products. Provide technical leadership by influencing engineering direction, mentoring peers, and helping resolve complex cross-functional challenges. Who You Are: You have significant experience building software in Linux environments,

We are developing advanced multi-rack, multi-tenant AI/ML datacenters with NVIDIA GB200, and upcoming GB300 GPUs. NVIDIA seeks a Senior Software Engineer for our CSP (Cloud Service Provider) Engagements team to focus on the cloud-native stack for datacenter products like GB200. In this role, You will define customer workflows, prototype stack enhancements, and debug the toughest Kubernetes + Slurm issues in multi-rack, multi-tenant AI datacenters. You'll tackle complex scheduling challenges across racks, tenants, and clouds as part of the CSP engagements team. What you’ll be doing: Perform deep-dive debugging of multi-rack, multi-tenant clusters: scheduler behavior, container runtime issues, device-plugin crashes, RDMA/IB fabric anomalies, etc. Gather customer requirements and prototype feature extensions for Kubernetes operators, Slurm plugins, and custom micro-services that expose new GPU capabilities. Drive joint architecture reviews and “whiteboard” sessions with CSP and internal platform teams; convert findings into RFCs and upstream pull requests. Create reproducible testbeds (Helm/Ansible/Terraform) that mirror customer environments; automate validation and benchmark suites. Deliver technical collateral-design docs, how-to guides, demo scripts-and present at customer on-sites, KubeCon, and SlurmUG. Collaborate with AE, FAE, and Solution Architect teams to deliver integrated customer solutions and technical documentation. What we need to see: Strong source-level expertise in Kubernetes internals (scheduler, CRI/CNI/CSI, operators) and Slurm (federation, power-save, plugins). Hands-on experience integrating next-gen GPUs (Blackwell/GB200/GB300) or comparable accelerators into containerized clusters. Proven track record debugging large-scale, cloud-native stacks across ne

Secure Every Identity, from AI to Human Identity is the key to unlocking the potential of AI. Okta secures AI by building the trusted, neutral infrastructure that enables organizations to safely embrace this new era. This work requires a relentless drive to solve complex challenges with real-world stakes. We are looking for builders and owners who operate with speed and urgency and execute with excellence. This is an opportunity to do career-defining work. We're all in on this mission. If you are too, let's talk. Senior Software Engineer, Security Engineering Okta is The World’s Identity Company. We free everyone to safely use any technology, anywhere, on any device or app. Our flexible and neutral products, Okta Platform and Auth0 Platform, provide secure access, authentication, and automation, placing identity at the core of business security and growth At Okta, we celebrate a variety of perspectives and experiences. We are not looking for someone who checks every single box - we’re looking for lifelong learners and people who can make us better with their unique experiences. Join our team! We’re building a world where Identity belongs to you. We are looking for a Senior Software Engineer that will join the Auth0 Security Engineering organization. In this role, you will join a team of Software Engineers dedicated to building and managing services that secure the platform that Auth0 runs on. You will be instrumental in expanding our platform security controls and architecting custom workflows in a scalable, reliable manner. What You Will Do Service Development: Build, maintain, and operate platform security services and controls using Python, Node.js, or Golang. Security Automation: Build and contribute to our security automation initiatives for custom tooling. Container Security: Provide technical support and engineering solutions for securing Kubernetes (K8s) and containerized workloads, including runtime security, network policies, and admissio

As a Senior Security Engineer within Application Security at Datadog, you will help secure emerging application architectures where established security patterns may not yet exist, including agentic applications, LLM-enabled products, protocol-driven tool use, and untrusted execution environments. You will establish and validate security models, requirements, and controls while partnering with product and platform teams to develop secure-by-default approaches. Your work will address evolving trust boundaries, agent identity, data access, tool permissions, and runtime behavior through threat modeling, security solutions, and practical implementation. You will have the opportunity to shape how Datadog safely supports new application paradigms and address emerging security risks before they become systemic. At Datadog, we place value in our office culture - the relationships and collaboration it builds and the creativity it brings to the table. We operate as a hybrid workplace to ensure our Datadogs can create a work-life harmony that best fits them. What You’ll Do: Define security models, requirements, and guardrails for emerging application architectures where established patterns are incomplete or do not yet exist. Design, implement, and validate practical security controls that reduce risk in emerging architectures and high-risk system behaviors. Shape how emerging systems authenticate, authorize actions, isolate workloads, and access data, applying least-privilege principles throughout. Partner with product, platform, Infrastructure Security, and other engineering teams to identify security gaps and embed secure-by-default requirements from early design through production. Lead threat modeling and security assessments for new and existing products, focusing on evolving threats, unfamiliar trust boundaries, and new interaction patterns. Prioritize and drive remediation of emerging security risks, including continuous security testing where it provides durable cover

At Snowflake, we are powering the era of the agentic enterprise. To usher in this new era, we seek AI-native thinkers across every function who are energized by the opportunity to reinvent how they work. You don’t just use tools; you possess an innate curiosity, treating AI as a high-trust collaborator that is core to how you solve problems and accelerate your impact. We look for low-ego individuals who thrive in dynamic and fast-moving environments and move with an experimental mindset — who rapidly test emerging capabilities to discover simpler, more powerful ways to deliver results. At Snowflake, your role isn't just to execute a function, but to help redefine the future of how work gets done. We are looking for talented systems developers and researchers to join the Snowflake AI Research team and advance the state of the art in LLM inference systems and optimization . Our mission is to build the next generation of high-performance and intelligent inference systems. We optimize not only how fast and efficiently models run, but also how quickly inference systems can adapt to new models, architectures, hardware, and workloads. Our work spans the full inference stack—from distributed serving and runtime systems to GPU kernels and model-system co-design. We explore techniques such as adaptive parallelism, speculative and parallel decoding, disaggregated inference, scheduling and batching, KV-cache optimization, model swapping, quantization, and GPU kernel optimization to push the frontier of latency, throughput, scalability, and cost. Beyond optimizing individual models, we are building intelligent and adaptive inference systems that can automate performance optimization—rapidly profiling new models and workloads, identifying bottlenecks, selecting effective execution strategies, and adapting system configurations with minimal manual tuning. We embrace AI-native engineering , using AI not only as the workload we optimize, but also as a tool to accelerate system devel