Staff Software Engineer- Foundation Model Inference

P-1285 About This Role As a staff software engineer for GenAI inference, you will lead the architecture, development, and optimization of the inference engine that powers Databricks Foundation Model API.. You’ll bridge research advances and production demands, ensuring high throughput, low latency, and robust scaling. Your work will encompass the full GenAI inference stack: kernels, runtimes, orchestration, memory, and integration with frameworks and orchestration systems. What You Will Do Own and drive the architecture, design, and implementation of the inference engine, and collaborate on model-serving stack optimized for large-scale LLMs inference Partner closely with researchers to bring new model architectures or features (sparsity, activation compression, mixture-of-experts) into the engine Lead the end-to-end optimization for latency, throughput, memory efficiency, and hardware utilization across GPUs, and accelerators Define and guide standards to build and maintain instrumentation, profiling, and tracing tooling to uncover bottlenecks and guide optimizations Architect scalable routing, batching, scheduling, memory management, and dynamic loading mechanisms for inference workloads Ensure reliability, reproducibility, and fault tolerance in the inference pipelines, including A/B launches, rollback, and model versioning Collaborate cross-functionally on Integrating with federated, distributed inference infrastructure – orchestrate across nodes, balance load, handle communication overhead Drive cross-team collaboration: with platform engineers, cloud infrastructure, and security/compliance teams Represent the team externally through benchmarks, whitepapers, and open-source contributions What We Look For BS/MS/PhD in Computer Science, or a related field Strong software engineering background (6+ years or equivalent) in performance-critical systems Proven track record of owning complex system components and driving architectural decisions end-to-end Deep understa

Why Harvey At Harvey, we’re transforming how legal and professional services operate. By combining frontier agentic AI, an enterprise-grade platform, and deep domain expertise, we’re reshaping how critical knowledge work gets done for decades to come. This is a rare chance to help build a generational company at a true inflection point. We have strong product-market fit and world-class investor support. We’re scaling fast and defining a new category in real time. The work is ambitious, the bar is high, and the opportunity for growth — personal, professional, and financial — is unmatched. Our team moves fast, takes ownership, and is deeply committed to the mission — operating with intensity, staying close to our customers, and pushing each other for excellence. We live by three values: Decisiveness, Simplicity, and Job's Not Finished. We act quickly on clear judgment over perfect information, we believe simplicity is what scales, and we're never satisfied with where we are. If you want to do the best work of your career alongside people who share that drive, we'd love to build with you. At Harvey, the future of professional services is being written today — and we’re just getting started. Role Overview Harvey is building the AI platform trusted by the world’s leading law firms and enterprises. Our infrastructure is the foundation that powers every customer interaction, every model inference, and every production workload. We’re looking for a Production Engineer to help build and operate Harvey’s core compute and networking infrastructure, Kubernetes platform, workflow orchestration platform, and production infrastructure foundations. You’ll work on the systems that enable engineering teams to move quickly and operate reliable services at scale. In this role, you’ll improve the reliability, scalability, security, and efficiency of Harvey’s infrastructure platform. You’ll solve complex production challenges across compute fleet management, capacity planning, infrastruc

Who are we? Cohere is the leading security-first enterprise AI company. We build cutting-edge foundation AI models and end-to-end products that are designed to solve real-world business problems. We’re training and deploying frontier models for enterprises who are building AI systems. We believe that our work is instrumental to the widespread adoption of AI and we are looking for folks that want to be part of that. We obsess over what we build. Each one of us is responsible for contributing to increasing the capabilities of our models and the value they drive for our customers. Cohere is a team of researchers, engineers, designers, and more, who are all passionate about their craft. We are a global technology company headquartered in Toronto with key offices in London, New York City, San Francisco, Montreal, Paris, Berlin and Seoul. Join us! Why this role? Are you energized by building high-performance, scalable and reliable machine learning systems? Do you want to help define and build the next generation of AI platforms powering advanced NLP applications? We are looking for Members of Technical Staff to join the Model Serving team at Cohere. The team is responsible for developing, deploying, and operating the AI platform delivering Cohere's large language models through easy to use API endpoints. In this role, you will work closely with many teams to deploy optimized NLP models to production in low latency, high throughput, and high availability environments. You will also get the opportunity to interface with customers and create customized deployments to meet their specific needs. You may be a good fit if you have: 5+ years of engineering experience running production infrastructure at a large scale Experience designing large, highly available distributed systems with Kubernetes, and GPU workloads on those clusters Experience with Kubernetes dev and production coding and support Experience with GCP, Azure, AWS, OCI, multi-cloud on-prem / hybrid serving Experienc

About Anthropic Anthropic’s mission is to create reliable, interpretable, and steerable AI systems. We want AI to be safe and beneficial for our users and for society as a whole. Our team is a quickly growing group of committed researchers, engineers, policy experts, and business leaders working together to build beneficial AI systems. About the role: The Safeguards ML Sampling Path team builds and operates the production services that power Claude's safety systems. These services sit on the token generation path across every platform Claude runs on: every request must pass through them, and each millisecond of added latency is wait time for our users. You’ll keep p99 latency flat as traffic grows, build for robustness as dependencies time out or partially fail, and ship changes safely to a system that cannot go down. What you'll do: Design, build, and operate the backend systems that process every token on the generation path for Claude requests, including the streaming contract with the API and inference engines. Own latency and reliability end to end: define and maintain SLOs and error budgets for added latency, time-to-first-token, and availability, and lead incident response and postmortem follow-through. Ship changes to the hot path rapidly but safely — canaried and gradual rollouts, error budget and latency gating, fast rollbacks — and drive per-token performance: chase tail latency and keep cost flat as traffic, models, and checks per request grow. Set technical direction for the sampling path: lead design reviews, make latency, reliability, and cost trade-off calls with the inference and research teams, mentor engineers, and raise the operational bar for the wider Safeguards organization. You may be a good fit if you: Have designed, built, and operated high QPS systems at global scale, and were accountable for them in production: incident response, outages, and postmortem-driven remediation. Have a strong foundation in distributed systems: replication, cons

Who are we? Cohere is the leading security-first enterprise AI company. We build cutting-edge foundation AI models and end-to-end products that are designed to solve real-world business problems. We’re training and deploying frontier models for enterprises who are building AI systems. We believe that our work is instrumental to the widespread adoption of AI and we are looking for folks that want to be part of that. We obsess over what we build. Each one of us is responsible for contributing to increasing the capabilities of our models and the value they drive for our customers. Cohere is a team of researchers, engineers, designers, and more, who are all passionate about their craft. We are a global technology company headquartered in Toronto with key offices in London, New York City, San Francisco, Montreal, Paris, Berlin and Seoul. Join us! Role Overview As a Member of Technical Staff in Post-Training Data Infrastructure, you will build and own the pipelines that produce the post-training data for our large language models (LLMs), from SFT and RL data to synthetic and human-annotated data. You will work closely with researchers and annotation teams so they can iterate on data as quickly as they iterate on models. Key Responsibilities Build and maintain data pipelines for SFT, preference, and RL post-training data. Own dataset quality, versioning, and reproducibility. Turn research data recipes into production-grade, reusable components. Scale model inference for data generation and model-based grading. Design data formats and schemas that keep post-training datasets consistent and efficient to train on. Collaborate with researchers and engineers to make sure data pipelines keep pace with the needs of cutting-edge language models. Qualifications Strong software engineering skills. Experience designing and building large-scale data pipelines for machine learning, ideally for LLM post-training. Track record of running distributed data processing and model inference at

We believe that the way people interact with their finances will drastically improve in the next few years. We’re dedicated to empowering this transformation by building the tools and experiences that thousands of developers use to create their own products. Plaid powers the tools millions of people rely on to live a healthier financial life. We work with thousands of companies like Venmo, SoFi, several of the Fortune 500, and many of the largest banks to make it easy for people to connect their financial accounts to the apps and services they want to use. Plaid’s network covers 12,000 financial institutions across the US, Canada, UK and Europe. Founded in 2013, the company is headquartered in San Francisco with offices in New York, Seattle, Washington D.C., Raleigh, London, and Amsterdam. We are the Data Foundation & AI team within Plaid’s Data organization. Our mission is to build the shared ML and AI infrastructure that powers intelligent capabilities across Plaid’s product suite. We develop the foundational systems, models, and data assets that transform Plaid’s unique financial network data into scalable, general-purpose representations that teams across the company can leverage. Our work spans the full ML lifecycle — from large-scale data curation and model pretraining to production serving, evaluation, and monitoring. As part of the team, you’ll work at the intersection of machine learning infrastructure, applied AI, and distributed systems, helping establish the core AI platform that enables innovation across Plaid. As a Staff Machine Learning Engineer, you will lead the technical strategy and development of Plaid’s foundation models, driving key decisions across pretraining objectives, model architecture, and fine-tuning approaches that power a wide range of downstream product applications. You will serve as the technical lead for the full machine learning lifecycle, overseeing everything from data curation and experimentation to production deployment,