Staff Software Engineer, Search Quality

RDQ427R351 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their business. Search plays a foundational role in this mission. Whether through keyword-based retrieval, semantic similarity via vector embeddings, or hybrid approaches that combine both, our Search technologies help customers find, discover, and understand information across massive, complex datasets. These capabilities power everything from Retrieval Augmented Generation (RAG), AI assistants, and recommendation systems to enterprise knowledge management, in-product search, and data exploration. As a Staff Engineer for Search, you’ll play a critical role in building and scaling our next-generation Search product — spanning Vector Search, Keyword Search, and Search Quality. You’ll drive the design and evolution of a highly-performant, cost-efficient, and developer-friendly Search stack that enables customers to build AI applications with accuracy, speed, and scale. Beyond hands-on contributions, you’ll help define the long-term vision, mentor senior engineers, collaborate across teams, and lead strategic efforts that expand the technical and business impact of Search at Databricks. Check out our recent blog post: Decoupled by Design: Billion-Scale AI Search The impact you will have: Drive the architecture and evolution of the Search product, delivering scalable, low-latency, and high-quality retrieval across massive multimodal datasets. Identify and solve challenges in indexing, storage, query execution, and ranking — blending innovations in vector search, keyword search, and hybrid retrieval techniques. Partner with infrastructure, product, and research teams to shape APIs and developer experiences that

P-1930 At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their business. Training and customizing state-of-the-art AI models is one of the most demanding workloads in computing, and it sits at the heart of Databricks' Mosaic AI mission. AI Runtime (AIR) is our managed platform for large-scale GPU training and fine-tuning. It gives customers on-demand access to fleets of the latest accelerators and a serverless experience that hides the complexity of provisioning, scheduling, and orchestrating multi-node jobs, with the resilience to keep training running for days or weeks across thousands of GPUs. AIR powers the full spectrum of custom training, from fine-tuning open models to pre-training frontier-scale foundation models, for some of the most sophisticated AI teams in the world. As a Staff Software Engineer for AI Runtime, you will play a critical role in building and scaling the systems that make large-scale training fast, reliable, and effortless. You will drive the architecture and evolution of the managed GPU training stack, spanning scheduling and capacity, distributed training performance, fault tolerance, and the developer experience of launching and operating jobs at scale. Beyond hands-on contributions to core systems, you will help define the long-term technical vision for AIR, mentor senior engineers, partner across product, research, and platform teams, and lead the initiatives that expand the technical and business impact of custom training at Databricks. Check out our recent blog posts: How we keep GPUs reliable across Databricks AI Fast, fault-tolerant PyTorch training on AI Runtime The impact you will have: Drive the architecture and evolution of AIR'

RDQ427R70 Our mission at Databricks is to radically simplify the data lifecycle with a unified Lakehouse platform for data engineering, analytics, and AI. The Lakehouse addresses major challenges in enterprise data platforms, including reliability, data staleness, operational complexity, total cost of ownership, and data lock-in. Learn more about the Lakehouse architecture. Lakeflow is a critical part of this vision, helping customers build and operate streaming and batch ETL pipelines that power business-critical data products. As these workloads become increasingly mission-critical, customers need them to continue operating through infrastructure failures and cloud-region outages. As a Staff Engineer on the part of the Lakeflow Disaster Recovery team, you will design and implement distributed systems that replicate and recover pipelines across regions. A pipeline is more than source code and output tables: it includes streaming checkpoints, source offsets, stateful operator state, table versions, transaction metadata, schedules, and dependencies across a dataflow graph. You will solve challenging problems involving consistency, idempotency, causal ordering, failover, failback, and safe recovery without silent data loss or duplication. You will work in one or more of the following areas: Cross-region replication and recovery for Lakeflow pipelines, streaming tables, and materialized views Distributed consistency across pipeline dependencies, table versions, and transaction logs Failover and failback workflows with conservative correctness guardrails Deep clone, metadata reconciliation, observability, and failure-injection testing High-fidelity recovery simulations, game-day testing, and formal reasoning about failure modes What we look for: A passion for distributed systems, databases, storage systems, streaming systems, or reliability engineering Strong software engineering skills in Java, Scala, C++, Go, Python, or a similar production language Understandin

P-1126 Summary At Databricks, we are obsessed with enabling data teams to solve the world's toughest problems, from security threat detection to cancer drug development. We do this by building and running the world's best data and AI infrastructure platform, so our customers can focus on the high-value challenges that are central to their missions. Our engineering teams build highly technical products that fulfill real, important needs in the world. We constantly push the boundaries of data and AI technology, while simultaneously operating with the resilience, security, and scale that is critical to making customers successful on our platform. Customers trust Databricks with their most valuable data and Trust & Safety has the mission to build the most trusted data analytics and ML platform in the world. Security Engineering is an integral part of Trust & Safety and has a critical role to play in keeping customer data from bad actors. We are looking for senior leaders such as yourselves to create the vision and define the strategy for this space. The impact you will have: Make Databricks safer for our customers by identifying and plugging key gaps in our infrastructure and services Attract top talent from across the industry. Represent the security engineering discipline throughout the organization, having a powerful voice to make us more data-driven Represent Databricks at academic and industry conferences & events What we look for: 7+ years of experience in Data Security or related areas and expertise in two or more of the following--Cryptography, Kubernetes Security, Web Security, Governance, Privacy, Trust, Safety, Authentication, Identity Management, Access Control, Key Management, Inter-Service Authentication, Secure Application Frameworks, Detection & Response. Experience building systems at large scale internet companies is a huge plus. 10+ years of experience building large scale distributed systems with high availability Leadersh

P-1124 Summary At Databricks, we are obsessed with enabling data teams to solve the world's toughest problems, from security threat detection to cancer drug development. We do this by building and running the world's best data and AI infrastructure platform, so our customers can focus on the high-value challenges that are central to their missions. Our engineering teams build highly technical products that fulfill real, important needs in the world. We constantly push the boundaries of data and AI technology, while simultaneously operating with the resilience, security, and scale that is critical to making customers successful on our platform. Customers trust Databricks with their most valuable data and Trust & Safety has the mission to build the most trusted data analytics and ML platform in the world. Security Engineering is an integral part of Trust & Safety and has a critical role to play in keeping customer data from bad actors. We are looking for senior leaders such as yourselves to create the vision and define the strategy for this space. The impact you will have: Make Databricks safer for our customers by identifying and plugging key gaps in our infrastructure and services Attract top talent from across the industry. Represent the security engineering discipline throughout the organization, having a powerful voice to make us more data-driven Represent Databricks at academic and industry conferences & events What we look for: 7+ years of experience in Data Security or related areas and expertise in two or more of the following--Cryptography, Kubernetes Security, Web Security, Governance, Privacy, Trust, Safety, Authentication, Identity Management, Access Control, Key Management, Inter-Service Authentication, Secure Application Frameworks, Detection & Response. Experience building systems at large scale internet companies is a huge plus. 10+ years of experience building large scale distributed systems with high availability Leadersh

P-188 Our mission at Databricks is to radically simplify the whole data lifecycle from ingestion to ETL, BI, and all the way up to ML/AI with a unified platform. To achieve this goal, we believe the data warehouse architecture as we know it today will be replaced by a new architectural pattern, Lakehouse ( CIDR 2021 paper ), open platforms that unify data warehousing and advanced analytics. The new architecture will help address several major challenges, including data staleness, reliability, total cost of ownership, data lock-in, and limited use-case support. A critical part of realizing this vision is the next generation (decoupled) query engine and structured storage system that can outperform specialized data warehouses in relational query performance, yet retain the expressiveness and of general purpose systems such as Apache Spark™ to support diverse workloads ranging from ETL to data science. As part of this team, you will be working in one or more of the following areas to design and implement these next gen systems that leapfrog state-of-the-art: Query compilation and optimization Distributed query execution and scheduling Vectorized execution engine Data security Resource management Transaction coordination Efficient storage structures (encodings, indexes) Automatic physical data optimization What we look for: A passion for database systems, storage systems, distributed systems, language design, or performance optimization Experience working towards a multi-year vision with incremental deliverables Motivated by delivering customer value and impact 8+ years of experience working in a related system (preferred) Optional: PhD in databases or distributed systems Pay Range Transparency Databricks is committed to fair and equitable compensation practices. The pay range(s) for this role is listed below and represents the expected salary range for non-commissionable roles or on-target earnings for commissionable roles. Actual compensation packages are