Senior Big Data Engineer - Databricks

Sr. Data Engineer Experience : 8+ Years Location : Bangalore - Hybrid Type : Full-time About Digit88 Digit88 is an AI-native product engineering partner helping startups and enterprises build, scale and operate intelligent software products. We are a lean, high-impact team of 75+ technologists, backed by leaders with deep experience across startups and global enterprises. We build strong, outcome-driven engineering teams that solve complex, real-world problems. From GenAI applications and data platforms to enterprise-grade SaaS, we deliver scalable, reliable, production-ready systems. Our expertise spans AI/ML, RAG systems, agentic workflows and large-scale data engineering - enabling businesses to move from idea to production with speed and confidence. Our teams operate as a true extension of our clients, with full ownership and flexible engagement models focused on measurable business outcomes - not just delivery. With 80+ AI implementations and proven success in scaling dedicated teams and driving significant cost efficiencies, we partner for long-term impact. We bring experience across B2B and B2C SaaS, web and mobile platforms, e-commerce and domains such as Conversational AI, HealthTech, IoT, ESG/Energy and Data Engineering - thriving in fast-paced, high-ownership environments. The Vision: To be the most trusted AI-native product engineering partner for innovative software companies worldwide delivering ownership, speed and measurable outcomes. The Opportunity: As a Senior Data Engineer, you will design, build and operate scalable data platforms and pipelines for global customers. You will work closely with customers, product teams and engineers to deliver reliable, production-grade data systems. You will play a key role in shaping data architecture, data engineering best practices and AI-driven data platforms at Digit88, enabling customers to move from raw data to actionable insights and intelligent systems. Key Responsibilities: Modernize legacy, non-scalable architectures and define a scalable target-state platform Design and implement Medallion Architecture (Bronze, Silver, Gold) using Delta Lake and Databricks components such as Delta Live Tables (DLT), Delta Sharing and Workflows (LakeFlow, LakeBase) Design and build scalable, reliable and production-ready ETL/ELT pipelines using PySpark, SQL and Databricks notebooks to ingest and transform data from diverse sources. Create and manage workflows using Databricks Workflows (Jobs) or orchestration tools to automate pipelines and dependencies Optimize Spark jobs for performance, scalability and cost efficiency (partitioning, caching, query tuning, cluster optimization) Implement data quality checks (e.g., DLT Expectations) and enforce governance via Unity Catalog (access control, PII masking, lineage) Design and implement event-driven and streaming pipelines (Kafka or equivalent) Ensure high data reliability through monitoring, observability and alerting Requirements: BE/MS in Computer Science or a related field with 8+ years of experience in data engineering Strong experience in ETL/ELT pipelines, data modeling, and distributed data systems, with hands-on expertise in Databricks Deep proficiency in PySpark, including performance optimization, job orchestration, and large-scale data processing Good understanding of event streaming systems such as Kafka or equivalent technologies Experience working with Azure data ecosystem (ADLS, Azure services, AHDS or similar data platforms) Strong foundation in event-driven architecture and scalable distributed systems Proven ability to design, review, and clearly articulate system architecture Experience leveraging AI-assisted development tools (e.g., Claude, Antigravity, etc.) to improve productivity Solid experience in Agile delivery, estimation, and program execution Experience working with global customers (US/EU) in a client-facing role Excellent written and verbal communication skills across engineering, business, and customer stakeholders Strong analytical thinking and structured problem-solving ability High ownership, reliability, and execution focus; consistently delivers despite constraints Strong attention to detail while maintaining a clear big-picture perspective Good to have skills: Experience in Healthcare, EHR/EMR data migration, Clinical Trials, or Life Sciences domains (at least one) Exposure to handling large-scale EMR/EHR integrations and reducing technical complexity across multiple data sources Experience working with healthcare data standards such as CDA, FHIR, and HL7, including data normalization into modern data models Ability to build and manage scalable data pipelines for ingestion, transformation (FHIR), data quality, governance, and near real-time processing Understanding of interoperability challenges across diverse healthcare systems and approaches to solve them Experience building patient-centric data platforms (e.g., Patient 360, Master Patient Index) Familiarity with data privacy, security, and compliance standards such as HIPAA Benefits/Culture @ Digit88: Comprehensive insurance coverage (Life, Health, Accident, parents and in-laws are optional) Flexible work model focused on outcomes Accelerated learning with non-linear growth opportunities Flat organization with high ownership & accountability Opportunity to work on cutting-edge AI and SaaS products with global customers (primarily North America, Australia, EU and UAE) Accomplished Global Peers - Working with some of the best engineers/professionals globally from the likes of Apple, Amazon, IBM Research, Adobe and other innovative product companies Direct exposure to building and scaling real-world systems across Conversational AI, Energy/Utilities, ESG, HealthTech, IoT and more High-impact roles with the ability to influence product, architecture and business outcomes globally ​​Learn from a founding team of serial entrepreneurs with multiple exits - high growth, high ownership and real challenges This is an exciting time to join Digit88 - build, scale and grow with us as part of our journey!