Lead Data Engineer

About Arcana Arcana was recently named one of LinkedIn's Top Startups of 2026 , ranking among 50 rising companies in the U.S. alongside Anthropic, Perplexity, Cognition, and others. Arcana is a front-office portfolio intelligence platform built for hedge funds and asset managers. We empower investment professionals—from CIOs and portfolio managers to analysts and risk teams—with the tools to understand performance and risk, construct portfolios, evaluate investments, and make better investment decisions. The institutional investment software stack hasn't kept up with how modern investment teams work, and we're rebuilding it from the ground up. The Role As a Lead Data Engineer, you will take ownership of designing and delivering high-performance data platforms that enable analytics, AI/ML, and real-time insights across the organisation. You will mentor junior engineers, drive architectural decisions, and collaborate with stakeholders to align the data platform roadmap with business objectives. This is a hands-on technical leadership role for someone who enjoys both coding and guiding a team toward innovative solutions. Responsibilities: Lead the design and implementation of scalable and reliable data pipelines using Distributed Computing, Kafka, Airflow and Temporal. Architect data solutions that support business analytics, and real-time processing. Mentor and guide junior engineers, conducting code reviews and setting coding standards. Collaborate with cross-functional teams to identify and solve data challenges. Drive improvements in infrastructure, automation, and monitoring to enhance scalability and performance. Ensure data quality, governance, and compliance with internal policies and industry best practices. Stay current with the latest data trends and technologies, and make recommendations to keep the platform ahead of the curve. Requirements: 6-8 years of experience in data engineering or distributed systems. Strong hands on programming skills in Python / Scala / Rust / Golang, and SQL. Proven experience with cloud-native services (AWS, GCP) and big data tools (e.g., Spark, Flink, Datafusion, Doris, Trino etc). Expertise in building real-time pipelines using Kafka, and batch pipelines. Experience working with CI/CD pipelines, Kubernetes, Docker, and IaC tools. Demonstrated ability to lead and mentor teams in a fast-paced, agile environment. Strong problem-solving and communication skills with the ability to collaborate with multiple stakeholders. Contributions to open-source projects or experience in a fast pace product-based company What We Offer: Competitive compensation and benefits package. Opportunity to work with modern technologies and drive innovation. Flexible working hours with Hybrid options. A collaborative environment focused on learning and growth.