Lead Data Engineer
About the role
The Lead Data Engineer will be responsible for leading the design and evolution of enterprise data platforms that support operational, analytical, AI/ML, and Generative AI workloads. The role involves owning and evolving data architecture, designing scalable data platforms, defining data models and integration standards, and establishing best practices for data engineering. The position also includes building and optimizing ETL/ELT pipelines, integrating data from various healthcare systems, evaluating new technologies, and mentoring a team of Data Engineers. The ideal candidate will have significant experience in data engineering, hands-on expertise with modern data technologies, and strong leadership skills.
Must have
- Bachelor’s degree in Computer Science, Information Technology, Engineering, or a related discipline.
- 6+ years of experience in Data Engineering, Data Platform Engineering, or Data Architecture.
- Hands-on experience with Apache NiFi, Airflow, dbt, Spark, Kafka, or equivalent modern data engineering technologies.
- Experience with Power BI or other enterprise BI platforms.
- Familiarity with healthcare interoperability standards such as HL7 and FHIR.
- Experience building data platforms for AI/ML and Generative AI workloads.
- Knowledge of CI/CD, Infrastructure as Code, and DevOps practices.
- Experience with containerized workloads using Docker and Kubernetes.
- Strong technical leadership, architecture, problem-solving, and communication skills.
Technologies
- Airflow
- CI/CD
- Data Engineering
- Data Pipelines
- Data Warehousing
- dbt
- DevOps
- Docker
- ELT
- ETL
- Feature Engineering
- FHIR
- Generative AI
- HL7
- Infrastructure as Code
- Kafka
- Kubernetes
- Lakehouse
- LLM
- Machine Learning
- Metadata Management
- NiFi
- Power BI
- RAG
- Spark
- Vector Databases
Highlights
The role requires working the 3:00 PM – 12:00 AM IST shift, Monday to Friday. The position is open to candidates located in India or Sri Lanka.
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka