Lead Data Engineer
About the role
The Lead Data Engineer role focuses on designing, building, and optimizing robust, scalable data pipelines and data warehousing solutions. The position requires significant experience in core data engineering, programming (Python and SQL), distributed computing frameworks, and advanced cloud data platforms. Responsibilities include developing and maintaining automated ETL/ELT pipelines, building data models, optimizing workflows, and collaborating with analysts and business teams. The role is suited for candidates with a proven track record in data quality, validation, and cleansing frameworks, and experience in high-growth technology environments.
Must have
- Core Data Engineering: 5+ years of experience designing, building, and optimizing robust, scalable data pipelines
- Programming Languages: 4+ years of strong production-grade coding experience in Python and advanced proficiency writing complex, optimized SQL queries
- Big Data Technologies: 3+ years of experience working with distributed computing frameworks such as Apache Spark (or PySpark).
- Data Warehousing Architecture: Advanced proficiency in cloud data platforms (such as Snowflake, Google BigQuery, or AWS Redshift) and building optimized dimensional models
- Problem-Solving Data Quality: Proven track record of implementing rigorous data validation data cleansing, and error-handling frameworks
Nice to have
- Cloud Infrastructure: 2+ years of hands-on experience with cloud platforms particularly with managed data services and serverless computing
- Orchestration Tools: Experience managing and monitoring complex workflows using tools like Apache Airflow, Prefect, or dbt.
- CI/CD & DevOps: Experience deploying pipelines via CI/CD frameworks and infrastructure-as-code
- Streaming Architecture: Familiarity with real-time data streaming technologies such as Apache Kafka or AWS Kinesis is a strong plus.
Technologies
- Agile
- Airflow
- AWS
- BigQuery
- CI/CD
- Data Engineering
- Data Modeling
- Data Pipelines
- Data Warehousing
- dbt
- DevOps
- Distributed Computing
- ELT
- ETL
- Generative AI
- Infrastructure as Code
- Kafka
- Kinesis
- Prefect
- PySpark
- Python
- Redshift
- Serverless
- Snowflake
- Spark
- SQL
Highlights
The role requires maintaining 90%+ weekly usage of approved GenAI tools to enhance coding, documentation, and analysis workflows. Candidates should have 5-7 years of relevant work experience. Experience in enterprise software or high-growth technology environments is preferred.
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka