Data Engineer
About the role
The Senior Data Engineer role focuses on building and optimizing scalable data pipelines and cloud data platforms to support real-time analytics and decision-making. Key responsibilities include developing batch and incremental ETL/ELT pipelines, designing medallion data layers, monitoring and troubleshooting production pipelines, optimizing compute and storage costs, and delivering high-quality datasets for analytics and data science teams. The position requires strong hands-on experience with big data or cloud data platforms, advanced SQL skills, and proficiency in Python or Java, as well as experience with tools such as Snowflake, Hadoop, AWS, Airflow, and Spark technologies. The role also involves documentation, code review, and Agile team collaboration.
Must have
- Degree in Computer Science, IT, Big Data or a related field, or equivalent experience
- 5+ years of hands-on data engineering on big data or cloud data platforms
- Strong production experience with Snowflake (preferred), Hadoop (HDFS) or AWS (S3)
- Strong Python or Java; hands-on with PySpark, Spark SQL or Snowpark
- Advanced SQL (MySQL, SQL Server or Oracle), including query tuning and optimisation
- Production experience with Airflow (preferred), Oozie or Informatica
- Proven experience implementing CDC, SCD and medallion architectures
- Track record of troubleshooting production pipelines and reducing compute and storage costs
- Git, CI/CD and code review experience
Technologies
- Agile
- Airflow
- AWS
- CI/CD
- Data Engineering
- Data Pipelines
- Data Science
- ELT
- ETL
- Git
- Hadoop
- Java
- MySQL
- Oracle
- PySpark
- Python
- S3
- Snowflake
- Snowpark
- Spark
- SQL
- SQL Server
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka