Senior Data Engineer
About the role
The Senior Data Engineer will be responsible for building and optimizing scalable data pipelines and cloud data platforms to support real-time analytics and decision-making. Key responsibilities include developing batch and incremental ETL/ELT pipelines, designing medallion data layers, monitoring and troubleshooting production pipelines, optimizing compute and storage costs, and delivering high-quality datasets for analytics and data science teams. The role requires strong hands-on experience with big data or cloud data platforms, advanced SQL skills, and proficiency in Python or Java, as well as experience with tools such as Snowflake, Hadoop, AWS, Airflow, and Spark technologies. The candidate will also be expected to document work, review code, and collaborate within an Agile team environment.
Must have
- Degree in Computer Science, IT, Big Data or a related field, or equivalent experience
- 5+ years of hands-on data engineering on big data or cloud data platforms
- Strong production experience with Snowflake (preferred), Hadoop (HDFS) or AWS (S3)
- Strong Python or Java; hands-on with PySpark, Spark SQL or Snowpark
- Advanced SQL (MySQL, SQL Server or Oracle), including query tuning and optimisation
- Production experience with Airflow (preferred), Oozie or Informatica
- Proven experience implementing CDC, SCD and medallion architectures
- Track record of troubleshooting production pipelines and reducing compute and storage costs
- Git, CI/CD and code review experience
Technologies
- Agile
- Airflow
- AWS
- CI/CD
- Data Engineering
- Data Pipelines
- Data Science
- ELT
- ETL
- Git
- Hadoop
- Java
- MySQL
- Oracle
- PySpark
- Python
- S3
- Snowflake
- Snowpark
- Spark
- SQL
- SQL Server
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka