Data Engineer/ Senior Data Engineer - ETL
About the role
The Data Engineer/Senior Data Engineer - ETL role involves designing and developing scalable data solutions for a major global food marketing and distribution corporation. The position requires implementing distributed, highly available data processing applications on cloud platforms such as Google Cloud Platform (GCP), developing and maintaining data pipelines, and integrating multiple source systems. The engineer will work collaboratively in a cross-functional Agile team, adhering to DevOps principles and best practices throughout the software lifecycle, including development, QA, deployment, and post-production support.
Must have
- A Bachelor’s Degree in Computer Science or equivalent, and years of experience in developing enterprise grade data processing applications as specified below: o 2+ for Data Engineers o 4+ for Senior Data Engineers
- A strong programming background in data ops (Python, Shell, SQL)
- Experience in processing large volumes of data
- Hands-on experience working with relational/NoSQL databases and distributed storage engines
- Hands-on experience in ETL/ELT design and development using ETL/ELT tools such as but not limited to Airflow, Google Cloud Services such as Cloud Composer, Cloud Dataflow, Cloud Dataproc and Amazon Web Services such as Data Pipelines, Glue, Lambda, EMR, Spark, Hive
- Hands-on experience with Google Cloud Platform (GCP): Experience with services & serverless functions (e.g., BigQuery, Datastream, DataFlow, Pub/Sub, Cloud Functions, Cloud Run, Cloud composer, etc.) for data processing and orchestration
- Experience working in a Scrum Agile delivery environment and DevOps practices
- Experience in code management and CI/CD tools such as Github, Gitlab and Jenkins
Nice to have
- Experience in working with streaming data (using tools such as Pub/Sub, Kafka, Storm, Spark) will be an added advantage
Technologies
- Agile
- Airflow
- Bash
- BigQuery
- CI/CD
- Data Pipelines
- DevOps
- ELT
- ETL
- GCP
- GitHub
- GitLab
- Hive
- Jenkins
- Kafka
- AWS Lambda
- NoSQL
- Python
- Scrum
- Serverless
- Spark
- SQL
Highlights
The role distinguishes between Data Engineer (2+ years experience) and Senior Data Engineer (4+ years experience) levels. Experience with streaming data tools is considered an advantage. The position requires hands-on work with GCP and various ETL/ELT tools. No explicit location, remote work, or application deadline details are provided.
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka