Data Engineer/ Senior Data Engineer - ETL
About the role
The Data Engineer/Senior Data Engineer - ETL role involves designing and developing scalable data solutions for a major global food marketing and distribution corporation. The position requires implementing distributed, highly available data processing applications on cloud platforms such as Google Cloud Platform (GCP), as well as designing, developing, and maintaining data pipelines that integrate multiple source systems. The role emphasizes adherence to software engineering best practices, continuous integration and delivery, and DevOps principles, while working collaboratively in an Agile, cross-functional team environment. Candidates should have a Bachelor's Degree in Computer Science or equivalent, with 2+ years of experience for Data Engineers and 4+ years for Senior Data Engineers, along with strong programming skills in Python, Shell, and SQL, and hands-on experience with ETL/ELT tools and cloud services.
Must have
- A Bachelor’s Degree in Computer Science or equivalent, and years of experience in developing enterprise grade data processing applications as specified below: o 2+ for Data Engineers o 4+ for Senior Data Engineers
- A strong programming background in data ops (Python, Shell, SQL)
- Experience in processing large volumes of data
- Hands-on experience working with relational/NoSQL databases and distributed storage engines
- Hands-on experience in ETL/ELT design and development using ETL/ELT tools such as but not limited to Airflow, Google Cloud Services such as Cloud Composer, Cloud Dataflow, Cloud Dataproc and Amazon Web Services such as Data Pipelines, Glue, Lambda, EMR, Spark, Hive
- Hands-on experience with Google Cloud Platform (GCP): Experience with services & serverless functions (e.g., BigQuery, Datastream, DataFlow, Pub/Sub, Cloud Functions, Cloud Run, Cloud composer, etc.) for data processing and orchestration
- Experience working in a Scrum Agile delivery environment and DevOps practices
- Experience in code management and CI/CD tools such as Github, Gitlab and Jenkins
Nice to have
- Experience in working with streaming data (using tools such as Pub/Sub, Kafka, Storm, Spark) will be an added advantage
Technologies
- Agile
- Airflow
- Bash
- BigQuery
- CI/CD
- Data Pipelines
- DevOps
- ELT
- ETL
- GCP
- GitHub
- GitLab
- Hive
- Jenkins
- Kafka
- AWS Lambda
- NoSQL
- Python
- Scrum
- Serverless
- Spark
- SQL
Also posted as
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka