Senior Data Engineer/ Data Engineer-ETL
About the role
The Engineer - Data Engineering role involves designing and developing data solutions for a major global corporation in the food marketing and distribution sector. The position focuses on implementing scalable, distributed data processing applications on cloud platforms such as Google Cloud Platform (GCP), as well as designing, developing, and maintaining data pipelines that integrate multiple source systems. The engineer will adhere to continuous integration and delivery practices, ensure high code quality, and collaborate within a cross-functional Agile team. Responsibilities also include following DevOps principles and participating in the full software lifecycle, from development through post-production support. Candidates should have a strong programming background, experience with large data volumes, and hands-on expertise with ETL/ELT tools and cloud services.
Must have
- ABachelor’s Degree in Computer Science or equivalent, and 1-2years of experience in developing enterprise grade data processing applications
- A strong programming background in data ops (Python,Shell, SQL)
- Experience in processing large volumes of data
- Hands-on experience working with relational/NoSQL databases and distributed storage engines
- Hands-on experience in ETL/ELT design and development using ETL/ELT tools such as but not limited to Airflow, Google Cloud Services such as Cloud Composer, Cloud Dataflow, Cloud Dataproc and Amazon Web Services such as Data Pipelines, Glue, Lambda, EMR, Spark, Hive
- Experience in working with streaming data (using tools such as Pub/Sub, Kafka, Storm, Spark) will be an added advantage
- Hands-on experience with Google Cloud Platform (GCP): Experience with services & serverless functions (e.g., BigQuery, Datastream, DataFlow, Pub/Sub, Cloud Functions, Cloud Run, Cloud composer, etc.) for data processing and orchestration
- Experience working in a Scrum Agile deliveryenvironment and DevOpspractices
- Experience in code management and CI/CD tools such as Github, Gitlaband Jenkins
- Experience in an agile environment and aligning POD members on technical visions and pathto implementation
- A strong desireto continue to grow your skillset
- Strong communication skills that are influential and convincing
- Experience with application monitoring(Datadog, or equivalent) will be an added advantage
- A passion for building and maintaining data solutions (data warehousing, data marts, data lakes, data mesh) will be an added advantage
Technologies
- Agile
- Airflow
- Bash
- BigQuery
- CI/CD
- Cloud
- Data Engineering
- Data Lake
- Data Pipelines
- Data Warehousing
- Datadog
- DevOps
- ELT
- ETL
- GCP
- GitHub
- Hive
- Jenkins
- Kafka
- AWS Lambda
- NoSQL
- Python
- Scrum
- Serverless
- Spark
- SQL
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka