Engineer - Data Engineering
About the role
The Engineer - Data Engineering role involves developing distributed, high-availability data processing applications and end-to-end batch/streaming pipelines on Google Cloud Platform (GCP) to meet enterprise requirements. Key responsibilities include implementing and maintaining low-latency Change Data Capture (CDC) pipelines, ensuring code quality and system stability, building automated CI/CD pipelines, adhering to DevOps principles, practicing cloud resource efficiency, maintaining documentation, collaborating within Agile teams, and leveraging AI-assisted development tools. The position requires a bachelor's degree in computer science, IT, or equivalent, and preferably 1-2 years of experience in enterprise data processing applications. Candidates should have strong programming skills in Python, SQL, and Shell/Bash, experience with relational/NoSQL databases, ETL/ELT pipelines, GCP services, ERP data ingestion, DevOps practices, and Agile environments.
Must have
- A bachelor’s degree in computer science, IT or equivalent qualification.
- Preferably 1-2 years of experience in developing enterprise grade data processing applications.
- Sound knowledge of Data engineering practices and processes.
- Solid programming skills in Python, SQL and Shell/Bash.
- Hands-on experience working with relational/NoSQL databases.
- Experience in building and maintaining ETL/ELT pipeline using Ingestion platform (Fivetran), frameworks & orchestration tools.
- Hands-on experience in implement, and maintain low-latency Change Data Capture (CDC) pipelines to stream data from source relational databases into enterprise data warehouse.
- Hands-on experience with Google Cloud Platform (GCP): Experience with services & serverless functions (e.g., BigQuery, Datastream, DataFlow, Pub/Sub, Cloud Functions, Cloud Run, Cloud composer, etc.) for data processing and orchestration.
- Hands-on experience with data ingestion and extraction from ERP systems like AS400, SAP, and Dynamics 365.
- Experience in code management, DevOps practices & CI/CD tools such as Github, Gitlab and Jenkins.
- Experience in Scrum Agile delivery environment.
- A strong desire to continue to grow knowledge & skillset.
- Strong communication, presentation skills that are influential and convincing.
- Excellent analytical and problem-solving skills.
Nice to have
- Understanding of Infrastructure as Code (e.g., Terraform) and data quality frameworks will be an added advantage.
- Experience with application monitoring (Datadog, or equivalent) will be an added advantage.
- Familiarity with streaming/batch patterns (Lambda/Kappa) and Medallion Architecture principles will be an added advantage.
Technologies
- Agile
- AS/400
- Bash
- BigQuery
- CI/CD
- Data Engineering
- Data Ingestion
- Data Warehousing
- Datadog
- DevOps
- ELT
- ERP
- ETL
- Fivetran
- GCP
- GitHub
- GitLab
- Infrastructure as Code
- Jenkins
- AWS Lambda
- NoSQL
- Python
- SAP
- Scrum
- Serverless
- SQL
- Terraform
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka