Engineer - Data Engineering
About the role
The Engineer - Data Engineering role involves developing distributed, high-availability data processing applications and end-to-end batch/streaming pipelines on Google Cloud Platform (GCP) to meet enterprise requirements. The position is responsible for implementing and maintaining low-latency Change Data Capture (CDC) pipelines, ensuring code quality and system stability, building CI/CD pipelines, and adhering to DevOps principles throughout the software lifecycle. The engineer will also focus on cloud resource efficiency, documentation, and collaboration within Agile/Scrum teams, leveraging AI-assisted tools to enhance productivity. Candidates should have a relevant bachelor's degree, programming skills in Python, SQL, and Shell/Bash, experience with relational/NoSQL databases, and hands-on experience with GCP and data ingestion from ERP systems. Experience with DevOps tools, ETL/ELT pipelines, and Agile environments is also required.
Must have
- A bachelor’s degree in computer science, IT or equivalent qualification.
- Preferably 1-2 years of experience in developing enterprise grade data processing applications.
- Sound knowledge of Data engineering practices and processes
- Solid programming skills in Python, SQL and Shell/Bash.
- Hands-on experience working with relational/NoSQL databases.
- Experience in building and maintaining ETL/ELT pipeline using Ingestion platform (Fivetran), frameworks & orchestration tools
- Hands-on experience in implement, and maintain low-latency Change Data Capture (CDC) pipelines to stream data from source relational databases into enterprise data warehouse
- Hands-on experience with Google Cloud Platform (GCP): Experience with services & serverless functions (e.g., BigQuery, Datastream, DataFlow, Pub/Sub, Cloud Functions, Cloud Run, Cloud composer, etc.) for data processing and orchestration.
- Hands-on experience with data ingestion and extraction from ERP systems like AS400, SAP, and Dynamics 365.
- Experience in code management, DevOps practices & CI/CD tools such as Github, Gitlab and Jenkins.
- Experience in Scrum Agile delivery environment.
- A strong desire to continue to grow knowledge & skillset.
- Strong communication, presentation skills that are influential and convincing.
- Excellent analytical and problem-solving skills.
Nice to have
- Understanding of Infrastructure as Code (e.g., Terraform) and data quality frameworks will be an added advantage.
- Experience with application monitoring (Datadog, or equivalent) will be an added advantage.
- Familiarity with streaming/batch patterns (Lambda/Kappa) and Medallion Architecture principles will be an added advantage
Technologies
- Agile
- AS/400
- Bash
- BigQuery
- CI/CD
- Data Engineering
- Data Ingestion
- Data Warehousing
- Datadog
- DevOps
- ELT
- ERP
- ETL
- Fivetran
- GCP
- GitHub
- GitLab
- Infrastructure as Code
- Jenkins
- AWS Lambda
- NoSQL
- Python
- SAP
- Scrum
- Serverless
- SQL
- Terraform
Also posted as
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka