Data Engineer
About the role
The Data Engineer role at Trident Recruitment Solutions involves developing and maintaining scalable, high-availability batch and streaming data pipelines on Google Cloud Platform (GCP), including low-latency Change Data Capture (CDC) solutions. The position is responsible for ensuring code quality, system reliability, security, and performance, as well as implementing automated CI/CD pipelines and following DevOps practices. The engineer will optimize cloud resource utilization, resolve security vulnerabilities, maintain technical documentation, and collaborate with cross-functional Agile teams to deliver data engineering solutions, leveraging AI-assisted tools to improve productivity and automate processes. Trident Recruitment Solutions is a specialized talent partner focused on delivering skilled professionals to leading organizations, offering opportunities to work on impactful projects.
Must have
- A bachelor’s degree in computer science, IT or equivalent qualification.
- Preferably 1-2 years of experience in developing enterprise grade data processing applications.
- Sound knowledge of Data engineering practices and processes
- Solid programming skills in Python, SQL and Shell/Bash.
- Hands-on experience working with relational/NoSQL databases.
- Experience in building and maintaining ETL/ELT pipeline using Ingestion platform (Fivetran), frameworks & orchestration tools
- Hands-on experience in implement, and maintain low-latency Change Data Capture (CDC) pipelines to stream data from source relational databases into enterprise data warehouse
- Hands-on experience with Google Cloud Platform (GCP): Experience with services & serverless functions (e.g., BigQuery, Datastream, DataFlow, Pub/Sub, Cloud Functions, Cloud Run, Cloud composer, etc.) for data processing and orchestration.
- Hands-on experience with data ingestion and extraction from ERP systems like AS400, SAP, and Dynamics 365.
- Experience in code management, DevOps practices & CI/CD tools such as Github, Gitlab and Jenkins.
- Experience in Scrum Agile delivery environment.
- A strong desire to continue to grow knowledge & skillset.
- Strong communication, presentation skills that are influential and convincing.
- Excellent analytical and problem-solving skills.
Nice to have
- Understanding of Infrastructure as Code (e.g., Terraform) and data quality frameworks will be an added advantage.
- Experience with application monitoring (Datadog, or equivalent) will be an added advantage.
- Familiarity with streaming/batch patterns (Lambda/Kappa) and Medallion Architecture principles will be an added advantage
Technologies
- Agile
- AS/400
- Bash
- BigQuery
- CI/CD
- Data Engineering
- Data Ingestion
- Data Pipelines
- Data Warehousing
- Datadog
- DevOps
- ELT
- ERP
- ETL
- Fivetran
- GCP
- GitHub
- GitLab
- Infrastructure as Code
- Jenkins
- AWS Lambda
- NoSQL
- Python
- SAP
- Scrum
- Serverless
- SQL
- Terraform
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka