Engineer - Data Engineering
About the role
The Engineer - Data Engineering role involves developing and maintaining enterprise-grade data processing applications. Key responsibilities include building and maintaining ETL/ELT pipelines, implementing low-latency Change Data Capture (CDC) pipelines, and working with various data platforms and orchestration tools. The ideal candidate will have a bachelor's degree in computer science, IT, or a related field, and preferably 1-2 years of relevant experience. The role requires strong programming skills in Python, SQL, and Shell/Bash, as well as hands-on experience with relational and NoSQL databases, Google Cloud Platform services, and data ingestion from ERP systems. Experience with DevOps practices, CI/CD tools, and working in a Scrum Agile environment is also important. The company context, benefits, and application process are not specified in the listing.
Must have
- A bachelor's degree in computer science, IT or equivalent qualification.
- Preferably 1-2 years of experience in developing enterprise grade data processing applications.
- Sound knowledge of Data engineering practices and processes.
- Solid programming skills in Python, SQL and Shell/Bash.
- Hands-on experience working with relational/NoSQL databases.
- Experience in building and maintaining ETL/ELT pipeline using Ingestion platform (Fivetran), frameworks & orchestration tools.
- Hands-on experience in implementing and maintaining low-latency Change Data
- Capture (CDC) pipelines to stream data from source relational databases into enterprise data warehouse.
- Hands-on experience with Google Cloud Platform (GCP): Experience with services & serverless functions (e.g., BigQuery, Datastream, Dataflow, Pub/Sub, Cloud Functions, Cloud Run, Cloud Composer, etc.) for data processing and orchestration.
- Hands-on experience with data ingestion and extraction from ERP systems like AS400, SAP, and Dynamics 365.
- Experience in code management, DevOps practices & CI/CD tools such as GitHub, GitLab and Jenkins.
- Understanding of Infrastructure as Code (e.g., Terraform) and data quality frameworks will be an added advantage.
- Experience with application monitoring (Datadog, or equivalent) will be an added advantage.
- Familiarity with streaming/batch patterns (Lambda/Kappa) and Medallion Architecture principles will be an added advantage.
- Experience in Scrum Agile delivery environment.
Technologies
- Agile
- AS/400
- Bash
- BigQuery
- CI/CD
- Data Engineering
- Data Ingestion
- Data Warehousing
- Datadog
- DevOps
- ELT
- ERP
- ETL
- Fivetran
- GCP
- GitHub
- GitLab
- Infrastructure as Code
- Jenkins
- AWS Lambda
- NoSQL
- Python
- SAP
- Scrum
- Serverless
- SQL
- Terraform