Data Engineer
About the role
The Engineer – Data Engineering role involves designing and developing scalable, enterprise-grade data-processing solutions for large-scale business requirements. The position requires building distributed and highly available data applications, developing and maintaining data pipelines, and working with both batch and near-real-time data processing. The engineer will use Google Cloud Platform services and tools such as Airflow, BigQuery, Dataflow, and others, and may also support AWS data-processing solutions. The role includes responsibilities for developing data-processing applications using Python, Shell scripting, and SQL, as well as contributing to data warehouse and data lake solutions. The position is offered by InTalent Asia for a client in the software industry in Sri Lanka, on a contract basis.
Must have
- Bachelor's degree in Computer Science, Software Engineering, Information Technology, or an equivalent field.
- Minimum 1 to 2 years of experience developing enterprise-grade data-processing applications.
- Strong programming skills in Python, Shell scripting, and SQL.
- Hands-on experience processing large volumes of data.
- Experience designing and developing ETL or ELT data pipelines.
- Practical experience with relational databases, NoSQL databases, and distributed storage engines.
- Hands-on experience with Google Cloud Platform.
- Experience with GCP services such as BigQuery, Dataflow, Dataproc, Datastream, Pub/Sub, Cloud Functions, Cloud Run, and Cloud Composer.
- Familiarity with ETL and orchestration tools such as Airflow.
- Experience with source-control and CI/CD tools such as GitHub, GitLab, and Jenkins.
Nice to have
- Exposure to AWS data services such as Glue, Lambda, EMR, Spark, Hive, or Data Pipeline will be an advantage.
- Experience working with streaming technologies such as Kafka, Pub/Sub, Storm, or Spark Streaming will be an advantage.
- Exposure to application-monitoring tools such as Datadog or an equivalent platform will be an advantage.
Technologies
- Airflow
- AWS
- Bash
- BigQuery
- CI/CD
- Cloud Native
- Data Engineering
- Data Lake
- Data Pipelines
- Data Warehousing
- Datadog
- ELT
- ETL
- GCP
- GitHub
- GitLab
- Hive
- Jenkins
- Kafka
- AWS Lambda
- NoSQL
- Python
- Spark
- SQL
Highlights
The role is based in Sri Lanka and is a contract position. The recruitment is managed by InTalent Asia for a client in the software industry. Application instructions are not specified in the listing.
Also posted as
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka