Data Engineer
About the role
The Engineer – Data Engineering role involves designing and developing scalable, enterprise-grade data-processing solutions. The position requires building distributed and highly available data applications, developing and maintaining data pipelines, and working with both batch and near-real-time data processing. The engineer will use Google Cloud Platform services, as well as tools like Airflow and Cloud Composer, and may also support AWS-based data-processing solutions. Responsibilities include developing applications using Python, Shell scripting, and SQL, optimizing solutions with various databases, and supporting streaming-data applications. The role also involves contributing to data warehouses, data lakes, and applying software engineering best practices.
Must have
- Bachelor's degree in Computer Science, Software Engineering, Information Technology, or an equivalent field.
- Minimum 1 to 2 years of experience developing enterprise-grade data-processing applications.
- Strong programming skills in Python, Shell scripting, and SQL.
- Hands-on experience processing large volumes of data.
- Experience designing and developing ETL or ELT data pipelines.
- Practical experience with relational databases, NoSQL databases, and distributed storage engines.
- Hands-on experience with Google Cloud Platform.
- Experience with GCP services such as BigQuery, Dataflow, Dataproc, Datastream, Pub/Sub, Cloud Functions, Cloud Run, and Cloud Composer.
- Familiarity with ETL and orchestration tools such as Airflow.
- Experience with source-control and CI/CD tools such as GitHub, GitLab, and Jenkins.
Nice to have
- Exposure to AWS data services such as Glue, Lambda, EMR, Spark, Hive, or Data Pipeline will be an advantage.
- Experience working with streaming technologies such as Kafka, Pub/Sub, Storm, or Spark Streaming will be an advantage.
- Exposure to application-monitoring tools such as Datadog or an equivalent platform will be an advantage.
Technologies
- Airflow
- AWS
- Bash
- BigQuery
- CI/CD
- Cloud Native
- Data Engineering
- Data Lake
- Data Pipelines
- Data Warehousing
- Datadog
- ELT
- ETL
- GCP
- GitHub
- GitLab
- Hive
- Jenkins
- Kafka
- AWS Lambda
- NoSQL
- Python
- Spark
- SQL
Highlights
This is a contract position based in Sri Lanka. The role is facilitated by InTalent Asia for their client. Candidates should have 1 to 2 years of relevant experience. Exposure to AWS data services, streaming technologies, and application-monitoring tools is considered advantageous.
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka