Senior Data Engineer
About the role
The Senior Data Engineer will lead the architecture, design, and deployment of enterprise data infrastructure, taking ownership of complex data pipelines and driving architectural decisions. The role involves building scalable, secure, and production-ready data ecosystems, bridging data engineering, backend services, and MLOps. Key responsibilities include designing and managing big data environments, developing backend and API integrations, orchestrating workflows, deploying containerized applications, supporting MLOps practices, and mentoring junior engineers. The ideal candidate will have significant hands-on experience in data engineering, strong programming skills, and expertise in big data, cloud platforms, and infrastructure management.
Must have
- Experience: Minimum 5+ years of hands-on experience in Data Engineering.
- Lead the small team and guide junior engineers.
- Best Practices need to follow on production ready setup.
- Efficient code design pattern usage.
- Multithreading knowledge.
- Expertise in git end point version handling.
- Heavily hands on experience expecting.
- Exposure to Google Cloud Platform services and architecture. (or any other vendor)
- Programming: Strong proficiency in Python. Solid experience with backend frameworks like FastAPI and Spring Boot.
- Big Data & Streaming: Extensive experience with Hadoop environments, Apache Spark, Kafka, and NiFi.
- ETL/Orchestration: Deep expertise in ETL pipeline creation, productionizing data workflows, Airflow, and dbt.
- Databases: Advanced SQL skills with deep knowledge of DB2 SQL and PostgreSQL. Capable of handling massive time- series datasets.
- Infrastructure & Security: Proficiency with Linux CLI, Docker, Kubernetes, network configuration, and OAuth2 implementation.
- MLOps: Proven knowledge and hands-on experience with MLOps principles and productionizing machine learning
Technologies
- Airflow
- CLI
- Data Engineering
- Data Pipelines
- Db2
- dbt
- DevOps
- Docker
- ELT
- ETL
- FastAPI
- GCP
- Git
- Hadoop
- Kafka
- Kubernetes
- Linux
- Machine Learning
- MLOps
- Model Deployment
- Multithreading
- NiFi
- OAuth
- PostgreSQL
- Python
- Spark
- Spring / Spring Boot
- SQL
- Time Series
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka