Senior Data Engineer (Azure Databricks / PySpark)
About the role
The Senior Data Engineer will be responsible for developing and supporting scalable data pipelines that connect enterprise source systems with Master Data Management (MDM) and governed downstream platforms. The role involves data ingestion, standardization, transformation, and ensuring data quality for reliable downstream consumption. Key responsibilities include building ingestion pipelines for various data sources, developing PySpark/Spark transformations on Azure Databricks, implementing data patterns, optimizing Spark jobs, writing automated tests, and supporting production monitoring and troubleshooting. The company is an award-winning software product engineering firm with a strong focus on AI-driven innovation and long-term global client partnerships.
Must have
- Overall 7+ years of experience and 5+ years data engineering with strong production PySpark/Spark experience.
- Strong Azure Databricks, Delta Lake, SQL and ADLS Gen2.
- REST/API ingestion and database integration experience.
- Incremental/CDC processing and schema-evolution experience.
- Strong data-quality/testing practices and Git/CI/CD.
- Experience handling large, messy enterprise datasets
Nice to have
- Databricks certification.
- MDM/entity-resolution project exposure.
- Azure Data Factory or equivalent orchestration.
- Exposure to ML-based entity resolution / fuzzy matching (PySpark MLlib), including deterministic and probabilistic matching design
- Purview/Unity Catalog governance exposure.
Technologies
- Azure
- Azure Data Factory
- CI/CD
- Data Engineering
- Data Ingestion
- Data Pipelines
- Databricks
- Delta Lake
- Git
- Machine Learning
- MDM
- PySpark
- REST
- Spark
- SQL
- Unity
Highlights
This is a 2-year contract position with a hybrid work arrangement requiring 3 to 4 days onsite per week. Working hours are Monday to Friday, 9:00 a.m. to 6:00 p.m. Good English communication skills (minimum 7/10 proficiency) are required.
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka