Senior Consultant - Databricks
About the role
As a Senior Consultant - Databricks, you will design, develop, deploy, and support scalable batch and streaming data pipelines using Azure Databricks, Spark, PySpark, Python, and SQL. The role involves building Lakehouse solutions, developing ETL/ELT frameworks, supporting data platform modernization, and applying best practices in data modeling, governance, and orchestration. You will also optimize Spark workloads, implement secure access controls, develop CI/CD pipelines, collaborate with cross-functional teams, and mentor junior engineers. The position requires a strong background in data engineering and Databricks, with a mandatory Databricks Professional Certification.
Must have
- Bachelor's degree in Computer Science, Data Science, or a related discipline
- 5+ years of experience in Data Engineering, Analytics, BI, or Software Engineering
- 2+ years of enterprise Databricks experience
- Strong expertise in Spark, PySpark, Python, SQL, Delta Lake, Databricks SQL, Notebooks, Workflows, and Repos
- Experience with Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Synapse, Microsoft Fabric, and Power BI
- Experience with CI/CD, Git, automated testing, and release management
- Strong analytical, communication, documentation, customer engagement, and teamwork skills
- Ability to work independently and mentor junior engineers in Agile environments
- Databricks Professional Certification is mandatory.
Nice to have
- Databricks Delivery Expert / Project Expert Certification
- Azure / Microsoft Fabric Certifications
- Exposure to Asset Bundles, IaC, MLflow, MLOps, GenAI, Purview, data observability, and event streaming
- Experience in presales, estimations, POCs, and customer demonstrations
Technologies
- Agile
- Azure
- Azure Data Factory
- Azure DevOps
- CI/CD
- Data Engineering
- Data Lake
- Data Modeling
- Data Pipelines
- Data Science
- Databricks
- Delta Lake
- ELT
- ETL
- Generative AI
- Git
- Infrastructure as Code
- Lakehouse
- Machine Learning
- Microsoft Fabric
- MLflow
- MLOps
- Observability
- Power BI
- PySpark
- Python
- Spark
- SQL
- Test Automation
- Unity
Highlights
Databricks Professional Certification is mandatory. Additional certifications such as Databricks Delivery Expert, Azure, or Microsoft Fabric are advantageous. Experience in presales, POCs, and customer demos is a plus.
Jobs across Sri Lanka, watched for you.Colombo, Sri Lanka