- Company Name
- DATATORII
- Job Title
- Data Engineer Databricks
- Job Description
-
**Job Title**
Data Engineer – Databricks
**Role Summary**
Design, build, and maintain end‑to‑end data pipelines and platforms on Microsoft Azure and Azure Databricks. Collaborate with cross‑functional teams to deliver high‑value analytics solutions, ensuring robust data architecture, security, and CI/CD practices.
**Expectations**
- Deliver reliable, scalable data solutions using Azure services (Azure Data Factory, SQL Database, Synapse).
- Drive data industrialization with ETL/ELT processes in PySpark, SparkSQL, Python, SQL, and Scala.
- Implement DevOps pipelines (GitHub, Azure DevOps, Databricks Bundles) for continuous integration and deployment.
- Participate in project scoping, data modeling, and solution definition, providing technical guidance to clients and internal stakeholders.
- Support innovation in UX BI, self‑service BI, and data engineering offerings, contributing to business proposals and client engagements.
- Continually improve skill set and share knowledge within the team.
**Key Responsibilities**
1. Develop and maintain Azure-based data pipelines and data platform solutions.
2. Build and manage Unity Catalog, Lakehouse, and Lakeflow architectures with Azure Databricks.
3. Design and implement ETL/ELT workflows using PySpark, SparkSQL, Python, SQL, Scala, and DBT.
4. Configure and automate CI/CD pipelines (GitHub, Azure DevOps, Databricks Bundles).
5. Ensure data governance, security, and compliance throughout the data lifecycle.
6. Collaborate with project managers, architects, and data analysts to gather requirements and shape technical solutions.
7. Contribute to project management, including scoping, estimation, and technology decisions.
8. Draft and present custom solutions and proposals to clients.
9. Mentor teammates, run internal trainings, and contribute to continuous learning initiatives.
**Required Skills**
- Expertise in Microsoft Azure services (Data Factory, SQL Database, Synapse, Fabric, Azure DevOps).
- Proficiency in Azure Databricks administration, Unity Catalog, and Databricks Apps.
- Strong command of SQL, Python, PySpark, SparkSQL, Scala, and DBT.
- Experience implementing CI/CD pipelines for data projects.
- Knowledge of data architecture, data management, and data governance principles.
- Project management experience in data engineering contexts.
- Excellent communication, collaboration, and knowledge‑sharing abilities.
- Proactive, detail‑oriented, and analytical mindset.
**Required Education & Certifications**
- Master’s degree (Bac +5) in Engineering, Data Engineering, Machine Learning, Mathematics, Statistics, or Computer Science.
- Relevant certifications (e.g., Microsoft Certified: Azure Data Engineer Associate, Databricks Certified Data Engineer) preferred.