- Company Name
- Euroclear
- Job Title
- Azure Data Platform Engineer (Databricks Focus)
- Job Description
-
**Job Title:** Azure Data Platform Engineer (Databricks Focus)
**Role Summary:**
Design, deploy, and operate scalable, secure Azure Databricks environments across multiple subscriptions. Integrate Databricks with core Azure services, optimize performance and cost, enforce security and governance, and automate infrastructure using Terraform and CI/CD pipelines.
**Expectations:**
- Deliver production‑grade Databricks platforms that meet reliability, scalability, and cost‑efficiency goals.
- Enable seamless data ingestion, batch/streaming pipelines, and analytics workloads.
- Apply robust security, compliance, and governance controls.
- Automate platform provisioning and configuration; maintain best‑practice IaC and CI/CD practices.
- Serve as a technical leader and subject‑matter expert, providing guidance to cross‑functional teams.
**Key Responsibilities:**
- Operate and evolve Azure Databricks workspaces, clusters, pools, job orchestration, and workspace configurations.
- Engineer Azure infrastructure components (networking, identity, storage, security) that support Databricks.
- Design and implement integrations with ADLS Gen2, Delta Lake, Unity Catalog, Azure SQL, Synapse, Event Hubs, and other data sources.
- Monitor platform health, performance, and usage; proactively address operational issues and drive improvements.
- Design and enforce security & governance controls: Azure AD, RBAC, SCIM, Unity Catalog policies, Azure Key Vault, Private Endpoints, network security.
- Develop and maintain Terraform‑based IaC for Databricks and supporting Azure services across multi‑subscription, multi‑environment deployments.
- Build and refine Azure DevOps CI/CD pipelines (YAML) for platform and infrastructure deployments.
- Create and maintain documentation, reference architectures, and runbooks.
- Mentor and collaborate with Cloud, Architecture, CI/CD, Data, AI, and Application teams.
**Required Skills:**
- 5+ years of Azure platform engineering with a strong focus on Azure Databricks.
- Deep expertise in Databricks (clusters, jobs, pools, Delta Lake, Unity Catalog, workspace engineering).
- Proficiency with Azure core services: Storage, Networking, Identity & Access Management.
- Experience with Terraform (multi‑subscription, multi‑environment, modular IaC, automated testing).
- CI/CD pipeline design and operation (Azure DevOps YAML preferred).
- Cloud security, governance, and platform engineering best practices.
- Strong communication, documentation, and collaboration abilities.
**Required Education & Certifications:**
- Bachelor’s degree in Computer Science, Engineering, or related field (or equivalent experience).
- Azure Certifications: Azure Administrator Associate, Azure Solutions Architect Expert.
- Databricks Certification: Databricks Certified Data Platform Engineer or Administrator.
- HashiCorp Terraform Certifications: Terraform Associate (003+) or Terraform Engineer.