- Company Name
- Montash
- Job Title
- SC Cleared DevOps Engineer (Azure)
- Job Description
-
**Job Title**
SC Cleared DevOps Engineer (Azure)
**Role Summary**
A 12‑month contract role for an Azure DevOps Engineer with active SC clearance to design, automate, and operate large‑scale Databricks data and analytics solutions. Focus on CI/CD pipelines, infrastructure‑as‑code, cost optimisation, security, and operational reliability for PySpark/Delta Lake workloads.
**Expectations**
- Execute under strictly governed security standards.
- Deliver end‑to‑end automation for Databricks and Azure infrastructure.
- Maintain platform reliability, monitoring, and incident response.
- Drive FinOps practices and secure data governance.
**Key Responsibilities**
- Design, build, and maintain CI/CD pipelines for Databricks notebooks, jobs, and configuration across environments.
- Automate provisioning and configuration of Databricks workspaces and Azure resources (ADLS Gen2, Key Vault, managed identities, serverless services) using Terraform or equivalent IaC tools.
- Standardise workspace settings, cluster policies, libraries, secrets, and access controls.
- Implement monitoring, logging, alerting, and runbooks for platform health, job reliability, and pipeline performance.
- Optimize Spark and Delta Lake workloads: cluster tuning, autoscaling, adaptive execution, caching, and time‑travel table management.
- Support development of PySpark ETL and streaming pipelines.
- Implement data governance, lineage, and access controls via Unity Catalog.
- Integrate Databricks with Azure Data Lake Storage Gen2, Key Vault, and other serverless Azure services.
- Enforce RBAC, managed identities, and secrets management per public‑sector security standards.
- Collaborate with cloud architects, data engineers, analysts, and cross‑functional teams on solution design and documentation.
- Contribute to platform standards, reusable templates, and DevOps best practices.
**Required Skills**
- Proven Azure DevOps Engineer experience.
- Hands‑on expertise with Databricks Data Intelligence Platform.
- Experience building and maintaining CI/CD pipelines for cloud and data platforms.
- Strong background in Spark, PySpark, and Delta Lake (platform & operational).
- Proficiency in infrastructure‑as‑code (Terraform or equivalent).
- Azure services: ADLS Gen2, Key Vault, managed identities, serverless offerings.
- Troubleshooting skills for distributed, cloud‑based environments.
- Monitoring, logging, and alerting implementation.
- FinOps/cost optimisation practices.
- Incident management, root‑cause analysis, and runbook creation.
- Knowledge of Unity Catalog, data governance, and lineage.
- Familiarity with Synapse, Power BI, or similar analytics services.
- Experience in regulated or public‑sector environments.
- Strong communication and collaboration skills.
**Required Education & Certifications**
- Bachelor’s degree (or equivalent) in Computer Science, Engineering, or related field.
- Azure certifications such as AZ‑400 (DevOps Engineer), AZ‑220 (Azure IoT Developer), or AZ‑203.
- Databricks Certified Associate/Professional certification preferred.
- Active Security Clearance (SC).
- FinOps Certificate (preferred).