- Company Name
- Big Red Recruitment Midlands Ltd
- Job Title
- Databricks Consultant
- Job Description
-
Job Title: Databricks Consultant
Role Summary:
Act as a senior architect for a 3‑month discovery engagement to assess and design a target‑state Databricks lakehouse architecture. Deliver assessment, architecture design, and migration roadmap for transition from Azure Synapse, Informatica, and FOCUS to a modern lakehouse platform.
Expectations:
* Complete a structured assessment of the current data platform, documenting architecture and dependencies.
* Define a scalable, secure lakehouse architecture using Azure Databricks, Delta Lake, and Spark.
* Produce a phased migration roadmap prioritised by business value, with high‑level estimates and risk mitigation.
* Collaborate with internal data engineering and business stakeholders, documenting requirements and best practices.
Key Responsibilities:
1. **Discovery & Assessment** – Evaluate existing pipelines, transformations, orchestration, storage, and reporting in Azure Synapse, Informatica, FOCUS.
2. **Platform Review** – Identify technical debt, performance gaps, and optimisation opportunities.
3. **Stakeholder Engagement** – Gather analytical and operational data needs from business and technical teams.
4. **Current‑State Documentation** – Produce detailed diagrams and dependency maps.
5. **Target Architecture Design** – Create patterns for ingestion, transformation, modelling, serving, environment management, CI/CD, security, and governance (Unity Catalog).
6. **Migration Strategy** – Define approach for workloads in Synapse/Informatica, prioritise migration, re‑engineering, or retention.
7. **Roadmap & Estimation** – Deliver phased roadmap with sequencing, high‑level cost/effort estimates, and risk mitigation.
8. **Knowledge Transfer** – Provide architectural guidance and best practices to internal teams.
Required Skills:
* Proven experience architecting Azure Databricks lakehouse solutions.
* Experience leading data platform discovery and architecture engagements.
* Deep knowledge of Delta Lake, Spark, lakehouse concepts, and scalable data pipelines.
* Hands‑on experience with Azure Synapse, Informatica, and related migration scenarios.
* Expertise in designing security and governance models in Databricks, including Unity Catalog.
* Strong stakeholder management, documentation, and communication skills.
Required Education & Certifications:
* Bachelor’s degree in Computer Science, Information Systems, Data Engineering, or related field.
* Relevant certifications highly valued (e.g., Microsoft Certified: Azure Data Engineer Associate, Databricks Certified Professional Data Engineer).