Job Specifications
Senior Data Engineer
Location - London
Hybrid - 3 days in the office
SC Clearance - ESSENTIAL
Rate 550-600 per day
Experience - 8+ years
We are seeking a highly skilled and experienced SC Cleared Senior Data Engineer to join our client and play a key role in developing and maintaining their Azure Databricks platform for economic data. This platform underpins critical Monetary Analysis, Forecasting, and Modelling activities, making this position essential to their mission.Key Responsibilities
Data Pipeline Development & Optimisation
Design, develop, and maintain robust, scalable data pipelines for ingesting, transforming, and loading data from diverse sources (APIs, databases, financial data providers) into Azure Databricks.
Optimise pipelines for performance, efficiency, and cost-effectiveness.
Implement data quality checks and validation rules within pipelines.
Data Transformation & Processing
Implement complex transformations using Spark (PySpark or Scala) and related technologies.
Develop logic for data cleaning, enrichment, and aggregation.
Ensure data consistency and accuracy throughout the life cycle.
Azure Databricks Implementation
Work extensively with Unity Catalog, Delta Lake, Spark SQL, and other Databricks services.
Apply best practices for development and deployment.
Optimise workloads for performance and cost.
Program in SQL, Python, R, YAML, and JavaScript.
Data Integration
Integrate data from relational databases, APIs, and streaming sources.
Implement data integration patterns and best practices.
Collaborate with API developers for seamless data exchange.
Data Quality & Governance
Hands-on experience with Azure Purview for governance and quality.
Implement monitoring, alerting, lineage tracking, and metadata management.
Ensure compliance with data governance policies and standards.
Collaboration & Communication
Work closely with data scientists, economists, and technical teams to translate requirements into solutions.
Communicate technical concepts to technical and non-technical audiences.
Participate in code reviews and knowledge-sharing sessions.
Automation & DevOps
Automate pipeline deployments and data engineering tasks.
Build CI/CD pipelines with DevOps teams.
Promote and implement DevOps best practices.
Essential Skills & Experience
10+ years in data engineering, with 3+ years hands-on Azure Databricks experience.
Strong proficiency in Python and Spark (PySpark or Scala).
Deep understanding of data warehousing, modelling, and integration patterns.
Extensive experience with Azure services (Data Factory, Blob Storage, SQL Database).
Expertise in large datasets and complex pipelines.
Proven Databricks implementation experience and certifications.
Skilled in SQL and NoSQL databases.
Experience with data quality, governance, and version control (Git).
Familiarity with Agile methodologies.
Excellent communication and problem-solving skills.
Experience with streaming technologies (Kafka, Azure Event Hubs).
Knowledge of visualisation tools (Tableau, Power BI).
Experience with DevOps tools (Azure DevOps, Jenkins, Docker, Kubernetes).
Background in financial services or economic data environments.
Relevant Azure certifications (eg, Azure Data Engineer Associate).
About the Company
Specialising in recruitment across the tech, digital, engineering, and government and defence sectors, iO Associates have extensive experience in securing the best talent to create the products and infrastructure of the future. Our dedicated team of highly experienced and knowledgeable consultants provide bespoke solutions, working with businesses to find the perfect fit across contract, interim, search, and permanent appointments. We offer end-to-end support, from Statements of Work and IR35 determination processes to full ...
Know more