- Company Name
- IO Associates
- Job Title
- Senior Data Engineer
- Job Description
-
Job title: Senior Data Engineer
Role Summary: Lead the design, development, and optimization of scalable data pipelines on Azure Databricks for economic and monetary analysis. Deliver high‑performance data ingestion, transformation, and loading solutions while ensuring data quality, governance, and cost efficiency.
Expectations: • 8+ years of data engineering experience with 3+ years in Azure Databricks. • SC Clearance required. • Proven ability to work independently and collaborate with data scientists, economists, and technical stakeholders.
Key Responsibilities:
- Design, build, and maintain robust, scalable data pipelines from diverse sources (APIs, databases, financial feeds) into Azure Databricks.
- Optimize pipeline performance, efficiency, and cost; implement monitoring, alerting, and lineage tracking.
- Develop complex transformations using PySpark or Scala, and implement data cleaning, enrichment, and aggregation logic.
- Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related Databricks services; apply best‑practice deployment strategies.
- Integrate relational, API, and streaming data sources; collaborate with API developers for seamless data exchange.
- Enforce data quality and governance through Azure Purview, ensuring compliance with policies and standards.
- Automate deployment and data engineering workflows; build CI/CD pipelines with DevOps teams (Azure DevOps, Jenkins, Docker, Kubernetes).
- Participate in code reviews, knowledge sharing, and agile sprint activities; translate technical concepts to non‑technical audiences.
Required Skills:
- Advanced proficiency in Python, Spark (PySpark/Scala), SQL, and NoSQL databases.
- Deep experience with Azure services: Data Factory, Blob Storage, SQL Database, Event Hubs/Kafka, Purview, and Databricks.
- Strong knowledge of data warehousing, modeling, integration patterns, and large‑dataset processing.
- Expertise in data quality, governance, version control (Git), and CI/CD practices.
- Familiarity with streaming technologies, visualization tools (Tableau, Power BI), and container orchestration (Kubernetes).
- Excellent problem‑solving, communication, and collaboration skills.
Required Education & Certifications:
- Bachelor’s or Master’s degree in Computer Science, Data Engineering, or related field.
- Relevant Azure certifications (e.g., Azure Data Engineer Associate, Azure Databricks Specialist).
- SC Clearance (essential).