- Company Name
- MindSource
- Job Title
- Data Engineer
- Job Description
-
Job title: Data Engineer
Role Summary: Design, build, and maintain scalable data pipelines and infrastructure to support analytical workloads, ensuring performance, reliability, and data quality across large datasets.
Expactations: Deliver robust, production‑ready data solutions using modern data platforms and programming languages; collaborate with cross‑functional teams to translate business requirements into technical specifications; maintain adherence to SDLC practices, version control, and CI/CD pipelines; continuously monitor, tune, and troubleshoot data pipelines and downstream services.
Key Responsibilities:
- Develop, deploy, and monitor data ingestion, transformation, and delivery pipelines with Airflow, Spark, Trino, Kafka, or equivalent tools.
- Write and optimize complex SQL queries for large relational and Big Data stores.
- Implement data quality checks, schema validation, and lineage tracking.
- Integrate new data sources and APIs into existing workflows.
- Collaborate with data scientists, analysts, and stakeholders to refine data models and reporting needs.
- Manage version control (Git) and automate builds, tests, and deployments via CI/CD workflows.
- Troubleshoot performance bottlenecks, data inconsistencies, and system failures.
Required Skills:
- 6+ years of software or data engineering experience with a strong focus on SQL and large datasets.
- Proficiency in Python, Java, or Scala.
- Hands‑on experience with Airflow, Spark, Trino, Kafka, or similar technologies.
- Strong analytical and problem‑solving abilities.
- Knowledge of SDLC best practices, version control, and CI/CD.
- Familiarity with cloud data services (AWS, GCP, Azure) is a plus.
Required Education & Certifications:
- Bachelor’s degree in Computer Science, Engineering, or a related field, or equivalent professional experience.
- Relevant certifications (e.g., AWS Certified Data Analytics, GCP Professional Data Engineer) are advantageous but not mandatory.