- Company Name
- Vivid Resourcing
- Job Title
- Senior Data Engineer
- Job Description
-
Job title: Senior Data Engineer
Role Summary: Lead the design, development, and maintenance of a cloud‑native, AWS‑based data platform utilizing PySpark to support real‑time operational feeds and large‑scale analytical workloads for the renewable energy domain.
Expectations: Deliver scalable ingestion pipelines, efficient data models, and seamless integration with production systems; apply modern data engineering best practices; collaborate cross‑functionally and potentially mentor junior engineers.
Key Responsibilities:
• Architect and implement data lake and warehouse structures on AWS (S3, Glue, Redshift, Lake Formation, Kinesis, Athena).
• Build and maintain ingestion frameworks for APIs, streaming sources, and external feeds such as weather data, ensuring near real‑time delivery.
• Develop and optimize PySpark jobs, data transformations, and schemas to meet operational and analytical query performance.
• Integrate data solutions with software engineering teams’ production systems and APIs.
• Automate orchestration, monitoring, and incident response for data pipelines.
• Conduct performance tuning, scalability planning, and architecture reviews.
• Enforce data quality, security standards, and compliance throughout the data lifecycle.
Required Skills:
- Strong command of AWS services for data engineering (s3, glue, redshift, lake formation, kinesis, athena).
- Advanced PySpark and distributed data processing expertise.
- Proficiency in Python, SQL, and data modeling for high‑volume and high‑frequency workloads.
- Experience with ETL/ELT pipeline design, orchestration tools (Airflow, Step Functions), and workflow automation.
- Knowledge of real‑time streaming platforms (Kafka, Kinesis) and message broker integration.
- Familiarity with data lakehouse concepts and data warehouse architecture.
- Ability to write clean, maintainable, and tested code; strong problem‑solving and communication skills.
Required Education & Certifications:
- Bachelor’s degree or equivalent in Computer Science, Information Technology, Data Engineering, or a related technical field.
- AWS certifications (e.g., Certified Data Analytics – Specialty, Solutions Architect, or Data Engineer) preferred but not mandatory.