- Company Name
- Sigmaways Inc
- Job Title
- Data Engineer (Python, Spark, AWS)
- Job Description
-
Job Title: Data Engineer (Python, Spark, AWS)
Role Summary: Design, build, and maintain scalable, secure data pipelines and supporting web application features that provide actionable cybersecurity risk visibility. Work cross‑functionally in an Agile environment to develop new features, optimize existing systems, and ensure high reliability and performance across distributed data platforms.
Expectations: • Deliver clean, maintainable, and well‑documented code. • Ensure data integrity, performance, and reliability at scale. • Collaborate effectively with product, QA, and operations teams. • Participate in on‑call production support and troubleshooting. • Continuous learning and application of emerging tools and best practices.
Key Responsibilities: • Develop and optimize Python scripts and Spark jobs for large‑scale data processing. • Build and enhance web application features, including risk assessment workflows and data visualizations. • Write efficient SQL queries against relational databases such as PostgreSQL. • Integrate with RESTful APIs and front‑end technologies (JavaScript, React, HTML, CSS). • Contribute to architectural decisions and improve system reliability. • Maintain code quality through peer reviews, unit/integration tests, and code style compliance. • Deploy and manage infrastructure using Docker, AWS services, Terraform, and Kubernetes. • Coordinate with QA to ensure comprehensive test coverage. • Participate in continuous integration/continuous delivery pipelines and on‑call rotation. • Stay current on industry trends and tools for data engineering and infrastructure.
Required Skills: • 5+ years of professional Python experience; scripting and data pipeline development. • Proficiency with Spark, Hadoop, and AWS data services (EMR, Glue, S3). • Deep understanding of SQL and experience with PostgreSQL. • Strong command of JavaScript and ability to develop dynamic front‑end interfaces. • Familiarity with Elixir, Ruby, React, HTML, CSS (full‑stack exposure). • Experience with Docker, Kubernetes, Terraform, and AWS infrastructure. • Knowledge of NoSQL (MongoDB, Elasticsearch) and stream processing (Kafka). • Solid grasp of software engineering fundamentals (data structures, design patterns, clean code). • Hands‑on experience in Agile (Scrum/Kanban) and DevOps practices. • Ability to troubleshoot and resolve production incidents.
Required Education & Certifications: • Bachelor’s degree in Computer Science, Information Technology, Computer Engineering, or related field; equivalent work experience accepted. • Optional certifications (e.g., AWS Certified Solutions Architect, AWS Certified Data Analytics).