- Company Name
- Artech L.L.C.
- Job Title
- Big Data Engineer
- Job Description
-
**Job Title:** Big Data Engineer
**Role Summary:**
Lead the design, development, and deployment of enterprise‑scale big data solutions using Hadoop, Spark, and related technologies. Manage a team of data engineers, ensure performance, quality, and compliance, and integrate data pipelines across cloud and on‑prem environments.
**Expectations:**
- Deliver high‑performance big data platforms within defined timelines.
- Mentor and grow a multi‑professional engineering team.
- Communicate effectively with technical and non‑technical stakeholders.
- Maintain industry‑standard security, compliance, and documentation practices.
**Key Responsibilities:**
- Lead a data engineering team through the full software development lifecycle.
- Architect and implement data pipelines using Hadoop (Cloudera), Hive, Impala, Kafka, HBase, and Spark.
- Develop and maintain batch and streaming solutions in Scala (Java background preferred).
- Design and enforce coding standards, design patterns, and best practices.
- Build and manage CI/CD pipelines with Jenkins, TeamCity, SonarQube, Kubernetes/Openshift, and cloud services (AWS EMR, S3, Snowflake, GCP).
- Conduct performance troubleshooting, capacity planning, and risk assessments.
- Facilitate agile ceremonies (sprints, backlogs, retrospectives) and participate in product planning.
- Mentor junior developers and promote knowledge sharing.
- Liaise with data scientists, product owners, and operations to align data strategies.
- Ensure compliance with data governance, security policies, and regulatory requirements.
**Required Skills:**
- 8–12 years of experience in big data application development.
- Deep expertise in Hadoop ecosystem (Cloudera/Hortonworks), Hive, Impala, Kafka, HBase, Spark.
- Strong Scala development; Java proficiency preferred.
- Experience with batch/stream processing, event‑based messaging, and real‑time analytics.
- Proficiency in relational SQL (Oracle, SQL Server), NoSQL (MongoDB), and caching systems (Couchbase).
- Data analysis and visualization (e.g., Tableau, Power BI).
- DevOps practices: CI/CD tools, container orchestration, monitoring.
- Agile methodology experience (Scrum/SAFe).
- Excellent written and verbal communication skills.
- Leadership, mentoring, stakeholder engagement, risk management.
- Familiarity with cloud platforms: AWS EMR, S3, Snowflake, GCP.
- API development using JSON/XML hypermedia.
**Required Education & Certifications:**
- Bachelor’s degree in Computer Science, Engineering, or related field.
- Master’s degree preferred.
- Relevant certifications in Big Data technologies (e.g., Cloudera Certified, Spark, AWS Big Data) are a plus.