- Company Name
- NTT DATA, Inc.
- Job Title
- [Staffing/Freelance] AI/Data Engineer
- Job Description
-
Job title: AI/Data Engineer
Role Summary: Design, build, and operate scalable data lakehouse and GenAI solutions on Azure and Databricks, automating pipelines via Terraform and CI/CD, while ensuring governance, security, and performance.
Expectations: Deliver production‑grade data and AI services on schedule, maintain high code quality, collaborate with stakeholders, and continuously improve infrastructure and processes.
Key Responsibilities:
- Build batch and streaming pipelines in Databricks (Python/PySpark, SQL) with Delta Lake and Unity Catalog.
- Apply code‑quality best practices, testing, documentation, lineage, and cost‑efficient performance tuning.
- Design GenAI workloads using Azure AI Foundry, Azure OpenAI, and Azure AI Search, implementing RAG patterns, grounding, safety, and evaluation metrics.
- Deploy models with MLflow, manage experiments, packaging, and model serving.
- Create CI/CD pipelines (Azure DevOps/GitHub Actions) for data and AI workloads, integrating monitoring, logs, and drift detection.
- Provision Azure and Databricks infrastructure via Terraform (modular design, state, policy enforcement).
- Enforce security and governance: RBAC, secrets, data masking, Unity Catalog/Purview, privacy‑by‑design compliance.
- Partner with product owners, architects, and SMEs to translate use‑cases into secure, cost‑aware solutions.
Required Skills:
- Python production coding, PySpark, SQL, unit testing.
- Databricks Repos/Workflows, Delta Lake, Unity Catalog, MLflow.
- Azure AI Foundry (projects, prompt flows, evaluation), Azure OpenAI (prompt design, safety filters, function calling), Azure AI Search (vector indexing, semantic ranking).
- Terraform (AzureRM & Databricks providers), modular IaC patterns.
- Git, CI/CD (Azure DevOps or GitHub Actions), code review, trunk‑based or GitFlow.
- Strong problem‑solving, communication, ownership, and stakeholder collaboration.
Required Education & Certifications:
- 3–7+ years data/AI engineering experience on Azure.
- BSc/MSc in Computer Science, Data Engineering, AI, or equivalent.
- Certifications (preferred): Microsoft Azure Data Engineer Associate, Azure AI Engineer Associate; Databricks Lakehouse Fundamentals/Data Engineer/Machine Learning; HashiCorp Terraform Associate.