- Company Name
- AVL
- Job Title
- Senior Software Engineer (Data & ML)
- Job Description
-
Job Title: Senior Software Engineer (Data & ML)
Role Summary: Lead design, development, and deployment of large‑scale ETL/ELT pipelines and machine‑learning models on Databricks, ensuring performance, scalability, and governance within Azure cloud environments.
Expectations: Own end‑to‑end architecture decisions, enforce clean code and MLOps best practices, deliver production‑ready solutions, and collaborate closely with data scientists and engineers.
Key Responsibilities:
- Build and optimize PySpark/Delta Lake ETL/ELT pipelines on Databricks.
- Develop, train, and production‑deploy TensorFlow/PyTorch models.
- Implement MLOps pipelines using MLflow, Unity Catalog, Azure DevOps CI/CD, and Databricks Asset Bundles.
- Perform performance tuning of Spark jobs, data pipelines, and ML workloads.
- Design cloud‑native architectures with Azure, Kubernetes, and Docker.
- Conduct automated testing, experiment tracking, and governance compliance.
- Partner with cross‑functional teams to move prototypes to production.
Required Skills:
- 4+ years of production experience with Databricks, PySpark, and Delta Lake.
- Strong Python OOP, packaging (uv, poetry, etc.), and clean‑code practices.
- Proven deployment of ML/DL models using TensorFlow or PyTorch.
- Advanced knowledge of statistics, time‑series analysis, and common data formats (CSV, JSON, Parquet, Delta).
- Expertise in Azure services: Azure Databricks, ADLS, AAD, Unity Catalog.
- CI/CD with Azure DevOps Pipelines, automated testing, and Databricks Asset Bundles.
- Experience with Kubernetes, Docker, and cloud‑native deployments.
- Familiarity with automotive data (vehicle signals, test benches, simulations) is a plus.
Required Education & Certifications:
- Master’s degree (or equivalent) in Computer Science, Data Engineering, or related field.
- Relevant certifications in Azure, Databricks, or MLOps are advantageous.