Job Specifications
Role: Data Engineer (Python)
Experience: 8–10 years
Location: Hybrid(Toronto)- 3 days a week
Long Term Contract(6 Months to start with)
Skills: Python, PostgreSQL, Microsoft Power BI
Overview:
We are seeking an experienced Data Engineer with strong Python and SQL expertise to build scalable, reliable data pipelines that transform semi‑structured data from ES (Elasticsearch) URLs into clean, analytics‑ready datasets. This role involves hands-on local development using Python, DBeaver, SQLite, Postgres, and Dremio, along with modeling data into structured tables to support downstream reporting in Power BI.
Key Responsibilities:
Data Ingestion & Transformation
Retrieve semi‑structured data from ES URLs and REST APIs (e.g., JSON/Elasticsearch topics).
Flatten, normalize, and model nested datasets into structured analytical tables.
Develop reproducible ETL/ELT pipelines in Python using pandas, requests, and SQLAlchemy.
Database Engineering
Design, create, and maintain schemas in SQLite, Postgres, and Dremio.
Configure and manage local database connections through DBeaver.
Optimize queries, indexing strategies, and overall database performance.
Implement data partitioning, incremental loads, and performance tuning techniques.
Data Quality & Governance
Define validation rules, deduplication logic, and anomaly detection checks.
Manage dataset versioning, maintain data lineage, and document data contracts and metadata.
Ensure secure handling and storage of credentials, tokens, and API endpoints.
Use Git for version control; support code reviews, unit testing, and CI processes.
Create clear technical documentation, operational runbooks, and provide support for ad hoc data requests.
Required Skills & Experience:
Python for Data Engineering/Data Science: pandas, NumPy, requests, SQLAlchemy, JSON parsing, API integration.
SQL Expertise: advanced proficiency with SQLite, Postgres, and querying via Dremio.
Data Modeling: dimensional and normalized modeling; handling semi‑structured and nested data.
Tools: DBeaver (database connections), Power BI (data prep for reporting).
Pipelines: ETL/ELT design, error handling, logging, performance optimization.
Collaboration: ability to translate business requirements into technical solutions; strong communication skills.
Preferred / Bonus Skills:
Experience working with Elasticsearch or ES-based endpoints.
Familiarity with schema‑on‑read technologies such as Dremio.
Exposure to Docker for environment reproducibility.
Experience with workflow schedulers such as Airflow.
Strong understanding of performance tuning (EXPLAIN plans, indexing strategies) and caching.
About the Company
Q1 is a professional organization that delivers quality products and services. Q1 specializes in Software Development, Business Consulting and Technology Integration. We provide end-to-end integrated solutions that include professional services, functional and technical support and ongoing maintenance using our on-site, off-site and off-shore resources. We offer a comprehensive range of managed services for enterprise business and technology solutions with a team of highly experienced professionals. Q1 is also a Value Added...
Know more