Job Specifications
Senior Data Engineer
Hybrid- Minneapolis, MN
Hourly Pay Rate Range $70-75/Hour
This project supports the firm’s Fixed Income organization, which currently relies on highly manual, Excel-based workflows to capture, maintain, and analyze deal and market data. These processes are time-intensive, introduce potential data quality risks, and limit the ability to leverage advanced automation and AI-driven insights.
The initiative focuses on modernizing this ecosystem by developing a scalable full-stack application that integrates seamlessly with Databricks and internal data platforms. The solution will incorporate LLM-powered capabilities to streamline data ingestion, enhance data quality, surface actionable insights, and enable intelligent, multi-agent workflows.
Unstructured Data Platform
Design and implement pipelines to ingest, parse, and transform unstructured data — PDFs, earnings transcripts, deal documents, research reports, and pitch materials
Build scalable document processing workflows on Databricks using Delta Lake, AutoLoader, and Spark-based NLP/embedding pipelines
Integrate with vector stores (e.g., LanceDB, Pinecone, Databricks Vector Search) to support RAG architectures and semantic search
Partner with AI engineers to prep and ground document corpora for LLM-powered applications
Genie Spaces
Architect and deploy Databricks Genie Spaces to enable natural language querying across deal, financial, and client data
Define and curate certified datasets, semantic models, and trusted metrics layers to ensure accuracy of AI-generated answers
Collaborate with business stakeholders to identify high-value use cases and onboard end users to Genie-powered analytics
Monitor and tune Genie Space performance, model grounding quality, and response fidelity
API & Integration Layer
Build and maintain REST and Model Serving APIs on Databricks to expose data and ML model outputs to downstream applications
Integrate Databricks Model Serving endpoints with Azure API Management for governance, rate limiting, and routing
Develop robust Python/FastAPI microservices and connectors linking Databricks assets to Cosmos DB, DealCloud, and Azure services
Ensure API reliability, observability, and versioning standards across the platform
Required Qualifications
5+ years of data engineering experience with at least 2 years in Databricks (Unity Catalog, Delta Live Tables, Workflows)
Hands-on experience with unstructured data processing — text extraction, chunking strategies, embedding pipelines
Proficiency in PySpark, Python, and SQL; experience with Databricks SQL and serverless compute
Working knowledge of Databricks Genie Spaces or equivalent NL-to-SQL/AI BI tools
Experience building and deploying REST APIs; familiarity with FastAPI or equivalent frameworks
Familiarity with vector databases and RAG pipeline design
Strong understanding of data modeling, lineage, and governance best practices
Preferred Qualifications
Experience integrating Databricks with Azure services (ADLS, API Management, Azure AI Foundry, Cosmos DB)
Exposure to LLM orchestration frameworks (LangChain, DSPy, LlamaIndex)
Background in financial services, investment banking, or professional services data environments
Familiarity with DealCloud, S&P Capital IQ, or PitchBook data structures
Databricks Certified Data Engineer Associate or Professional certification
About Us:
At Brooksource, relationships are the foundation of everything we do. Since 2000, we’ve built lasting partnerships with clients, consultants, and internal teams to deliver an exceptional experience across every engagement. As a trusted IT and Engineering services provider, Brooksource supports Fortune 500 organizations through Experience-Driven Staffing, Professional Services, and Elevate, our proprietary Workforce Transformation program. Whether you're hiring for software development, cloud computing, cybersecurity, data analytics, or enterprise IT, our customized staffing solutions are designed to align with your company’s unique goals, culture, and technology stack. We offer flexible hiring models, including contract, contract-to-hire, and direct placement to meet your evolving business needs.
We are a certified partner of leading platforms, including Salesforce, AWS, Microsoft, and Google Cloud, enabling us to deliver scalable, end-to-end technology solutions. With a growing national footprint, Brooksource is redefining expectations in IT consulting, engineering services, and technology workforce solutions.
EEO Statement:
Brooksource is an equal opportunity employer that does not discriminate on the basis of actual or perceived race, color, creed, religion, national origin, ancestry, citizenship status, age, sex or gender (including pregnancy, childbirth, lactation and related medical conditions), gender identity or gender expression, sexual orientation, marital status, military service and veteran status, physical or mental disability, protected medical condition as defin