Home
Jobs
Bioscope AI
Data Scientist (AI Quality & Evaluation)

Data Scientist (AI Quality & Evaluation)

Hybrid

Boston, United states

Full Time

03-03-2026

Share this job:

Score my CV

Skills

Communication Python Test Quality Assurance Research Machine Learning PyTorch Deep Learning benchmarking

Job Specifications

About The Role

We're looking for a Data Scientist to own the quality, reliability, and trustworthiness of our clinical AI outputs. You'll build the systems that ensure our AI "knows what it doesn't know" — developing evaluation frameworks, calibrated confidence scoring, and automated quality assurance that physicians can actually trust.

What You'll Do

Design and implement automated evaluation pipelines that assess AI output quality, accuracy, and safety at scale
Develop uncertainty quantification systems where confidence scores meaningfully correlate with accuracy
Build comprehensive evaluation frameworks combining automated assessment with clinician-validated test cases
Implement feedback loops that continuously improve model outputs based on validation signals
Establish scalable quality gates that catch errors before they reach end users
Contribute to model alignment and fine-tuning efforts

Qualifications

Required

Strong foundation in deep learning frameworks (PyTorch) and LLM architectures
Experience with model evaluation, benchmarking, and quality metrics
Proficiency in Python and modern ML development tools
Strong statistical foundations
Ability to read, implement, and extend research papers
Excellent communication skills

Preferred

Master's degree in Computer Science, Machine Learning, Statistics, or related quantitative field (PhD preferred)
Publications in top ML/AI venues (NeurIPS, ICML, ICLR, ACL)
Experience with RLHF, DPO, or preference optimization techniques
Background in healthcare AI or regulated industries
Experience building evaluation systems for production LLM applications

About the Company

Bioscope AI is a cloud-based, per-patient subscription service for primary care physicians, especially those in concierge or functional medicine practices. It integrates with their EMR to serve as an artificially intelligent consultant that can collaborate with the physician to optimize patient care. The first year of the service includes whole genome sequencing (WGS), so Bioscope AI can help physicians deliver truly personalized, precision care. Know more