Data / Evaluation Engineer
This role helps build Lucy, the proactive AI coach from Lucy Labs that lives on your desktop as a cloud-backed app, while the product is in development with design partners.
Builds the ground truth that keeps Lucy from being confidently wrong.
About the role
At Lucy Labs, this role owns the purpose described below.
The Data / Evaluation Engineer builds the ground-truth and quality system that prevents Lucy from shipping plausible but wrong interpretations of work. This role owns labeling pipelines, inter-rater agreement, shadow-mode validation, eval datasets, quality gates, rejection reasons, and regression tests for Work Modeling systems.
This is an engineering role, not an analyst role. It builds the data and eval infrastructure that makes behavior inference measurable and improvable.
The mission
At Lucy Labs, this role's mission is defined by the outcome and work below.
Build evaluation systems that:
- Convert messy work evidence into usable labeled datasets
- Measure action, task, goal, workflow, and coaching quality
- Catch regressions in inference, privacy, and product behavior before release
- Give product and engineering clear quality gates for design-partner deployment
- Turn user corrections into structured learning loops
What you'd own
At Lucy Labs, this role turns its mission into concrete responsibilities in each area.
1. Labeling and Ground Truth
For 1. Labeling and Ground Truth, Lucy Labs expects this role to own the responsibilities below.
- Build labeling workflows, rubrics, and adjudication processes
- Track inter-rater agreement and label quality by work context
- Design data schemas for evidence, labels, corrections, and eval outcomes
- Partner with Workflow/Product Research on observed work practices and ambiguous cases
2. Evaluation Infrastructure
For 2. Evaluation Infrastructure, Lucy Labs expects this role to own the responsibilities below.
- Build shadow-mode eval pipelines for inference and coaching systems
- Maintain golden datasets, regression suites, and quality dashboards
- Define precision, recall, abstention, correction, and rejection metrics
- Integrate eval gates into release workflows and model/prompt versioning
3. Data Quality and Analytics Engineering
For 3. Data Quality and Analytics Engineering, Lucy Labs expects this role to own the responsibilities below.
- Build data contracts and quality checks with backend/data platform engineering
- Monitor missing evidence, policy filtering effects, and downstream inference quality
- Create quality instrumentation for user correction loops
- Support analytics without weakening privacy or provenance boundaries
4. Cross-Functional Partnership
For 4. Cross-Functional Partnership, Lucy Labs expects this role to own the responsibilities below.
- Work with Behavior Inference on failure analysis and model/rule improvement
- Work with AI Platform/MLOps on automated eval execution and model observability
- Work with KG/Ontology on taxonomy quality and relationship validation
- Work with QA/Release on release-blocking quality checks
What success looks like
Lucy Labs defines these milestones as role outcomes, not metrics for measuring the person.
- 60 days
- Labeling rubric and first eval dataset are live
- 90 days
- Shadow-mode evaluation reports quality by work context
- 6 months
- Release gates catch inference and privacy regressions before users do
- 12 months
- Eval platform is a core engineering function with repeatable quality loops
Experience and fit
Lucy Labs looks for the essential experience listed here and treats the rest as helpful, not an automatic barrier.
Must have
- 4+ years data engineering, analytics engineering, ML evaluation, or applied data science experience
- Strong Python and SQL
- Experience building datasets, quality checks, dashboards, or test harnesses
- Ability to design metrics that measure real product behavior, not vanity accuracy
- Strong written communication and cross-functional synthesis
Nice to have
- LLM or ML eval harness experience
- Labeling operations, human-in-the-loop systems, or annotation QA experience
- Process mining, workflow analytics, or task-mining data experience
- Experience with privacy-aware analytics or regulated data
How you work
Lucy Labs looks for these working traits alongside technical experience.
Role DNA
Lucy Labs evaluates these traits as part of the fit for this role.
- Evidence-driven
- Quality-system builder
- Skeptical of unvalidated model claims
- Comfortable with ambiguous human behavior data
- Collaborative with research and engineering
Apply for this role
The Lucy Labs application form opens with this role already selected. You can review the message before sending it.
Apply for this roleFrequently asked questions
Is this role open at Lucy Labs?
Yes. Lucy Labs confirmed this role is open on 25 August 2026.
Where does Lucy Labs base this role?
Lucy Labs bases this role in Madrid, Spain. The exact working arrangement is confirmed during the application conversation.
What does this role own at Lucy Labs?
At Lucy Labs, this role has one central responsibility: Builds the ground truth that keeps Lucy from being confidently wrong. The full description sets out the mission, responsibilities, and relevant experience.
How do I apply to Lucy Labs?
At Lucy Labs, use the application form. The role is selected automatically, and you can review it before sending.
