Planned hiring
AI Evaluation Engineer
Lucy is a proactive AI coach from Lucy Labs that lives on your desktop as a cloud-backed app.
Planned hiring. This description explains the work we are preparing to hire for; it is not a confirmed start date or offer.
Show where Lucy's methods succeed, where they should abstain and what evidence is still missing. Before a coaching suggestion reaches a user, we need credible evidence that Lucy understood the work, used relevant knowledge and respected the user's boundaries. You will build the independent evaluation practice that makes those judgments testable.
The work
Show where Lucy's methods succeed, where they should abstain and what evidence is still missing. Before a coaching suggestion reaches a user, we need credible evidence that Lucy understood the work, used relevant knowledge and respected the user's boundaries. You will build the independent evaluation practice that makes those judgments testable.
An apparently helpful recommendation can still be based on the wrong goal, an unsupported tool claim or an inappropriate intervention. Your work will show where a method succeeds, where it should abstain and what evidence is still missing. A good result includes an honest reason to delay a release.
What you will own
- Create versioned golden datasets for the first selected work contexts: observed actions, candidate tasks and goals, tool knowledge, recommendations and user corrections. Record provenance, permitted use, coverage gaps and dataset lifecycle requirements.
- Write labeling rubrics that distinguish observations from inferred intent. Build annotation workflows, measure agreement and organize adjudication when reasonable reviewers disagree. Preserve uncertainty rather than forcing a confident label.
- Design reproducible evaluation code and regression suites for extraction, inference and recommendation quality. Report precision, recall, calibration, abstention and failure modes by relevant context, with sample-size limitations.
- Prevent leakage between development and evaluation examples. Keep holdouts and result interpretation independent of the people tuning the system, while giving builders actionable examples and clear reproduction steps.
- Test effects of filtering, missing data, customer vocabulary, policy restrictions and changing tool knowledge. Work with Security on permitted test data and verify that privacy restrictions are represented in evaluation coverage.
- Publish release evidence comparing an agreed baseline with a candidate version. Document dataset, model and rubric versions, unresolved failures and the scope the result actually supports. Carry production corrections back into reviewed regression cases.
Your partners
Applied Science develops inference methods; AI Engineering operates production models and integrates evaluation execution. You own evaluation datasets, rubrics, adjudication and independent findings. Knowledge Graph Engineering supplies semantic contracts. Behavioral Science studies intervention mechanisms, and Quality Assurance verifies integrated product behavior. Product and accountable release owners decide scope and release disposition using the evidence; builders do not certify their own work as independently validated.
What you bring
You have built evaluations for machine learning, language models or another system with ambiguous ground truth. Bring strong Python and SQL, sound measurement judgment and examples of dataset versioning, labeling or reproducible test infrastructure. You should be able to find misleading averages, investigate disagreement and explain a failure to both scientists and product colleagues. Experience protecting sensitive evaluation data is useful; a particular academic credential is not required.
Your first 90 days
Define the evaluation boundary for one coaching use case and audit the available examples. Produce a reviewed rubric, an initial labeled corpus and an agreement report. Establish an independent holdout, run a baseline and a shadow evaluation, then deliver a release-evidence template with reproducible failures and explicit limits. The initial coverage should be small enough to inspect carefully and useful enough to guide the next build decision.
Next step
Express your interest
Email people@lucylabs.ai with the role title, a short introduction and relevant CV or portfolio. This is planned hiring; we will confirm current availability and the individual working arrangement.
Lucy Labs confirms the working arrangement, employer, location, eligibility, compensation and terms for each role. Our mailing address does not determine the job location or guarantee remote employment, worldwide eligibility or visa sponsorship.
Do not send passwords, government identifiers or bank details. Read the application privacy notice.
Frequently asked questions
Is this role open at Lucy Labs?
Lucy Labs lists this role as planned hiring. Availability depends on the need, team readiness and individual arrangement; it does not promise an immediate vacancy or start date.
Where does this work take place at Lucy Labs?
Lucy Labs confirms the working arrangement, employer, location, eligibility, compensation and terms for each role. Our mailing address does not determine the job location or guarantee remote employment, worldwide eligibility or visa sponsorship.
What does this role own at Lucy Labs?
Lucy Labs: Show where Lucy's methods succeed, where they should abstain and what evidence is still missing. Before a coaching suggestion reaches a user, we need credible evidence that Lucy understood the work, used relevant knowledge and respected the user's boundaries. You will build the independent evaluation practice that makes those judgments testable.
How do I express interest in Lucy Labs?
Email people@lucylabs.ai with the role title, a short introduction and relevant CV or portfolio. Lucy Labs uses the application to assess and respond to it; retention for future opportunities is a separate explained purpose and choice.
