An AI product can look impressive in a demo. But how do you prove it's actually intelligent, accurate and delivering value? That's where you come in.
We're working with an established business making significant investments in AI and generative technology, and we're looking for a commercially minded, technically strong AI Product Analyst to help answer the questions that really matter.
Is the model producing accurate answers? Are users getting what they need? Is retrieval working properly? Are hallucinations decreasing? Are product changes actually improving the experience?
This isn't simply about building dashboards or reporting on usage. You'll be evaluating the performance and quality of AI models and the products they power, helping shape what gets improved, developed and launched next.
What will you actually be doing?
Evaluating LLM and RAG product outputs, focusing on accuracy, groundedness, hallucinations, relevance and answer quality.
Developing evaluation datasets, scoring frameworks and KPIs to measure model and product performance.
Investigating why AI responses fail, why certain queries underperform and where improvements are needed.
Using SQL and Python to analyse product behaviour, identify patterns and uncover actionable insights.
Measuring the impact of model changes, retrieval improvements and new product features.
Designing experiments, analysing user behaviour and identifying opportunities to improve the experience.
Working alongside AI Engineers, Product Managers and technical teams to turn evaluation findings into meaningful improvements.
Presenting evidence-backed recommendations to senior stakeholders and influencing product decisions.
Who would be a great fit?
You might currently be a Product Analyst, Senior Data Analyst, AI Analyst, Data Scientist or working in a role involving LLM evaluation.
We're particularly interested in people with:
Strong SQL and Python skills and the ability to independently investigate complex datasets.
Experience in product analytics, experimentation, KPIs, user behaviour or A/B testing.
Exposure to AI, machine learning, LLMs or generative AI products.
An understanding of model evaluation, including output quality, accuracy, retrieval performance or evaluation frameworks.
The ability to connect technical findings to product improvements and commercial outcomes.
Confidence to challenge assumptions, question results and influence technical decisions.Important: We're looking for someone interested in evaluating how well AI models and products actually perform, rather than simply evaluating AI tools or tracking their adoption.
Why should you consider this?
Because your analysis won't sit in a report that nobody reads.
You'll be helping define what good AI performance looks like, working closely with the people building the technology and influencing decisions that directly shape the product.
You'll have visibility, ownership and the opportunity to make a measurable difference as the business expands its AI capabilities.
If you're fascinated by what makes AI genuinely good, and how to prove it with data, let's talk.
52461MSR1
INDLON
The Portfolio Group are acting on behalf of our client in recruiting for this position
