Making an AI product reliable enough to ship: a QA and evaluation case

NetForemost helped a US health and wellness company take an AI-powered consumer app to production by pairing large-scale data processing with a continuous evaluation and QA loop. The hard part was never generating results. It was making those results reliable enough to put in front of real users.

linkThe real risk was trust, not throughput

The app lets someone photograph a grocery product and returns a rating of how healthy it is, with the reasoning behind it. Behind that simple interaction sits a catalog of more than a million products sold across US supermarkets. Each one had to be found, normalized into a clean database, and rated by an AI pipeline against the client's own reference criteria.

Building that pipeline was demanding, but it was not where the risk lived. The risk was a single wrong rating. For a company in health and wellness, an AI that labels something harmful as healthy is not a cosmetic bug. It is a liability that reaches the brand directly. AI models make mistakes and they hallucinate. Early in the work, the pipeline rated coconut water as bad for you, which it is not. At scale, an incorrect rating like that could undermine trust in the product and the brand behind it.

linkA continuous loop, not a one-time check

Reliability here could not come from a better model alone. It came from a system of validation that ran continuously.

The loop ran in four repeating steps: discovery, to understand the data and the criteria the ratings had to meet; evaluation at scale, using cloud tooling to process millions of rows and surface the ratings that broke those criteria; correction, to fix the flagged cases; and re-evaluation, to run the checks again on the corrected data. The team used each cycle to identify and reduce errors.

Two things made the loop work beyond the tooling.

The first was quality assurance as an audit layer over the AI, not a final gate. The team did not accept the model's first output. Every rating passed through checks, several of them automated, that caught the cases where the model drifted from the client's criteria. The aim was to find and correct errors before release, treating the model's output as something to audit rather than trust on sight. This is the same discipline NetForemost applies to conventional software, pointed at a new problem: keeping generated output inside defined bounds.

The second was the client's ownership of the domain. The product owner joined the team daily to explain the criteria and the context the team should not have to guess. Reliable ratings depended on getting the domain right, and that knowledge lived with the client. The result came from automated processing, QA, and the client's expertise working as one system, not from automation on its own.

linkValidated with real users before launch

Before production, the app was tested with 200 real users in the United States over one week. They logged everything they ate, from breakfast to dinner, using the app in daily life.

The test gave the team real-world feedback on the app's core functionality and surfaced improvements before launch. One clear request was the ability to group products eaten together, so a daily breakfast of fruit and cereal did not have to be searched item by item each morning. Feedback like this shaped the next set of features, and passing this stage was the condition for releasing to a far larger audience.

linkWhat the work produced

The app reached production and now serves more than 10,000 daily active users, a number that has kept growing in the months since launch. That growth belongs to the product and its audience. What the work delivered was the reliability and validation that made shipping to that audience defensible in the first place.

The scale of the problem is its own result. More than a million products were processed and rated by a team of three: one project manager, one QA engineer, and one developer. A lean team carried a large, data-heavy AI product to production by leaning on evaluation and QA rather than on headcount.

linkThe lesson for AI products

Generating results quickly is the easy part now. The teams that ship AI products successfully are the ones that build a system to keep the output reliable: evaluation and QA to hold generated results inside defined bounds, and domain ownership from the people who set the criteria. That system, not the model alone, is what makes an AI product safe to put in front of real users. It is where NetForemost does its work.