Machine learning model testing checks that a model, its data and its code behave correctly before and after release. It starts with splitting data into training, validation and test sets, then adds data checks, behavioural tests, fairness and robustness tests, and production monitoring. In 2026 it also includes evaluating LLMs and AI agents, where outputs vary and "correct" often needs a scoring rubric.
Updated October 2026
What changed in 2026
- Testing has a legal deadline in Europe. The Digital Omnibus on AI, published in July 2026, set 2 December 2027 for stand-alone high-risk AI obligations, which include accuracy, robustness and documentation requirements.
- GenAI evaluation joined the MLOps toolchain. MLflow 3 added an evaluation harness and custom scorers for LLM apps next to classic model metrics.
- Weak testing kills projects. Gartner predicts over 40% of agentic AI projects will be cancelled by the end of 2027 due to costs, unclear value or inadequate risk controls (Gartner).
- Risk frameworks such as the NIST AI Risk Management Framework are now common reference points for what "tested enough" means.
Training, validation and test data
Every model test starts with how you split the data:
| Set | Purpose | Rule of thumb |
|---|---|---|
| Training set | The model learns patterns from it | Largest share, often 60–80% |
| Validation set | Tune hyperparameters and pick the best model | Often 10–20%; or use cross-validation |
| Test set | Final, unbiased estimate of performance | Often 10–20%; touch it once, at the end |
The ratios are conventions, not laws. What matters more:
- No leakage: the same customer, patient or machine should not appear in both training and test data.
- Time-aware splits: for forecasting or fraud, train on the past and test on the future.
- Representative slices: make sure rare but important groups (regions, languages, device types) are in the test set.
- A frozen "golden" set: keep a hand-checked set that every new model version must pass.
What to test in an ML system
Google's ML Test Score rubric groups tests into data, model, infrastructure and monitoring. That structure still works in 2026.
1. Data tests
Schema checks, value ranges, missing values, duplicates, label quality and distribution shift between training and live data.
2. Model tests
- Performance: accuracy, precision, recall, F1, AUC or error, overall and per slice.
- Behavioural tests: invariance (changing a name should not change a credit score), directional (more damage should mean a higher claim estimate) and minimum functionality tests.
- Robustness: noise, typos, blurred images, adversarial inputs.
- Fairness: compare error rates across groups.
- Baselines: the new model must beat a simple baseline and the current production model.
3. Code and infrastructure tests
Unit tests for feature code, integration tests for the full pipeline, and checks that training and serving compute features the same way. The Made With ML testing guide has practical examples.
4. Production monitoring
Live accuracy (when labels arrive), prediction drift, latency, cost and alerts with clear owners. Microsoft's post on testing ML systems with MLOps shows how to wire these checks into CI/CD.
Testing LLMs and AI agents
Generative models need extra methods because there is rarely one right answer:
| Test | What it checks | How |
|---|---|---|
| Golden Q&A sets | Correctness on known questions | Reference answers + similarity or rubric scoring |
| Groundedness | Answers come from retrieved sources | Citation checks, LLM-as-judge |
| Safety and policy | No toxic, leaking or off-policy output | Red-team prompts, guardrail filters |
| Tool-use accuracy | Agents call the right tool with the right arguments | Scenario tests with mocked tools, trace review |
| Regression | New prompt or model version doesn't break old cases | Run the full suite on every change |
For language-specific metrics, see sentiment analysis with NLP, which walks through precision, recall and F1 on real feedback data.
A testing workflow that fits the ML pipeline
- Validate incoming data automatically.
- Train and evaluate on validation data; log every run.
- Run the gate suite: test-set metrics, slices, fairness, behavioural and robustness tests.
- Compare with the production model; promote only if better and safe.
- Deploy gradually (shadow or canary) and monitor live metrics.
- Feed new labelled cases back into the golden set.
Each step maps onto the stages in our machine learning pipeline guide. For edge models that run on thousands of devices, see MLOps for edge AI.
Tools in 2026
- Data validation: Great Expectations, TFX Data Validation, Deepchecks.
- Experiment tracking and evaluation: MLflow, Weights & Biases.
- Monitoring: Evidently, cloud model monitors, Prometheus and Grafana.
- LLM evaluation: MLflow GenAI evaluation, promptfoo, DeepEval, Ragas.
- Classic testing: pytest for feature and pipeline code.
Compare platforms in our ML platform tools guide.
Who owns model testing?
Testing fails when it belongs to nobody. A simple split works for most teams: data engineers own data tests, ML engineers own model and pipeline tests, a product or risk owner signs off on fairness and business thresholds, and platform teams own monitoring and alert routing. Write the release criteria down before training starts, so a model is judged against agreed numbers rather than whatever looks good on the day.
Where it's heading: 2026 → 2030
| Period | Model testing |
|---|---|
| 2020 (history) | Accuracy on a held-out test set; little production monitoring |
| 2026 (now) | Data, behavioural, fairness and LLM evaluation gates in CI/CD |
| 2027 (expected) | EU high-risk rules apply from December 2027, making documented test evidence routine |
| 2030 (our view) | Continuous evaluation of agents in simulation and production, with test agents probing other agents |
How NexaStack helps
NexaStack runs models and agents on your own cloud, data centre or edge and gives teams one place to deploy them in shadow or canary mode, collect traces and monitor accuracy, drift, latency and cost. That makes it easier to test on real, private data without moving it. ElixirData, the decision harness for enterprise AI agents, adds policy checks and audit trails where model output drives business decisions.
FAQ
How can I test a machine learning model?
Hold out a test set, measure metrics overall and by slice, run behavioural, robustness and fairness tests, compare with a baseline and monitor the model after release.
What is the difference between training and testing data in machine learning?
Training data teaches the model. Testing data is kept separate and used once at the end to estimate how the model will perform on new, unseen data.
Why do we need a validation set?
It lets you tune settings and choose between models without peeking at the test set, which would make the final score too optimistic.
How do you test an LLM or AI agent?
Use golden question sets, groundedness and safety checks, tool-use scenario tests and regression runs on every prompt or model change, with LLM-as-judge scoring checked by people.
What are the latest ML model testing trends in 2026?
GenAI evaluation inside MLOps tools, behavioural and fairness gates in CI/CD, agent trace testing and test evidence for AI regulation.
Test models on real data, safely
Want to shadow-test models and agents on private data before they go live? Book a NexaStack demo to see deployment, tracing and monitoring on your own infrastructure.
