The first test I write for any AI feature has nothing to do with AI

That is it. A schema check. All required fields present, types correct, no extra fields the model invented on its own.

I know how unimpressive that sounds. Months of testing AI agents in production and my opening move is one boring expect(output).toMatchSchema().

But the model rarely fails by being stupid. It fails by being creative. The wording is fine, the answer sounds smart, and then one day there is a field in the response that nobody asked for, and some downstream code that trusted the shape quietly falls over.

What the test looks like

The matcher isn't built into Jest — it comes from a package called jest-json-schema, and expect.extend does the wiring:

npm install --save-dev jest-json-schema @types/jest-json-schema

The types package matters if you're on TypeScript. Without it the code runs perfectly but the compiler complains that toMatchSchema doesn't exist, because TypeScript only trusts matchers it has been told about.

// simplified example — real schemas are bigger, the idea is identical
import { matchers } from 'jest-json-schema';
import { extractMovieFields, sampleReview } from './extract';

expect.extend(matchers);

const movieSchema = {
  type: 'object',
  required: ['title', 'director', 'rating', 'release_date'],
  // catches the fields the model invents
  additionalProperties: false,
  properties: {
    title:        { type: 'string' },
    director:     { type: 'string' },
    rating:       { type: 'number', minimum: 0, maximum: 10 },
    release_date: { type: 'string', format: 'date' },
  },
};

test('AI extraction returns the shape it promised', async () => {
  const output = await extractMovieFields(sampleReview);

  // not checking if the answer is good yet — only if it's the right shape
  expect(output).toMatchSchema(movieSchema);
});

The line doing the real work is additionalProperties: false. Required fields catch what the model forgot. That line catches what it made up, which is the harder failure to notice in review.

What it looks like when it fails

Make the model misbehave on purpose — drop a required field, let it invent one nobody asked for — and Jest tells you exactly what happened:

● AI extraction returns the shape it promised

  received
    must have required property 'release_date'
    must NOT have additional properties, but found 'sequel_title'

It hallucinated a sequel. Caught in milliseconds, with no eval framework and no golden dataset.

Why start here

• It's the failure that hurts most. An awkward sentence annoys a user. A missing field crashes the code downstream from it.
• It's cheap. No golden dataset, no human review, no meeting about whether the answer felt right. It runs on every commit.
• It fails loudest, earliest. When a prompt changes or the model drifts, this alarm usually rings first.

The clever evaluation work matters too — semantic similarity, LLM-as-judge, human review panels. It just comes second. Took me a while to accept that.

The order I actually work in

1. Structure. Is it the shape you promised? The schema check above.
2. Boundaries. Does the output ever touch data it shouldn't, or return a value outside a legal range?
3. Resilience. What happens on a malformed input, an empty input, an enormous input?
4. Quality. Only now — is the answer actually any good?

Most teams start at step four because it is the interesting one. Then they spend a quarter building an evaluation harness while a hallucinated field sits in production.

Boring test first. Clever tests after.

---

If you're making the move from manual testing into automation, the 90-day roadmap is the place to start. I post twice a week on LinkedIn.