Test what your app means.
semantic-assert lets you assert the requirement instead. Write the claim the way the PRD states it, hand it to a model that acts as the judge, and keep the pass/fail threshold in your code. Thanks to latest AI, it's fast, cheap, and reliable.
ts
const alert = page.getByRole("alert");
await alert.waitFor({ state: "visible" });
await judge.expectPageTo("The alert explains how to recover from the error", {
region: alert,
});That's the Playwright fixture. For API responses and other JSON, use the framework-independent core judge.
Start with a problem you've probably hit
Your PM changed the copy... and yeah the CI turned red. →Check that an alert explains the failure and recovery step across copy edits.The bot just promised a refund? How did the CI pass? →Catch a forbidden promise even when the reply never uses the word “refund”.The route changed but the tab title didn't... and every test stayed green. →Check that the title, breadcrumb, and heading agree, even when the page is named by whatever the user typed.
Or one of these:
- Zero results... or did it just not load yet? Zero rows can mean no results, a request still in flight, or an error.
- A passage is highlighted... in the wrong place. The styling is there. It's attached to the wrong text.
- The answer says “30 days”... yet the advice is still wrong. The right keywords, the wrong conclusion.
How it works
Every semantic assertion follows the same three steps:
- Capture state. Return JSON from your app, or let the Playwright adapter snapshot a page.
- Write claims. Describe what a user should be able to tell from that state. Related claims travel in one request.
- Assert the result. The judge compares the probabilities that come back with your thresholds, and fails the test when a claim doesn't clear its bar.
Who's the judge?
By default, Jev, TypeSafe's first System One model. Jev doesn't generate text. You give it state and a yes/no or multiple-choice question, and it returns a typed answer with a calibrated probability. That's what lets a threshold in your code act as a real pass mark. Any Provider implementation can stand in for it; see providers.
One evaluation by default
Assertions evaluate once (timeoutMs: 0). A Playwright region waits up to 5 s for its element to attach before that single evaluation, so locator assertions still auto-wait. If content settles later than that, wait for it with Playwright first. For state that's genuinely changing, opt into repeated checks with a positive timeoutMs. Usage metrics record every call, its tokens, and the time spent waiting on the provider.
Where semantic assertions fit
| Use semantic assertions for | Keep ordinary assertions for |
|---|---|
| Whether an error gives a concrete recovery step | Whether the alert is visible |
| Whether a reply promises a refund | Exact order IDs and status codes |
| Whether a response answers the customer's question | Counts, totals, and arithmetic |
| Whether a highlighted passage supports a claim | Exact CSS values and class names |
Model judgments are probabilistic
Calibrate thresholds with both good and bad examples from your own app. A passing assertion is the model's assessment, not proof of correctness.
Get started
Follow the quick start for a runnable check, then add the Playwright adapter for HTML and browser tests.