Holdout

About Holdout

A publication about the work after the demo

Holdout is written by four practitioners who have shipped AI features and been paged for them. We cover what we can test: evals, retrieval, agents, latency, cost and the design choices that make users trust the result.

We started Holdout because most AI writing we read was about launches. The questions on our own desks were smaller and harder: why did answers change when nothing was deployed, why did the bill double, why did users stop trusting a feature that was right 94% of the time.

We do not cover funding rounds, leaderboards or model release rumors. When an example needs a model or a company, we invent one and say so. The numbers in our posts are illustrative, worked from pricing shapes and test benches you can reproduce, and we show the arithmetic.

Every post is edited by someone who did not write it, and every code block is run before publication. If we get something wrong, we correct it at the top of the post with a date, not quietly.

Holdout is reader-funded. There are no sponsored posts and no affiliate links, and nobody we write about gets to see a draft.

practitioner authors
4
sponsored posts, ever
0
newsletter readers
18.4k

The writers

Portrait of Mara Ekström (illustration)

Malmö, Sweden

Mara Ekström

Evals and reliability editor · Malmö, Sweden

Mara spent six years building evaluation pipelines for search and support products. She writes about evals, regressions and the postmortems nobody wants to publish. Her favorite metric is the one that disagrees with the dashboard.

Portrait of Tobi Adeyemi (illustration)

Toronto, Canada

Tobi Adeyemi

Infrastructure and cost · Toronto, Canada

Tobi ran inference infrastructure for a mid-sized SaaS company and now writes about tokens, caches, latency and invoices. He keeps a spreadsheet for everything and will send it to you if asked.

Portrait of Kenji Arai (illustration)

Osaka, Japan

Kenji Arai

Open and on-device models · Osaka, Japan

Kenji tests small and open-weight models on hardware people actually own: laptops, phones, one tired workstation. He covers quantization, memory math and how to read a model card without believing all of it.

Portrait of Inês Valente (illustration)

Lisbon, Portugal

Inês Valente

AI product and interaction design · Lisbon, Portugal

Inês designed AI features for a fintech app and a writing tool before joining the desk. She writes about agents, latency as a design material and building features that users forgive when they are wrong.

One careful essay, every other Thursday

Each issue takes one problem, like a cache that never hits or an eval that passed a bad release, and works it through with the numbers and the code.

Free. No sponsored posts. Unsubscribe in one click. Read by 18,400 engineers, PMs and designers.

Live demo · Holdout— View template →