Small and on-device
Field test: three small models, two laptops and 200 invoices
We ran 3B, 7B and 14B open-weight models on two ordinary laptops to extract invoice fields. Speed, accuracy, memory and battery, measured rather than guessed.
Page 2 of 2
Latest from the desk
Softmax & Sons is an independent, reader-funded publication about applied AI for the engineers, PMs and designers who ship and maintain AI features.
Cost and latency
The real cost of long context: a worked example at 40,000 calls a day
Stuffing 120,000 tokens into every request is easy to build and expensive to run. We price it, time it and compare it against a 6,000-token retrieval setup.
Evals
What an eval suite actually catches (and the three things it never will)
A 240-case eval suite for a ticket summarizer, a cheaper model swap, and an honest accounting of which failures the suite caught and which reached users.