A 401 and a blurry receipt.

A finance team photographs receipts all day. A model reads each one and pulls out the merchant, the date and the total. Anything it is unsure of goes to a review queue, where a person checks it by hand. That queue is the safety net. It is also the most expensive part of the system, because it runs on people.

I like these pipelines to never throw. One bad photo should not stop a batch of five hundred good ones. So every document produces a result. A failure is folded into that result as low confidence. The batch finishes. The person sees what needs a person.

Then the question: what happens when this fails? Not when a photo is bad. When the system is.

The morning the key was revoked

Someone rotates a credential and the extraction service starts answering with a 401. Every call fails. The pipeline does exactly what it was built to do. It never throws. It marks each receipt low confidence and moves on.

The review queue fills with receipts that are perfectly sharp. The reviewers work faster. Nobody is paged, because nothing crashed. From the outside the outage looks like a busy day.

A blurry photo and a revoked key arrive at the same place, wearing the same label. Try it below: break the pipeline both ways, then turn reasons on and do it again.

Try it

Break the pipeline

Receipts flow into extraction. Revoke the key or feed a blurry photo, then turn reasons on and do it again.

Carry a reasonRevoke the API keyFeed a blurry photo

15 Sep$46.9014 Sep$72.1514 Sep$18.40

InboxExtractSort

Filed

0

ReviewReview queue

0

Held

0

On callQuiet

Nobody has been paged.

What the reviewer sees

• Nobody is waiting on a person

Pause the streamNext receiptReset

Keep never-throw, add a reason

The fix is not to start throwing again. The never-throw design is right. What it lost was one piece of information: whose problem is this?

A failure is either about the document or about us. A blurry photo, a torn receipt or the wrong kind of file is about the document. A person can fix it, so it belongs in the queue. A rejected key, a rate limit or a service that timed out is about us. No reviewer can fix a 401 by looking harder at a receipt.

So every failure still comes back as a result, never an exception. It just carries a reason alongside the confidence. The router reads the reason before it reads the confidence:

• a document reason goes to the review queue, as before
• an operational reason raises an alert and holds the document for a retry
• the queue only fills with work a person can actually do

When the key comes back, the held documents go through again and most of them file themselves. Nobody had to review a receipt that was never the problem.

How to tell if you have this

You probably do if your pipeline folds errors into confidence. Look for these signs:

• The review queue grows faster than the documents coming in.
• Reviewers say the items look fine and approve them unchanged.
• Queue spikes line up with deploys, credential changes or a provider's status page.
• The alert history is quiet on the days the queue was loudest.

What happens when this fails?

The reason can be wrong too. I plan for three cases:

• An error nobody named. When the pipeline cannot tell whose problem it is, I treat it as ours. It pages. A person in the queue cannot fix an unknown error. An engineer can.
• Alert noise. Paging on every failed call trains people to ignore the pager. I alert on the rate of operational failures, not on each one.
• Held work that never comes back. Holding documents is only safe if something re-runs them. I count what is held and alert if the count stops falling after the fix.

This does not make review smaller on a normal day. It keeps review for what it was built for: documents a person needs to read.