I built an AI assistant for small-business bookkeeping. A user writes what happened the way they'd say it, in English, Burmese or both, for example "paid the shop rent from the bank today, 5 lakh", and the assistant drafts a proper double-entry journal:
| Account | Debit | Credit |
|---|---|---|
| Shop rent (expense) | 500,000 | |
| Bank | 500,000 |
The demo is easy. The real question, from every accountant, is: what stops it from writing something wrong into the books? My answer was to design the system so the model can't write at all. Here are the four guardrails, which apply to almost any agent that works on important data.
1. Give the model no write action
Each turn, the model returns exactly one structured action, validated against a schema:
- Reads: search accounts, look up customers, run a report.
- Proposals: a draft entry, shown to the user as a card.
- Words: an answer, or one clarifying question.
There is no "post" action. Writing happens only when a person confirms the proposal. This one decision changes the risk completely. A hallucination, a prompt injection or a bad model version can, at worst, show a wrong card, which the user sees, with a short explanation, before anything happens.
A practical tip: give the model a simple, flat schema for generation (small models follow simple shapes much better), then validate the response with a stricter schema in code that adds cross-field rules, like "a credit sale must name a customer" or "the date must be a real day".
2. Fence the identifiers
Models are very good at inventing plausible IDs. So the agent loop keeps a list of every ID that a read in this turn actually returned. A proposal that refers to anything else is rejected:
// simplified
const unseen = referencedIds(proposal).filter((id) => !seenThisTurn.has(id))
if (unseen.length > 0) return reject(`unknown ids: ${unseen.join(", ")}`)
The rejection goes back to the model as feedback, and the loop tries again a few times. If it still can't produce something valid, it says so instead of guessing.
3. Don't let the model decide the numbers
The most expensive mistake is a misread amount: "5 lakh" posted as 5. So a proposal carries the exact text the user typed for the amount, and a small parser in code (not the model) turns it into a number. The model's own reading is kept only as a cross-check.
The parser's rule is refuse instead of guess. If a phrase could mean two different numbers, the assistant asks one short question instead of picking one.
4. Put the final rule in the database
The first three guardrails live in application code, and application code has bugs. The last rule lives where nothing can skip it: in the database (PostgreSQL). A deferred constraint trigger rejects any journal entry that has fewer than two lines or where debits don't equal credits. It runs at commit, so it holds for the AI flow, the manual form, imports and anything added later.
Around it, the usual accounting controls make the books audit-ready: locked periods, corrections only by reversing entries, and an audit log of who changed what.
Testing an agent like a system, not a chatbot
- Real database tests for the ledger rules, permissions and row-level security.
- An evaluation set of real-world phrasings, scored against the expected proposals, so prompts and models (including small local ones) can be compared with numbers instead of impressions.
The takeaways
- Let the model read and propose; let a person decide.
- Don't trust the model with identifiers or numbers. Fence IDs, and parse figures from the user's own words.
- Enforce the critical rule where nothing can skip it. Then the AI is just one more user of rules that already hold.
These ideas aren't specific to accounting. They fit any agent that can change orders, payments, stock or user data: make the safe path the only path.