Accounting in real-time

For an introduction to Accounting concepts you can read my previous post.

In the time before computers, accounting was done by writing entries representing your double-entry transactions into ledgers (probably using a quill), adding things up as you went, or adding totals for each page. This was pretty much immutable, and you need a pen, and the ledgers to do it, so it was probably a job for the end of the day/month/year, using a collection of receipts and other documentation as notes.

In many ways, computers didn't actually change much. You still needed to transcribe your transactions into whatever you were using (be it Excel, or specialist software), and you still needed to collect receipts and bank statements. Things are a little less immutable, which lets you correct mistakes while entering the data, and the computer does the adding up for you (which means no arithmetic errors).

Countingup is a bit different. It's an account, a card, and a smartphone app. You can do the accounting in real-time: when you spend money on your card, you get an immediate notification, can attach a photo of the receipt as a record, then apply the right accounting code, and check how it affected your monthly profit, all before you've left the shop. We even make a guess at the correct code, based on what the shop is.

This immediacy comes with its own challenges…

Mutable data

Transactions are no longer static. They have a potential life cycle, and users can manipulate them in multiple ways. We need this to flow through to your accounts in a predictable and intuitive way.

Here's an everyday scenario to illustrate some of the issues… catching a bus using my Contactless card:

The initial transaction is for 10p, but it just reserves the money and doesn't actually move it. This is what Transport for London does to check the card being used is valid, but unfortunately, the bus trip will eventually cost more than 10p. In the meantime Countingup configures your ledger entries:

tx1  Countingup account   Cr 0.10
tx1  Travel               Dr 0.10

At this point, I realise I used the wrong card. This wasn't a business expense -- I'm going to see some friends. I diligently mark the transaction as Personal use in my Countingup app while sitting on the bus, and it reconfigures the accounts:

tx1  Countingup account   Cr 0.10
tx1  Personal use         Dr 0.10

Assume I catch a few more buses or a tube. Sometime later (probably the next day) TfL will claim the money from the account. The transaction amount changes, and £7.70 is taken from my account. At this point, we need to update the transaction, but we already have ledger entries. What do we do? Intuitively, this is what we want to end up with:

tx1  Countingup account   Cr 7.70
tx1  Personal use         Dr 7.70

which seems straightforward. But there's quite a lot going on here: we manually changed a category, then updated the credit side amount and had to increase a debit to match. What if there was VAT taking up 20% of the transaction?

Applying transactions

To make this work, we draw a distinction between derived data, and "source of truth" data. Whenever any of the source data is updated, the transaction is "reapplied" to generate a virtual instance of the derived data. The derived data is then atomically created, updated or deleted within the database as needed.

If you've used web frameworks like React, this should look a little familiar: props & state changes cause a component to rerender in full, and then React applies any differences to the browser DOM. This is the same principle.

Correctly balancing the books is therefore the job of the Classifier. The algorithm looks a little like this:

type ClassifyTxInfo struct {
  tx             *TransactionDoc
  chosenCategory *Category
  chosenVatRate  *VatRate
}

func classifyTx(info *ClassifyTxInfo) ([]*LedgerEntry, []*VatRecord) {
	category := info.chosenCategory
	if (category == nil) {
		category = determineAutomaticCategoryFor(info.tx)
	}

	vatRate := info.chosenVatRate
	if (vatRate == nil) {
		vatRate = determineAutomaticVatRateFor(info.tx, category)
	}

	vat := calculateVat(info.tx, vatRate)
	ledgerEntries := calculateLedger(info.tx, category, vat)
}

func applyTx(tx *Transaction) {
    database.WithTransaction(func (ledger *Ledger) {
        entries, vatRecords := classifyTx(ledger.collectClassifyInfo(tx))

        ledger.updateLedgerEntries(tx, entries)
        ledger.updateVatRecords(tx, vatRecords)
        ledger.updateTxInfo(tx)
    })
}

For our example, the sequence of events looks a little like this:

card: tx = authorisedTx()                            -> applyTx(tx)
app:  setChosenCategory(tx, "Personal use")          -> applyTx(tx)
card: newTx := merge(tx, State=Cleared Amount=7.70)  -> applyTx(tx)

All the complex allocation logic appears in calculateLedger(), and never has to worry about mutation. The entirety of classifyTx() is a pure function without side effects, which makes it predictable and simple to unit test.

Transaction life-cycle and accounting reports

In real world payment systems transactions don't "just" happen. As shown with the bus, there's often a life-cycle:

For accounting purposes, we only care about the "pending" and "cleared" states — pending transactions affect your accrual balance, and cleared transactions affect both accrual and cash balances. We can ignore transactions in the other states as if they don't exist as they ultimately have zero impact on your account balance or available money.

So how does this affect the classifier above? Well, it often doesn't. We can generate ledger entries and other derived data without regard for the transaction's state, and then simply filter which transactions we include in reports.

As discussed in Accounting concepts, for performance reasons we:

maintain a current account balance
store regular checkpoints for accumulated credits and debits (lazily calculated as needed)

Transactions moving between state can effect both of these, but the above apply algorithm gives us a sure way to ensure they're always in sync:

func (ledger *Ledger) updateLedgerEntries(newTx *Transaction, newEntries []*LedgerEntry) {
    oldTx := ledger.findTxByID(newTx.ID)
    oldEntries := ledger.findLedgerEntries(oldTx)
	oldEffectOnBalances := calculateBalanceChanges(oldTx.State, oldEntries)
	newEffectOnBalances := calculateBalanceChanges(newTx.State, newEntries)

    ledger.SetLedgerEntries(newTx, newEntries)
	if !oldEffectOnBalances.IsEqual(newEffectOnBalances) {
        ledger.UpdateAccountBalances(oldEffectOnBalances, newEffectOnBalances)
        ledger.ClearAccrualCheckpointsAfter(minDate(oldTx.AccrualDate, newTx.AccrualDate))
	}
}

Don't touch the ledger entries

In Countingup a "journal" is a type of transaction which allows an accountant to directly affect the ledger entries. Rather than representing a "real"-world transaction, it is instead a document explaining a correction or housekeeping of some kind. This is literally a list of balancing debits and credits on the relevant account codes.

One way to implement this would be to make a new transaction and let the app edit the ledger entries directly. But this doesn't fit our model.

Instead, we write a JournalDocument which has a rich model including per line item descriptions and VAT handling — this is what the user edits. Whenever a change is made the applyTx function is re-run (there's some extra logic in there to extract relevant information from journals).

Invoices and Bills within Countingup work the same way.

Conclusion

We simplified the complex problem of mutating transactions by treating the accounting records as derived-only data. Each mutation instead maintains a set of facts from a particular viewpoint. It's then up to a pure, testable function, to merge the known information into the accounting records without losing the original data.

The database mutation is then left to a storage layer with little actual accounting logic to worry about.