>Imagine your bank being non-deterministic.

That's already the case. Payments are not deterministic. It can take multiple days for things to settle. The real world is messy.

When I make a payment I have no clue if the money is actually going to make it to a merchant or if some fraud system will block it.

The bank can very much determine if the payment has been made or not (although not immediately, as you mentioned). As a rule, banks like to keep track of money.

Yes it settles deterministically. With AI it claims to be settled and goes on, and it's up to you to figure it out how deterministic the whole transaction actually was.

Is it the main issue? Payments suffer from race conditions, but the processes themselves are deterministic, auditable and may be rolled back. Not sure how many of these important attributes would remain with a neural network at the helm.