The way to make computer systems accountable is by designing them using logic rather than "AI" or ML.
Consider the steps taken my law makers in drawing up legislation. Although legal language sometimes seem somewhat convoluted, the aim is to make the law as unambiguous as possible.
A computer system is a formalisation of these rules. Obviously the computer system as with a written rule is still open to interpretation - and therefore the rules should always be open to appeal and evalution by human beings.
I find it hard to imagine a piece of legislation that could be derived precisely by feeding in a large amount data.
Staying with the legal theme, has anyone brought an unfair dismissal case against companies that use automation to sack employees? If no clear explanation can be given for the reasons for the dismissal then obviously any court should immediately decide in the plaintiffs favour. I see "AI" and ML as creating a field day for lawyers.
With an RDBMS we have a system that is directly based on logic. We can take advantage of this to build rule based systems that are fully accountable.
Human beings can produce a system of unfathomable obscurity, but it takes years. It looks as if "AI" or ML can do this in a fraction of the time - this could be seen as an advance in productivity, but not productivity that actually does anything useful - quite the reverse.