There is already precedent for issues in this area. Neural nets have been proven to be pretty good at selecting loan candidates, but legally they cannot be used for that purpose, because the law requires precise rules about what information is used and how. The basic premise of AI is to hand off some decision making to the machine.
[Even having an algorithmic basis is not always enough as Ally bank found out. They went strictly by things like credit rating, payment history etc, and were still hit with a big fine for 'racial discrimination'. In fact they did not even know the race of their customers, and neither did the government, but the government was allowed, for this purpose, to guess based on things like name and neighborhood]
PS. While there is a big legitimate concern with privacy, increased government regulation is unlikely to provide much improvement (both the US and UK have poor track records themselves in preventing leaks) but WILL provide a rigid structure that will inhibit real innovation.
-- FORTRAN manual for Xerox Computers --