does a car do a full emergency stop for a pigeon ? potentially injuring the people in car behind who were (at their own risk) travelling too closely. cars can't make those choices. the car can only protect the people in the vehicle... the "pigeon" could have been just a shopping bag blown by the wind.
Now here's something that has happened a few times to me: I don't have a self-drive car, but I do have one of those new 'forward look breaking assistants', aka an auto-break. It's intended to sound an alarm if I'm too close to something, and it will break if I'm doing less than 30mph.
only... todate, it's slammed the breaks on for a) rain, b) a car to the left of me turning left, c) the sign for the bend in the road ahead of me, d) I think it was a squirrel dashing across the road, but could have been a rat, e) the car that pulled out in front of me and sped off (I was already breaking when the alarm sounded and the brakes slammed on harder...) f) I think just because it felt like it - there was nothing in the road ahead at all... and that's about it. The alarm's sounded when I'm doing over 30mph for and of the former cases, plus cars that cut lanes, and again, some odd reasons I couldn't identify and as there was no impact, wasn't a problem. It's enough for me to consider driving at 31mph for safety (the alternative is to turn it off at the start of every trip, at which point, why have the system at all).
So I agree with Kiwood: Tech is no substitute for a human brain, and at that point the idea of morality becomes moot: The AI simply lacks the understanding from the information it has available from which to make a moral decision. Even with ML and object recognition, there would simply be too many mistakes for morality to come into the equation. Indeed, it would simply add to the complexity and result in erratic, unreliable results.
Much like the brake assistant wanting to stop me hitting rain.
Answer: Don't have the systems override the driver. Have them inform the driver instead. Then they can learn from the driver and eventually, perhaps, they'll know enough to be able to drive safely and handle the odd unexpected event.