What do we do with a model running under SQL Server Machine Learning Services? The output from those scripts and models is often created by the model, without any obvious way to determine how the results are determined. The requirement to explain is enshrined in a law, one that many people are concerned about. With all the ways that ML and AI systems can get gamed and perhaps contain biases based on the data used to train the model, I can certainly see no shortage of people asking for explanations of decisions or conclusions.
I don't know how SQL Server Machine Learning Services works specifically, but judging on the help files, it seems to allow professionals to create machine learning packages using R or Python. If this is the case, then it really depends on the business and how they are utilizing machine learning. For example, all of my data scientist can explain what the model is doing for everything they are running in R. They can show the math behind the models and the steps the algorithm is going through in order to generate the desired output. This is because they have to prove the results, not just assume they are right.
In my case, when I do things, I often can code it, but have no freaking clue what just happened when I'm done. I likely could not fully explain what just happen outside of providing the code and saying this was the output. I would fail any attempt to explain the data process to the data owners.