"Production" is a bit more nuanced than it used to be.

Would I endorse data scientists conducting experiments on a DB server providing critical services? NO.

Would I provide them with near real-time data and historical data from production systems? A qualified yes.

The qualification being sufficient security clearance and processes to make sure that no laws are broken and no-one steals anything.

A lot of what data scientists do fails anf that is good because they are testing hypotheses and scenarios. A success is great, but a failure is almost as good. The worse result is a 50:50 because you learn nothing.

One of the really exciting innovations I've seen is one that generates and tests thousands of scenarios and can run many in parallel. This finds strange correlations where no human would think to look.

On Big retail websites a test using data science datasets only needs to run for a few minutes to determine it's impact. At the extreme end Amazon only need a feature switched on for a few seconds.

The challenge is in architecting a system that allows such experimentation in safety. This experimentation takes place in production because it needs real customer interaction to determine success/failure