I think that one of the most important parts of all of this can be found in the conclusion section of the article. In particular, I'm talking about the following paragraph....
It is common to see non-significant test results after conducting an A/B test. What constitutes significance comes from product teams who take the 'cost of making a change' into consideration. Since a lot of the times this cost can be large, even if the test is barely a success, the product team may decide to go against making that change.
It definitely IS common to seen "non-significant test results" and that's actually a HUGE problem because of all the time already spent on developing a change and then going through A/B and a whole bunch of other testing.
What is that HUGE problem? It can be any one of any combination of the following...
- Poor evaluations of perceived problems. No change was actually needed.
- Poor code or poor implementation of code. People just couldn't get out of the same rut others were in when it came to better code and how it should be implemented.
- Really short schedule. No one actually had time to make good changes and so they did as little as possible.
- Really short schedule. People didn't actually have the time to understand the original problem well enough to improve on it.
- Mistake... someone forgot to do a code regression check or even forgot to check-in code.
- Not likely but maybe the original code was the optimal code (or screen, whatever) to begin with and someone decided to make a change because they didn't have that understanding or their decision making was based on insufficient data.
- Someone thought it was their job to "drive change". One of my favorite laughs comes from a quote my cousin repeated from his old place of work. Apparently some hungry manger said "We have to do something even if it's wrong". <Headdesk>
is pronounced "ree-bar
" and is a "Modenism
" for R
First step towards the paradigm shift of writing Set Based code:
________Stop thinking about what you want to do to a ROW... think, instead, of what you want to do to a COLUMN.
"Change is inevitable... change for the better is not".
"Dear Lord... I'm a DBA so please give me patience because, if you give me strength, I'm going to need bail money too!"
How to post code problems
How to Post Performance Problems
Create a Tally Function (fnTally)