June 13, 2018 at 9:12 pm
Comments posted to this topic are about the item Building Test Data
June 13, 2018 at 9:26 pm
A couple of decades ago, long before the words "Test Driven Development" ever parted anyone's lips, we called such reliable and safe test data a "Gold Set". And, yes, if the space that they occupied could be weighed, they were worth their weight in Gold.
--Jeff Moden
Change is inevitable... Change for the better is not.
June 14, 2018 at 2:40 am
Totally agree with Jeff. A good set of test data is priceless. It's also expensive to set it up. I've set up realistic test data for demo websites to the point that they want to see aggregate data improve over time. You know, just so that the client can see that things will definitely get better!
To make test data realistic is very difficult and time consuming and I've always used real data as the basis for the test data.
We set up a very useful function on our development website where the internal tester can see the external mail addresses in the notification emails. The dev web application effectively only uses our domain email addresses and adds the external email addresses to the end of the email for reference. When testing the internal client can see which external client would have received the emails so they can be sure the right people will be notified on the live site.
June 14, 2018 at 3:06 am
I think a key component in making this feasible is to stop designing systems to achieve a result and instead have them model a process accurately. It would mean we'd have specs for which data can exist and when/where it exists. Generating test data from these specs would also be relatively easy, since you're not trying to guess what the representative set in production is and instead have a recipe for creating one. Detailed data contracts for external communication would be a prerequisite for this though.
June 14, 2018 at 3:36 am
We use a subset of real data as a basis to ensure that linkages are all going to work, but it's then all anonymised and reviewed by a second person to ensure that it really is going to be okay. This way, the system will work in the same way as live (rejecting records that are invalid or orphaned for example) and loading data that has the full referential integrity as expected. Due to the fact that we're testing batch loads into data warehouses most of the time, we have rafts of test data (separate from the development samples that we build for the developers) with a myriad of scenarios that will ensure that we cover most bases - new scenarios are added all the time. We build processes to build the test data so that it can be regenerated on demand. We're not talking huge volumes of data here, as this is essentially functional testing at this stage.
All of our emails are sent via a separate SQL / SSIS application (we've had issues with email servers going offline just as we were about to send out a "success" email which then caused the job to keel over). This means that the email requests are dropped into a SQL table, and then batch sent later. The test version of the email system doesn't actually send emails until you ask it to (the live version is a 24x7 job, test isn't) and so we can review what should be sent before it is. The test system doesn't have authority to relay outside the organisation either as a separate safety feature. The other safety feature which helps prevent us sending out emails to the wrong people is that all email distribution lists are managed within the email app, rather than in the SSIS / SQL process. This also means that we can easily change the recipients of emails when job changes happen without having to re-deploy the system. The system just requests an email to be sent to "Client1" of type "File Loaded Successfully" and the distribution list is retrieved from the email system at send time. Given that we have a separate test system, we have a separate test email table so we have different email lists pointing to the same email type.
Ben
June 14, 2018 at 3:58 am
Does anyone outside safety-critical industries (eg aerospace) actually work with proper test sets? Perhaps it's just my (30-year) experience, but I'm yet to work anywhere which has proper test data sets (that are not yesterdays/last weeks' backup restored), and that's across insurance, pensions, oil exploration, finance,merchant banking, and logistic sectors... everywhere wants their software yesterday and is not prepared to put in the time/money/people to test it properly.
June 14, 2018 at 6:53 am
Jeff Moden - Wednesday, June 13, 2018 9:26 PMA couple of decades ago, long before the words "Test Driven Development" ever parted anyone's lips, we called such reliable and safe test data a "Gold Set". And, yes, if the space that they occupied could be weighed, they were worth their weight in Gold.
Yes, they are.
June 14, 2018 at 7:04 am
ben.kentzer - Thursday, June 14, 2018 3:36 AM...
- new scenarios are added all the time.We build processes to build the test data so that it can be regenerated on demand. We're not talking huge volumes of data here, as this is essentially functional testing at this stage.
The key parts of this
June 14, 2018 at 8:02 am
Frankly, when I want to unit test a stored procedure in an environment where the data, system specs, and workload as as close as possible to actual production, there have been (some) occasions where I tested it in... production. Well it's sort of in production. OK, I don't unit test this way with procedures which modify data, and I'm not talking about deploying a development version of a procedure on top of an existing production procedure. I'm simply talking about deploying a development version of a stored procedure to production in a special schema called UnitTest, and then intentionally execute it during normal business hours to confirm it's performing as expected and doesn't cause blocking issues. I'll also run the unit test under the context of an account with permissions limited only to executing procedures within the UnitTest schema. I'm not saying (when) or (where) I've done this, only that I've done it in the past.
This is more secure than copying data from production to a development or test environment, and performance tuning a stored procedure in an isolated environment is not a true test of how it will perform in production.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
June 14, 2018 at 8:13 am
kerry_hood - Thursday, June 14, 2018 3:58 AMDoes anyone outside safety-critical industries (eg aerospace) actually work with proper test sets? Perhaps it's just my (30-year) experience, but I'm yet to work anywhere which has proper test data sets (that are not yesterdays/last weeks' backup restored), and that's across insurance, pensions, oil exploration, finance,merchant banking, and logistic sectors... everywhere wants their software yesterday and is not prepared to put in the time/money/people to test it properly.
We won't deploy to production without having gone through the full gauntlet including proper test data sets that have also gone through a strong validation gauntlet. But, I do agree... according to a lot of the posts on this very forum and others, it seems that there are a whole lot of people that view testing (especially real QA/UAT) as some sort of annoyance ("expert" arrogance) on a daily basis. Really spooky out there.
--Jeff Moden
Change is inevitable... Change for the better is not.
June 14, 2018 at 10:35 am
Jeff Moden - Thursday, June 14, 2018 8:13 AMkerry_hood - Thursday, June 14, 2018 3:58 AMDoes anyone outside safety-critical industries (eg aerospace) actually work with proper test sets? Perhaps it's just my (30-year) experience, but I'm yet to work anywhere which has proper test data sets (that are not yesterdays/last weeks' backup restored), and that's across insurance, pensions, oil exploration, finance,merchant banking, and logistic sectors... everywhere wants their software yesterday and is not prepared to put in the time/money/people to test it properly.We won't deploy to production without having gone through the full gauntlet including proper test data sets that have also gone through a strong validation gauntlet. But, I do agree... according to a lot of the posts on this very forum and others, it seems that there are a whole lot of people that view testing (especially real QA/UAT) as some sort of annoyance ("expert" arrogance) on a daily basis. Really spooky out there.
A stored procedure can pass QA and still cause problems like critically poor performance in Production, and there really is no way to simulate a production environment. Sure, you can spin up a test VM with the same number of logical CPU and memory as production, but 2 sockets and 16 cores does not equal 8 sockets and 16 cores, the storage is totally different, and you're not testing within the context of a typical business day workload.
To me, it seems dangerous to deploy a stored procedure to Production, without first having unit tested the new version in production. So long as you use a UnitTest schema to avoid altering any existing production objects and the procedure doesn't modify data, then the risk is minimal. It's essentially no different than running an ad-hoc query in production, except you're executing the code within a compiled stored procedure.
"Do not seek to follow in the footsteps of the wise. Instead, seek what they sought." - Matsuo Basho
June 14, 2018 at 10:41 am
A couple things.
Production data is fine, PII isn't. You can mask/change this.
Tetsing in production might be necessary, but really you ought to simulate the same thing in QA and look at reads and workload in the plan if you're worried. Dont' look at timing, but resources used. That will help you understand. It won't be perfect, and I agree 2/16 isn't the same as 8/16, and storage can be slower, but it often gives you some factor, especially if you include a workload that's replayed from production and not just the QA script of stuff.
This is separate from normal QA. It's a load test, to make a load.
June 14, 2018 at 2:18 pm
kerry_hood - Thursday, June 14, 2018 3:58 AMit seems that there are a whole lot of people that view testing (especially real QA/UAT) as some sort of annoyance ("expert" arrogance) on a daily basis. Really spooky out there.
It seems like this would be fairly easy to measure. How much time and money is going into QA and how much the release issues are costing.
412-977-3526 call/text
June 14, 2018 at 9:16 pm
Eric M Russell - Thursday, June 14, 2018 10:35 AMA stored procedure can pass QA and still cause problems like critically poor performance in Production, and there really is no way to simulate a production environment. Sure, you can spin up a test VM with the same number of logical CPU and memory as production, but 2 sockets and 16 cores does not equal 8 sockets and 16 cores, the storage is totally different, and you're not testing within the context of a typical business day workload.To me, it seems dangerous to deploy a stored procedure to Production, without first having unit tested the new version in production. So long as you use a UnitTest schema to avoid altering any existing production objects and the procedure doesn't modify data, then the risk is minimal. It's essentially no different than running an ad-hoc query in production, except you're executing the code within a compiled stored procedure.
We actually do programmed load testing on a copy of the production database on the production server. We also run PEN testing both at the same time.
--Jeff Moden
Change is inevitable... Change for the better is not.
June 15, 2018 at 3:28 am
Jeff Moden - Thursday, June 14, 2018 9:16 PMWe actually do programmed load testing on a copy of the production database on the production server. We also run PEN testing both at the same time.
So what industry do you guys work in? Or do you work for software houses (as opposed to in-house) and do they require a higher bar for testing/release, agile and quality, than other companies? I've found that management judgement on these issues does not normally use metrics to come to these conclusions; programming teams are measured by delivery, and quite often (like now) the code base has older and not-well-written code, and we need to release the fixes on them... yesterday...
Viewing 15 posts - 1 through 15 (of 28 total)
You must be logged in to reply to this topic. Login to reply