March 2, 2022 at 12:00 am
Comments posted to this topic are about the item The Test Mistake
March 2, 2022 at 1:39 pm
Definitely a clbuttic mistake and one I've made - once. I've also been aware of other people making it and prostrated myself on their behalf! Even if the live data update is made an automatic step sometimes that can fail to happen for some reason. Now when testing the first step is to check the data. When customers are not too fond of the emails they are getting it does make it worse....
March 2, 2022 at 4:16 pm
I've got a question about this article. In the article Steve talked about scrubbing data out of test/dev databases or environments, which reflect what's in production. Where I work using production data is test/dev environments is the ONLY acceptable approach. Don't DARN ask or suggest anything else.
So, why is it a bad idea to use production data in test/dev?
Kindest Regards, Rod Connect with me on LinkedIn.
March 2, 2022 at 5:48 pm
So, why is it a bad idea to use production data in test/dev?
Off the top of my head, Only a few accounts have access to production limiting the accounts that can be compromised to gain access to production data. If this data is in a lower environment the all those developer and QA accounts now have access to production data increasing the chance that a lower level account could be compromised to get production data. Also, there may be regulations that limit the people who can have access to production data depending on what that data contains.
March 2, 2022 at 6:53 pm
Almost never is test/dev secured as well as production. This is a place where many data breaches have occurred.
While this isn't as much of an issue for you, Rod, in the US, it is in many countries, and in CA. You can get fined for losing data this way.
You don't need production data. What often happens is that most devs and companies don't bother trying to recreate the test cases and problem domains.
March 2, 2022 at 8:19 pm
Almost never is test/dev secured as well as production. This is a place where many data breaches have occurred.
While this isn't as much of an issue for you, Rod, in the US, it is in many countries, and in CA. You can get fined for losing data this way.
You don't need production data. What often happens is that most devs and companies don't bother trying to recreate the test cases and problem domains.
I'm not sure you can make that a blanket statement. We have many test cases where the receiving system of a test from our system requires actual valid data, otherwise the tests won't complete correctly. Especially on volume tests with thousands of items. It helps that our test systems are protected from breaches as thoroughly as our production systems.
March 2, 2022 at 9:21 pm
I've worked on a number of financial and high volume systems. Many of those systems have test accounts in them, precisely to allow system-to-system tests.
However, for most tests. it's a question of mocking the problem space rather than specific data. Specific data often brings with it other issues, such as not expecting a variety of the potential things that can happen. Production data isn't needed, though it might be convenient.
If you protect non-prod systems, then using production data wouldn't be an issue. That isn't the case for many organizations, where security is less lax.
March 2, 2022 at 10:16 pm
If you protect non-prod systems, then using production data wouldn't be an issue. That isn't the case for many organizations, where security is less lax.
So as long as your security is the same in production and dev/test it's all good 🙂
March 2, 2022 at 10:23 pm
I've got a question about this article. In the article Steve talked about scrubbing data out of test/dev databases or environments, which reflect what's in production.
At the risk of sounding as if I'm splitting hairs, Steve actually talked about scrubbing sensitive data. A really important distinction. It may be enough for some applications to anonymise the data while keeping the numerical values, dates etc unchanged and therefore available for realistic testing. Definitely change or obscure identifiable customer data. This does require an excellent understanding of your system's data model, which may be challenging in a high-change environment or if you're using someone else's packaged solution.
Bear in mind that anonymisation has limitations. If you have a few customers who are outliers in terms of their data profile, even obscuring their identifying data may not be enough to hide who they are in real life, and then their data in test is exposed to the risks which others have already outlined. Depending on your industry sector and applicable privacy laws, that may be a risk which is simply not worth taking.
In my experience, even in shops where the developers rule, a discussion with the company's lawyers about the risks to which their approach exposes the company has good potential to bring about a change of behaviour.
March 2, 2022 at 10:38 pm
I feel like the use of prod data for testing is more accepted in a "look the other way" fashion that we want to admit. Long answer follows 🙂
The security issues of this are rarely understood by the development team, even if they built the security protections in the system.
Development teams - and I include QA staff in this definition of dev team - often come up against "bugs" (yes, they may not actually be bugs) that they try to replicate & can't. They immediately go to "it only occurs in production" or "must be a quirk in the data in use" to justify running tests against prod data.
Then there is just the complex enhancement that needs a lot of in-depth data to build and test. There's either a time constraint or an effort constraint that prevents some teams from building out scripts to create the in-depth data to build and then test against.
In a past job, we had a perfect example of this last case - building a billing & rebates system. The rules around rebates were complex, based on $$ turnover for clients, plus in some cases also number of transactions recorded. Throw in different currencies, different revenue for different sorts of transactions, and the billing system quickly gets very complex. The solution chosen to enable development, was for the (very senior) team member to grab regular copies of the live data to develop and test the billing system.
It was somewhat questionable if there was permission from further up the chain to do this, but that team member eventually lost his job. And the company changed their security policies AND levels of access granted to the production data to both make it clear this is a big fail to do, and prevent casual, accidental use of production data.
So, the use of production data was a shortcut. But it was enabled by lax policy, lax education AND lax implementation of the actual rules around that.
It wasn't just the developer doing the wrong thing.
It was that there was no explicit policy they had been shown & agreed to, and no well monitored restrictions in place to prevent it.
Very senior dev should have known better, & did deserve to lose his job. But companies collecting very sensitive data should have established policy that is reviewed regularly, along with implementations of the restrictions in that policy (as much as possible) that are also reviewed regularly.
March 3, 2022 at 8:26 am
If production data is something like sensor data with no personal or personally identifiable (PII) data then replicas of that data are useful for a number of purposes. But sensors produce a highly disciplined data stream anyway.
GDPR fines are up to €20 million or 4% of global revenue for breaches. Up to €10 million or 2% of global revenue for failing to have the process and procedures in place to handle PII.
One of the principles of GDPR is there must be a legitimate purpose with agreement from the customer for using their data. Another is that only those people within the organisation whose role is to fulfill that purpose should have access to that data. Using such data for a broad spectrum role like software development does not meet those principles.
In the wake of the last global financial crisis financial regulatory bodies were shown to be toothless tigers. That too got overhauled to be replaced by something hungry with sharp teeth and claws. If your company is listed on the stock exchange then, even if the data isn't personal, access to it can leave you vulnerable to accusations of insider trading. Once you've had a poor (or worse) auditors report the next few audits are far more detailed and onerous.
Worst case is that your company accounts won't be signed off and for a publicly listed company that is likely a mortal blow.
To be honest, I like the ethos of such a regime. I think the world is too casual with its data
March 4, 2022 at 12:52 pm
My most embarrassing data scrubbing mistake occurred years ago when working at a food distributor. Our first automated system was order entry where union CRT operators entered several hundred orders a day into a flat-file system. Picking documents for each order were batch printed 24 hours a day and taken to a warehouse full of union workers to load on trucks for delivery by union drivers. After the trucks were loaded and ready, the picking tickets were returned to the CRT operators for corrections, product substitutions, and batch invoice printing from the order file. It's been so long I don't remember how I did it, but suddenly I 'scrubbed' ALL the order data. Our open-order file was EMPTY. There was no record of the invoices that had been produced and none for the orders yet to be finalized and invoiced.
Trucks were delayed as we called in additional operators on overtime to re-enter the complete day of orders while drivers waited to be able to deliver the goods. not one of my best days.
Rick
Disaster Recovery = Backup ( Backup ( Your Backup ) )
March 30, 2022 at 11:10 am
Just got an email from Microsoft, an unfilled template. So even Microsoft can get this one wrong!
Viewing 14 posts - 1 through 13 (of 13 total)
You must be logged in to reply to this topic. Login to reply