SQLServerCentral / Editorials / SQLServerCentral.com / Rogue Algorithms / Latest Posts

RE: Rogue Algorithms

L' Eomot Inversé — Sat, 08 Sep 2012 10:16:08 GMT

[quote][b]Steve Jones - SSC Editor (9/6/2012)[/b][hr][quote][b]Scott D. Jacobson (9/6/2012)[/b][hr][quote][b]John Hanrahan (9/6/2012)[/b][hr]Steve,I'm surprised no one mentioned agile development practices. It seems like agile is another word for minimal testing. That's fine when the consequences of your code are small but maybe not so much when it's a stock trading program. Do you know if they used 'agile' development. In the Seattle area it is the buzzword.John[/quote]I suggest you read up on agile programming. Unit testing and acceptance testing both play big roles in the process. Anyone who uses agile to mean "develop and deploy" with no testing whatsoever isn't doing agile. That's cowboy coding. See: [url=http://en.wikipedia.org/wiki/Agile_software_development#Characteristics]http://en.wikipedia.org/wiki/Agile_software_development#Characteristics[/url][/quote]I think that many people view Agile as a way to speed development and if that speed causes a lack of testing, that's OK.I think it's managers that need to read up more on Agile, CI, and other practices.[/quote]I agree with the idea that many people have a dangerous diea of what "Agile" is, and that managers need educating. I made some comments elsewhere in response to the idea that Agile is always a good thing, might as well repeat them here:-[hr]Sure, some people can do something sensible with Agile - after all it's only the bringing together of much that was best practise long before the term "Agile" was coined. On the other hand, many companies use something that they call "Agile" but is actually a distillation of the worst cowboy development practises imaginable. The majority of people using "Agile" fall into the latter group, not the former. So any survey of what Agile does in the software development inductry will inevitably give it a resounding panning, because most companies using something they call "Agile" are not using what the people at outfits like Pixar or Perforce call "Agile".This of course is caused by having utterly incompetent management in charge of deciding how development will be done, and by the fact that when utterly incompetent managers see a shiny new buzzword like "Agile" they go and skim-read enough about it to extract some disconneted misinterpretations that support their lunatic pre-conceived ideas that what technical people call best practise is just a way of making development too expensive and then go an impose those on the development teams under the banner of the shiny new buzzword.[hr]

RE: Rogue Algorithms

L' Eomot Inversé — Sat, 08 Sep 2012 10:09:41 GMT

[quote][b]Tom Brown (9/7/2012)[/b][hr]They have changed the numbers several times. London changed first, from 01 to 071 and 081 I think around 1990 - before computer systems were all that common, then the whole country changed in 1995 [url]http://www.independent.co.uk/news/uk/bt-braced-for-wrongnumber-barrage-on-phoneday-1615636.html[/url] which is the event where the rogue algorithm was used and London was changed 0171 / 0181. And a 3rd change happened around 2000 when London went to 0207 and 0208, by which time we had learned our lesson and got the brokers to do the changes manually.Maybe you missed it. How long have you lived in Titerrogaka now?[/quote]Yes, you're right. I had misremembered and conflated those two changes, although they were actually 10 years apart.

RE: Rogue Algorithms

Tom Brown — Fri, 07 Sep 2012 13:33:17 GMT

[quote][b]L' Eomot Inversé (9/6/2012)[/b][hr][quote][b]Tom Brown (9/6/2012)[/b][hr]Anyone from the UK remember BT Phone Day - back in the 1990s sometime, the UK phone company changed all number prefixes to add a 1 - so 071 became 0171 - and 021 became 0121 etc., Well I worked for this software company back then, whose clientelle were all insurance brokers - and pretty much survived on their phone lists. They would have a heck of a job changing all those numbers manually, there were a some exceptions, but few enough for some bright spark to have an idea well out of sync with his IQ.As part of the monthly update - sent out to over 250 brokers - with the insurance rating prices, an extra script was run.... can you see where this is going ... and yes it did happen. All instances of '0' in all the phone number field were replaced by '01' - sometimes causing field overflow, and failure of the update - so the person applying the update would attempt to run the update again - which would overwrite the backup the original update had made.I was a lowly support person at the time and still shudder at the fallout. The company still exists, though the name has changed a few times.[/quote]Your description of the change is broken - adding a 1 to the front of the area code was for non-London numbers only, the London numbers didn't chage from 01... to 011... but to something beginning 02.If the algrithm you described was applied to London numbers that would be another source of catastrophe.[/quote]They have changed the numbers several times. London changed first, from 01 to 071 and 081 I think around 1990 - before computer systems were all that common, then the whole country changed in 1995 [url]http://www.independent.co.uk/news/uk/bt-braced-for-wrongnumber-barrage-on-phoneday-1615636.html[/url] which is the event where the rogue algorithm was used and London was changed 0171 / 0181. And a 3rd change happened around 2000 when London went to 0207 and 0208, by which time we had learned our lesson and got the brokers to do the changes manually.Maybe you missed it. How long have you lived in Titerrogaka now?

RE: Rogue Algorithms

Jeff Moden — Fri, 07 Sep 2012 09:37:35 GMT

[quote][b]Scott D. Jacobson (9/6/2012)[/b][hr][quote][b]John Hanrahan (9/6/2012)[/b][hr]Steve,I'm surprised no one mentioned algile development practices. It seems like agile is another word for minimal testing. That's fine when the consequences of your code are small but maybe not so much when it's a stock trading program. Do you know if they used 'agile' development. In the Seattle area it is the buzzword.John[/quote]I suggest you read up on agile programming. Unit testing and acceptance testing both play big roles in the process. Anyone who uses agile to mean "develop and deploy" with no testing whatsoever isn't doing agile. That's cowboy coding. See: [url=http://en.wikipedia.org/wiki/Agile_software_development#Characteristics]http://en.wikipedia.org/wiki/Agile_software_development#Characteristics[/url][/quote]+1.

RE: Rogue Algorithms

L' Eomot Inversé — Thu, 06 Sep 2012 16:03:53 GMT

[quote][b]Tom Brown (9/6/2012)[/b][hr]Anyone from the UK remember BT Phone Day - back in the 1990s sometime, the UK phone company changed all number prefixes to add a 1 - so 071 became 0171 - and 021 became 0121 etc., Well I worked for this software company back then, whose clientelle were all insurance brokers - and pretty much survived on their phone lists. They would have a heck of a job changing all those numbers manually, there were a some exceptions, but few enough for some bright spark to have an idea well out of sync with his IQ.As part of the monthly update - sent out to over 250 brokers - with the insurance rating prices, an extra script was run.... can you see where this is going ... and yes it did happen. All instances of '0' in all the phone number field were replaced by '01' - sometimes causing field overflow, and failure of the update - so the person applying the update would attempt to run the update again - which would overwrite the backup the original update had made.I was a lowly support person at the time and still shudder at the fallout. The company still exists, though the name has changed a few times.[/quote]Your description of the change is broken - adding a 1 to the front of the area code was for non-London numbers only, the London numbers didn't chage from 01... to 011... but to something beginning 02.If the algrithm you described was applied to London numbers that would be another source of catastrophe.

RE: Rogue Algorithms

Lynn Pettis — Thu, 06 Sep 2012 13:09:56 GMT

[quote][b]Tom Brown (9/6/2012)[/b][hr][quote][b]Lynn Pettis (9/6/2012)[/b][hr][quote][b]Tom Brown (9/6/2012)[/b][hr]Anyone from the UK remember BT Phone Day - back in the 1990s sometime, the UK phone company changed all number prefixes to add a 1 - so 071 became 0171 - and 021 became 0121 etc., Well I worked for this software company back then, whose clientelle were all insurance brokers - and pretty much survived on their phone lists. They would have a heck of a job changing all those numbers manually, there were a some exceptions, but few enough for some bright spark to have an idea well out of sync with his IQ.As part of the monthly update - sent out to over 250 brokers - with the insurance rating prices, an extra script was run.... can you see where this is going ... and yes it did happen. All instances of '0' in all the phone number field were replaced by '01' - sometimes causing field overflow, and failure of the update - so the person applying the update would attempt to run the update again - which would overwrite the backup the original update had made.I was a lowly support person at the time and still shudder at the fallout. The company still exists, though the name has changed a few times.[/quote]Where was QA to test the update?[/quote]Well I'm sure the developers did their own testing using suitable test dataPhone 01234567 changed to 011234567 - so it all right for release innit?[/quote]Now that takes me back a ways. Had a developer add code in an application to handle memberships. Worked great when that particular 'vendor' was used, but she failed to test for all the others. Totally broke the application and we had to roll back a version until she figured out what she had done wrong.

RE: Rogue Algorithms

Scott D. Jacobson — Thu, 06 Sep 2012 13:08:28 GMT

[quote][b]Tom Brown (9/6/2012)[/b][hr][quote][b]Lynn Pettis (9/6/2012)[/b][hr][quote][b]Tom Brown (9/6/2012)[/b][hr]Anyone from the UK remember BT Phone Day - back in the 1990s sometime, the UK phone company changed all number prefixes to add a 1 - so 071 became 0171 - and 021 became 0121 etc., Well I worked for this software company back then, whose clientelle were all insurance brokers - and pretty much survived on their phone lists. They would have a heck of a job changing all those numbers manually, there were a some exceptions, but few enough for some bright spark to have an idea well out of sync with his IQ.As part of the monthly update - sent out to over 250 brokers - with the insurance rating prices, an extra script was run.... can you see where this is going ... and yes it did happen. All instances of '0' in all the phone number field were replaced by '01' - sometimes causing field overflow, and failure of the update - so the person applying the update would attempt to run the update again - which would overwrite the backup the original update had made.I was a lowly support person at the time and still shudder at the fallout. The company still exists, though the name has changed a few times.[/quote]Where was QA to test the update?[/quote]Well I'm sure the developers did their own testing using suitable test dataPhone 01234567 changed to 011234567 - so it all right for release innit?[/quote]I think as we all know, nothing can take the place of good planning. In IT, it's not [i]if [/i]things go wrong but [i]when[/i]. Having a good plan for when things go wrong could prevent this. "Just run the update again" obviously isn't the right thing to do since the update is creating a backup. If you run it again, you just backed up the bad data. I don't think you have to make this kind of mistake to learn the lesson. Just plan better.

RE: Rogue Algorithms

Tom Brown — Thu, 06 Sep 2012 13:01:27 GMT

[quote][b]Lynn Pettis (9/6/2012)[/b][hr][quote][b]Tom Brown (9/6/2012)[/b][hr]Anyone from the UK remember BT Phone Day - back in the 1990s sometime, the UK phone company changed all number prefixes to add a 1 - so 071 became 0171 - and 021 became 0121 etc., Well I worked for this software company back then, whose clientelle were all insurance brokers - and pretty much survived on their phone lists. They would have a heck of a job changing all those numbers manually, there were a some exceptions, but few enough for some bright spark to have an idea well out of sync with his IQ.As part of the monthly update - sent out to over 250 brokers - with the insurance rating prices, an extra script was run.... can you see where this is going ... and yes it did happen. All instances of '0' in all the phone number field were replaced by '01' - sometimes causing field overflow, and failure of the update - so the person applying the update would attempt to run the update again - which would overwrite the backup the original update had made.I was a lowly support person at the time and still shudder at the fallout. The company still exists, though the name has changed a few times.[/quote]Where was QA to test the update?[/quote]Well I'm sure the developers did their own testing using suitable test dataPhone 01234567 changed to 011234567 - so it all right for release innit?

RE: Rogue Algorithms

Lynn Pettis — Thu, 06 Sep 2012 12:57:38 GMT

RE: Rogue Algorithms

Tom Brown — Thu, 06 Sep 2012 12:54:39 GMT

Anyone from the UK remember BT Phone Day - back in the 1990s sometime, the UK phone company changed all number prefixes to add a 1 - so 071 became 0171 - and 021 became 0121 etc., Well I worked for this software company back then, whose clientelle were all insurance brokers - and pretty much survived on their phone lists. They would have a heck of a job changing all those numbers manually, there were a some exceptions, but few enough for some bright spark to have an idea well out of sync with his IQ.As part of the monthly update - sent out to over 250 brokers - with the insurance rating prices, an extra script was run.... can you see where this is going ... and yes it did happen. All instances of '0' in all the phone number field were replaced by '01' - sometimes causing field overflow, and failure of the update - so the person applying the update would attempt to run the update again - which would overwrite the backup the original update had made.I was a lowly support person at the time and still shudder at the fallout. The company still exists, though the name has changed a few times.

RE: Rogue Algorithms

jay-h — Thu, 06 Sep 2012 11:16:14 GMT

[quote] ... (remember this checking has to happen in microseconds). ...[/quote]Trading programs in the Asia Pacific rim have to be careful about leap seconds which are added to the year every so often as those markets are active when the correction is applied.

RE: Rogue Algorithms

Matt Miller (#4) — Thu, 06 Sep 2012 10:21:24 GMT

[quote][b]Lynn Pettis (9/6/2012)[/b][hr][quote][b]John Hanrahan (9/6/2012)[/b][hr]I laugh and laugh. Agile often means (to the managers in charge) get it done and skip whatever you think you can. That may not be the book definition but it's how many companies operate and I bet not a one calls what they do cowboy coding. I will say that the bigger the company I've worked with the better it gets (I worked as a consultant for almost 20 years) as in more testing etc.[/quote]Then they aren't using Agile as it is supposed to be done. Testing is still an integral part of the development process.[/quote]Who's to say it's either's fault. It could be any number of root causes: - the checking algorithm (the one that validates that the sell price is higher that the price) doesn't get promoted at the same page as the "perform the trade" service because it's too slow (remember this checking has to happen in microseconds). So the thing you tested functionally in QA isn't what actually made it to prod. - the ranges are data-driven (so the ranges to buy and sell are in a data structure). The ranges were completely sane at the time when it was promoted, but someone then adjusted the table AFTER the fact, but only updated the "buy" price, and not the "sell" price. - a data caching issue. - etc...Without a root cause analysis, it's going to be difficult to assign blame to anyone. I do think you need to consider a LOT more controls when dealing with data not within your control, whether for attacks like injection, or for someone or something trying to "game" your system, or simply that there are lots of other players like you in the market which means things might change outside of your control.

RE: Rogue Algorithms

Lynn Pettis — Thu, 06 Sep 2012 10:12:18 GMT

[quote][b]John Hanrahan (9/6/2012)[/b][hr]Scott I definitely agree that it's the coder's responsibility. I managed a group of 8-12 programmers for a few years and we used agile practices but we included robust testing, code reviews, etc. but ultimately the quality of the product was the lead programmer responsibility (and he/she was held accountable).[/quote]Looks like you gave them the resources and tools to do the job correctly. Unfortunately, there are probably just as many managers out there that don't.

RE: Rogue Algorithms

John Hanrahan — Thu, 06 Sep 2012 10:06:29 GMT

Scott I definitely agree that it's the coder's responsibility. I managed a group of 8-12 programmers for a few years and we used agile practices but we included robust testing, code reviews, etc. but ultimately the quality of the product was the lead programmer responsibility (and he/she was held accountable).

RE: Rogue Algorithms

Lynn Pettis — Thu, 06 Sep 2012 10:02:33 GMT

Not at all. The people writing the code are responsible for what they write, but they also have to answer to management. If management doesn't give them the time or resources, what are they supposed to do?

RE: Rogue Algorithms

Scott D. Jacobson — Thu, 06 Sep 2012 09:49:39 GMT

So far we have "it's management's fault" and that's all? I agree that's probably more often the case than not but doesn't the person writing the code bear some responsibility for what they've created?

RE: Rogue Algorithms

Lynn Pettis — Thu, 06 Sep 2012 09:40:38 GMT

[quote][b]John Hanrahan (9/6/2012)[/b][hr]I laugh and laugh. Agile often means (to the managers in charge) get it done and skip whatever you think you can. That may not be the book definition but it's how many companies operate and I bet not a one calls what they do cowboy coding. I will say that the bigger the company I've worked with the better it gets (I worked as a consultant for almost 20 years) as in more testing etc.[/quote]Then they aren't using Agile as it is supposed to be done. Testing is still an integral part of the development process.

RE: Rogue Algorithms

John Hanrahan — Thu, 06 Sep 2012 09:29:43 GMT

I laugh and laugh. Agile often means (to the managers in charge) get it done and skip whatever you think you can. That may not be the book definition but it's how many companies operate and I bet not a one calls what they do cowboy coding. I will say that the bigger the company I've worked with the better it gets (I worked as a consultant for almost 20 years) as in more testing etc.

RE: Rogue Algorithms

Steve Jones - SSC Editor — Thu, 06 Sep 2012 09:22:23 GMT

[quote][b]Scott D. Jacobson (9/6/2012)[/b][hr][quote][b]John Hanrahan (9/6/2012)[/b][hr]Steve,I'm surprised no one mentioned agile development practices. It seems like agile is another word for minimal testing. That's fine when the consequences of your code are small but maybe not so much when it's a stock trading program. Do you know if they used 'agile' development. In the Seattle area it is the buzzword.John[/quote]I suggest you read up on agile programming. Unit testing and acceptance testing both play big roles in the process. Anyone who uses agile to mean "develop and deploy" with no testing whatsoever isn't doing agile. That's cowboy coding. See: [url=http://en.wikipedia.org/wiki/Agile_software_development#Characteristics]http://en.wikipedia.org/wiki/Agile_software_development#Characteristics[/url][/quote]I think that many people view Agile as a way to speed development and if that speed causes a lack of testing, that's OK.I think it's managers that need to read up more on Agile, CI, and other practices.

RE: Rogue Algorithms

Scott D. Jacobson — Thu, 06 Sep 2012 09:06:43 GMT

[quote][b]John Hanrahan (9/6/2012)[/b][hr]Steve,I'm surprised no one mentioned algile development practices. It seems like agile is another word for minimal testing. That's fine when the consequences of your code are small but maybe not so much when it's a stock trading program. Do you know if they used 'agile' development. In the Seattle area it is the buzzword.John[/quote]I suggest you read up on agile programming. Unit testing and acceptance testing both play big roles in the process. Anyone who uses agile to mean "develop and deploy" with no testing whatsoever isn't doing agile. That's cowboy coding. See: [url=http://en.wikipedia.org/wiki/Agile_software_development#Characteristics]http://en.wikipedia.org/wiki/Agile_software_development#Characteristics[/url]

RE: Rogue Algorithms

John Hanrahan — Thu, 06 Sep 2012 08:54:12 GMT

Steve,I'm surprised no one mentioned algile development practices. It seems like agile is another word for minimal testing. That's fine when the consequences of your code are small but maybe not so much when it's a stock trading program. Do you know if they used 'agile' development. In the Seattle area it is the buzzword.John

RE: Rogue Algorithms

RBarryYoung — Wed, 05 Sep 2012 18:36:03 GMT

[quote][b]Miles Neale (9/5/2012)[/b][hr]... Best read and understand parts of TAOCP vols 4 and 5 to get background and then find other particulars unique to the challenge you are faced with. ...[/quote]Huh? Sorry, I don't get this at all, what is in Volumes 4 and 5 that specifically applies to this discussion? I have in fact read every bit of Knuth's master work that actually exists. Volume 4 is on Combinatorial Algorithms, and Volume 5 doesn't exist yet. So what are you talking about and why wouldn't volumes 1, 2, and 3 apply equally well?

RE: Rogue Algorithms

cdonlan 18448 — Wed, 05 Sep 2012 12:39:06 GMT

[quote][b]Miles Neale (9/5/2012)[/b][hr]If you code what you think and not what you know the eventual result is failure. Best read and understand parts of TAOCP vols 4 and 5 to get background and then find other particulars unique to the challenge you are faced with. I fear for those who have "no fear" and 'just do it" when it comes to development of personal or cowboy algorthms in real business. You have to understand the business, and the data to make information and informed choices. Without the application of true wisdom we are condemed to foolishness and folly.End rant, restart work!M. [/quote]Agreed. When I was a junior programmer, I was sent to the business unit I was going to be supporting and spent the next 6 months learning the business. After that it was almost 6 months of doing nothing but learning the change control process inside and out by moving code in and out of test for my team. Probably doesn't happen anywhere any more. While there are probably more efficient ways of doing it, the general idea is still very valid. (Sorry for the old geezer story....but man I'm not that old! ):-)

RE: Rogue Algorithms

Miles Neale — Wed, 05 Sep 2012 09:56:13 GMT

If you code what you think and not what you know the eventual result is failure. Best read and understand parts of TAOCP vols 4 and 5 to get background and then find other particulars unique to the challenge you are faced with. I fear for those who have "no fear" and 'just do it" when it comes to development of personal or cowboy algorthms in real business. You have to understand the business, and the data to make information and informed choices. Without the application of true wisdom we are condemed to foolishness and folly.End rant, restart work!M.

RE: Rogue Algorithms

cdonlan 18448 — Wed, 05 Sep 2012 08:41:59 GMT

Knight Capital is useful ammo when countering dangerous management demands on your development time lines. Usually management tries to hide 'problems' like this, but that one was just too big. We don't know all facts yet, but I'm willing to bet management was the primary culprit. Risk management is their game and they obviously dropped ball. To survive, they had to sell a large piece of their equity to their competitors like TD Ameritrade who now basically own them. Too many shops think change control is source control or that prototyping in production is somehow clever (or some other crazy short-cut) but I would argue that every software manager should have a healthy appreciation of the core elements of the System Development Life Cycle like planning, analysis, design, testing, etc. In my universe, these concepts are as immutable as gravity or electromagnetism. Ignore them at your own risk.

RE: Rogue Algorithms

TravisDBA — Wed, 05 Sep 2012 08:37:12 GMT

"Even more unfortunate is the fact some managers never learn that. "Actually, that is very easy for them when they are only concerned about themselves first.:-D

RE: Rogue Algorithms

jfogel — Wed, 05 Sep 2012 07:07:39 GMT

I'm about half way in to Automate This which is a book about the rise of the use of algorithms and it is a very interesting book. It can be pretty spooky when if then else goes wrong.

RE: Rogue Algorithms

jay-h — Wed, 05 Sep 2012 06:58:05 GMT

Something to worry about when algorithms are being used to control our cars:http://jalopnik.com/5648126/volvo-pedestrian-avoidance-crash-test-fails-spectacularly

Rogue Algorithms

Steve Jones - SSC Editor — Tue, 04 Sep 2012 22:07:03 GMT

Comments posted to this topic are about the item [B]Rogue Algorithms[/B]