April 21, 2013 at 6:46 am
Comments posted to this topic are about the item A Billion Transactions
April 21, 2013 at 8:08 am
Steve, thanks a great read particularly if one follows your link to
It has me wondering if someone will create software equal for our private homes ... and if someone does what it will mean to our individual energy bills, and our life styles .
April 22, 2013 at 4:23 am
I don't think 1 billion transactions per day is that scary any more but then it depends what you mean by transaction. A transaction can be a simple record insert or a batch of records inserted as a single transaction.
It also depends on the read/write mix of traffic to the server, page life expectancy, buffer cache hit ratio etc.
One of our lines of business pushes 100 million records per day precisely how that breaks down to transactions is hard to say but we have another 48 lines of business and running a mix of versions of SQL Server from 2000 onwards and they certainly aren't on high end boxes.
April 22, 2013 at 5:48 am
It would be great if stock market activity was evenly distributed throughout the day but the actual profile has huge spikes. Here in the US, normal market hours are 9:30-16:00 Eastern and much of the activity on a typical day is at market open as well shortly before market close. Also, news events can cause spikes at any time throughout the day.
This sort of transaction profile is a real challenge in capacity planning because one must plan for the peaks rather than the average and every millisecond counts due to the time-sensitive nature of these transactions.
April 22, 2013 at 7:10 am
I also think that we are going to have learn to throw away more data too. This will probably be a rough road in data management. Those of you who are major influencers in such things will have to assist businesses in deciding, perhaps more than ever, what data to keep in what granularity and for how long. This may be BI data related process and it feels somewhat like that to me. Perhaps this is just part of some of your jobs in the data warehousing arena.
Of course, I hope Steve wasn't slighting us developers as being unprofessional when he said "SQL Server professionals and developers" 😛
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
April 22, 2013 at 8:46 am
Gary Varga (4/22/2013)
I also think that we are going to have learn to throw away more data too. This will probably be a rough road in data management. Those of you who are major influencers in such things will have to assist businesses in deciding, perhaps more than ever, what data to keep in what granularity and for how long. This may be BI data related process and it feels somewhat like that to me. Perhaps this is just part of some of your jobs in the data warehousing arena.Of course, I hope Steve wasn't slighting us developers as being unprofessional when he said "SQL Server professionals and developers" 😛
Good points, and I think over time the idea of classifying data, aggregating some of it and tossing other parts will become more important.
I wasn't trying to slight anyone, but make sure our focus is on building these apps that help people in some way, not just meet some spec without necessarily accomplishing something meaningful for the task.
April 22, 2013 at 9:39 am
I remember when I had to stop using int in favor of bigint in the identity field of a few of my tables. I don't think I have any that record a billion a day yet though.
April 22, 2013 at 9:53 am
Steve Jones - SSC Editor (4/22/2013)
Gary Varga (4/22/2013)
I also think that we are going to have learn to throw away more data too. This will probably be a rough road in data management. Those of you who are major influencers in such things will have to assist businesses in deciding, perhaps more than ever, what data to keep in what granularity and for how long. This may be BI data related process and it feels somewhat like that to me. Perhaps this is just part of some of your jobs in the data warehousing arena.Of course, I hope Steve wasn't slighting us developers as being unprofessional when he said "SQL Server professionals and developers" 😛
Good points, and I think over time the idea of classifying data, aggregating some of it and tossing other parts will become more important.
I wasn't trying to slight anyone, but make sure our focus is on building these apps that help people in some way, not just meet some spec without necessarily accomplishing something meaningful for the task.
For the most part I agree, meet needs do not just meet specifications. But some well thought out specifications can be very useful. They should be subject to review, subject to summarizations etc, but some of those specifications are really nice to have in place since the developer might not know what the ancillary need is the specifications meet. 🙂
Not all gray hairs are Dinosaurs!
April 22, 2013 at 11:24 am
Steve Jones - SSC Editor (4/22/2013)
Gary Varga (4/22/2013)
I also think that we are going to have learn to throw away more data too. This will probably be a rough road in data management. Those of you who are major influencers in such things will have to assist businesses in deciding, perhaps more than ever, what data to keep in what granularity and for how long. This may be BI data related process and it feels somewhat like that to me. Perhaps this is just part of some of your jobs in the data warehousing arena.Of course, I hope Steve wasn't slighting us developers as being unprofessional when he said "SQL Server professionals and developers" 😛
Good points, and I think over time the idea of classifying data, aggregating some of it and tossing other parts will become more important.
I wasn't trying to slight anyone, but make sure our focus is on building these apps that help people in some way, not just meet some spec without necessarily accomplishing something meaningful for the task.
I don't think anyone would really think you were trying to slight anyone. A good editorial again. Thanks.
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
April 23, 2013 at 5:39 pm
bitbucket-25253 (4/21/2013)
Steve, thanks a great read particularly if one follows your link toIt has me wondering if someone will create software equal for our private homes ... and if someone does what it will mean to our individual energy bills, and our life styles .
+1
Some of those systems are already becoming available for homeowners. I can foresee many more over the years.
Jason...AKA CirqueDeSQLeil
_______________________________________________
I have given a name to my pain...MCM SQL Server, MVP
SQL RNNR
Posting Performance Based Questions - Gail Shaw[/url]
Learn Extended Events
April 24, 2013 at 7:23 am
“Give me a lot of data and I’ll save the world." That ought to be the model for data analysts. As SQL Server professionals and developers, we should be helping others to do just that.
Taken as a blanked statement, I couldn’t be disagreeing more with this! I am of the opinion we actually should not be recording everything just because we technically have the means to and are asked/ordered to do so!
People in power always want to record just about everything, in case it will be needed in the future or to look and feel in control. Using it just rarely happens and when it does, it is for the worst of the worst purposes, namely to infringe on freedoms. Besides data without properly designed structure, clear context and no standardization is next to useless.
I believe the trend seen in the last few decades is becoming more like a poison in more than one way. The problem with improved IT and more data is that the people in power get more ways to sucker everyone else with rules that seem practical enaugh to push trough. They have access and thus the means to make it happen which is destabilizing societies. We socially aren't developed enaugh to prevent outright abuse. So for now it is a lot like live living in tribal society with deep rivalries and totalitarian control that suddenly develop automatic weapons…it is quite dangerous!
IT driven efficiency gains which brought us prosperity in the private sector, do simply not work in public sectors. There is simply too much power concentrated in those sectors. Power corrupts, and this is undoing most of the gains.
This might seem politically loaded, but be fair, we still live in a politically driven world. Information technology was only born a few decades ago and only recently really took off (as in part of everyones life). This makes this a relevant angle. Always consider the context and possible contexts with what you do!
Viewing 11 posts - 1 through 10 (of 10 total)
You must be logged in to reply to this topic. Login to reply