SQLServerCentral / Editorials / SQLServerCentral.com / NoSQL Is Not Everywhere / Latest Posts

RE: NoSQL Is Not Everywhere

ron.wright — Wed, 15 Dec 2010 16:47:02 GMT

We are now sponsoring an open source JBDC driver for Amazon's SimpleDB called SimpleJDBC.For more information:www.jnetdirect.com/index.php/products/simplejdbc.html

RE: NoSQL Is Not Everywhere

Andrew Peterson-472853 — Wed, 22 Sep 2010 15:11:57 GMT

Just like Steve noted, it is a tool. ACID databases were targeted for financial transaction systems, where ACID was a must, and hard, known relations existed. And the ACID recovery model adds overhead. Vertica, Greenplum and the Hadoop/BigTable/MapReduce/Cassandra tools add great value to loose data where relations are not known ahead of time.Its all about using the right tool.

RE: NoSQL Is Not Everywhere

David.Poole — Tue, 21 Sep 2010 15:26:21 GMT

I'm not sure is a solution such as Ab Initio counts as NoSQL but if so its industrial strength in a way that would make I.K.Brunel's work look a bit flimsy.

RE: NoSQL Is Not Everywhere

mtillman-921105 — Tue, 21 Sep 2010 14:15:18 GMT

[quote][b]Eric Russell 13013 (9/21/2010)[/b][hr][quote][b]mtillman-921105 (9/21/2010)[/b][hr]Well, we knew that VFP was a dead-end road once it was skipped over when .NET came ou... Hey, wait a minute! When's MS going to include .NET stuff in SQL Server? :angry:(And no, CLR doesn't count, either.)[/quote]Give me an example of what .NET stuff you would like so see [i][in][/i] SQL Server.[/quote]Actually, I was just joking. :-DSteve, my apologies for hijacking the thread here - we seem to have gotten a little off topic.

RE: NoSQL Is Not Everywhere

Eric M Russell — Tue, 21 Sep 2010 13:59:35 GMT

[quote][b]mtillman-921105 (9/21/2010)[/b][hr]Well, we knew that VFP was a dead-end road once it was skipped over when .NET came ou... Hey, wait a minute! When's MS going to include .NET stuff in SQL Server? :angry:(And no, CLR doesn't count, either.)[/quote]Give me an example of what .NET stuff you would like so see [i][in][/i] SQL Server.

RE: NoSQL Is Not Everywhere

mtillman-921105 — Tue, 21 Sep 2010 13:49:45 GMT

[quote]You're right, VFP 3.0, which was released in '95, made some improvements. There was this database container thing that would group related table and enforce referential integrity. Also, the newer .CDX indexes were linked to the .DBF files and would automatically open when the table was opened. The language and IDE was far more advanced, but still it was not client server, just a collection of files sitting on a file server, and the files would go bad on your from time to time. Now the IDE for VFP 3.0 was way ahead of Visual Basic 3.0; it supported object classes, OOP, compiling GUI controls, etc. Visual Basic didn't support that same level of functionality until VB.NET in 2000. [/quote]Well, we knew that VFP was a dead-end road once it was skipped over when .NET came ou... Hey, wait a minute! When's MS going to include .NET stuff in SQL Server? :angry:(And no, CLR doesn't count, either.)

RE: NoSQL Is Not Everywhere

Eric M Russell — Tue, 21 Sep 2010 13:33:23 GMT

[quote][b]mtillman-921105 (9/21/2010)[/b][hr]Yeah Eric, I wish I could say that the last line of FoxPro code I wrote was in '98. I waited too long to switch over, but I was stuck there for a while. By the way, FoxPro has a bad rep, but for a lot of applications that are running today - FoxPro would do just fine if MS wouldn't cap it off at 2GB per table. The newer versions are a lot more advanced than a lot of people think. I shake my head every time I see some minor 4 GB database running on SQL Server (or worse, Oracle) since it's like swatting a fly with a nuclear bomb. I have to hand it to MS's marketing for convincing everybody that it's "SQL Server or nothing" basically. (Or "If you must, then use Access.")Anyway, yes, back then the state of the desktop database tech was not impressive by today's standards. But still, you didn't have to resort to using single, stand-alone flat tables - you could normalize the data (and make it "relational").[/quote]You're right, VFP 3.0, which was released in '95, made some improvements. There was this database container thing that would group related table and enforce referential integrity. Also, the newer .CDX indexes were linked to the .DBF files and would automatically open when the table was opened. The language and IDE was far more advanced, but still it was not client server, just a collection of files sitting on a file server, and the files would go bad on your from time to time. Now the IDE for VFP 3.0 was way ahead of Visual Basic 3.0; it supported object classes, OOP, compiling GUI controls, etc. Visual Basic didn't support that same level of functionality until VB.NET in 2000.

RE: NoSQL Is Not Everywhere

mtillman-921105 — Tue, 21 Sep 2010 13:14:12 GMT

[quote][b]Eric Russell 13013 (9/21/2010)[/b][hr][quote][b]mtillman-921105 (9/21/2010)[/b][hr][quote][b]Eric Russell 13013 (9/21/2010)[/b][hr][quote][b]mtillman-921105 (9/21/2010)[/b][hr]To what extent were relational DBMS created because drive space wasn't cheap and processing power was weak? Big, flat tables weren't helpful in either case years ago. Now I've staked my career on SQL Server, so don't just jump all over me. But I've seen this point brought up elsewhere and wondered how bad it would be to just use flat tables. All the other software may not be dead, but it [i]is[/i] bloated! Why not databases too? Summary tables could be created on the side too, which of course helps performance. Video graphics cards work in a brute force, but kind of a dumb way. But they do work - right?Someone respectfully prove me wrong. Please. (Because frankly, I'm a little worried.) :w00t:[/quote]I've still got a shrink wrapped copy of FoxPro 2.6 for DOS, if you're willing to trade for a SQL Server 2008 Enterprise license.[/quote]That is funny, but FoxPro was relational believe it or not. (I was a FoxPro programmer and have actually worked with that version.) In fact, I think that even dBase III+ was relational, for that matter. [/quote]I think the last line of code I wrote in FoxPro was back in '98. A FoxPro table was basically a flat text file with a header record on the front containing column names and the offset of the last record. I know there are still people out there using FoxPro or some variant, because over the years I occasionally have to import from it into SQL Server. As for FoxPro 2.6 being relational, we would create a cursor by opening a table and then joining it with other tables using a primeval version of SQL, and then loop between each record. Many FP developers would simply put all their columns in the same table and called it a 'database'. Indexes were stored in seperate files. When opening a table, we had to explicitly opening it's related indexes too, and if we didn't, then any records we inserted or updated would simply not be updated in the index. You knew when the indexes were not updated correctly whenever users called in the middle of the night to complain about missing data. There was no client / server engine, the tables were put out on a network file share, and every page read had to be pulled across the wire to the user's local PC. If a user shut down their PC with the application open (or if the application crashed), then whatever tables or indexes were open at the time would get corrupted and couldn't be opened without re-indexing or sometimes manually editing the table header with a HEX editor. :w00t:Processing, spooling and caching were all pulled across the wire to the user's PC, so it made more efficient use of (the server's) disk storage space. I recall that sometimes whenever a user was running a really large report, the spool tmp file would fill their local HD and crash the PC. I'll gladly shell out $1,000 of my own money for a couple more hard disks just to avoid retrograding back to ISAM databases and applications. [/quote]Yeah Eric, I wish I could say that the last line of FoxPro code I wrote was in '98. I waited too long to switch over, but I was stuck there for a while. By the way, FoxPro has a bad rep, but for a lot of applications that are running today - FoxPro would do just fine if MS wouldn't cap it off at 2GB per table. The newer versions are a lot more advanced than a lot of people think. I shake my head every time I see some minor 4 GB database running on SQL Server (or worse, Oracle) since it's like swatting a fly with a nuclear bomb. I have to hand it to MS's marketing for convincing everybody that it's "SQL Server or nothing" basically. (Or "If you must, then use Access.")Anyway, yes, back then the state of the desktop database tech was not impressive by today's standards. But still, you didn't have to resort to using single, stand-alone flat tables - you could normalize the data (and make it "relational").

RE: NoSQL Is Not Everywhere

Eric M Russell — Tue, 21 Sep 2010 12:55:21 GMT

[quote][b]mtillman-921105 (9/21/2010)[/b][hr][quote][b]Eric Russell 13013 (9/21/2010)[/b][hr][quote][b]mtillman-921105 (9/21/2010)[/b][hr]To what extent were relational DBMS created because drive space wasn't cheap and processing power was weak? Big, flat tables weren't helpful in either case years ago. Now I've staked my career on SQL Server, so don't just jump all over me. But I've seen this point brought up elsewhere and wondered how bad it would be to just use flat tables. All the other software may not be dead, but it [i]is[/i] bloated! Why not databases too? Summary tables could be created on the side too, which of course helps performance. Video graphics cards work in a brute force, but kind of a dumb way. But they do work - right?Someone respectfully prove me wrong. Please. (Because frankly, I'm a little worried.) :w00t:[/quote]I've still got a shrink wrapped copy of FoxPro 2.6 for DOS, if you're willing to trade for a SQL Server 2008 Enterprise license.[/quote]That is funny, but FoxPro was relational believe it or not. (I was a FoxPro programmer and have actually worked with that version.) In fact, I think that even dBase III+ was relational, for that matter. [/quote]I think the last line of code I wrote in FoxPro was back in '98. A FoxPro table was basically a flat text file with a header record on the front containing column names and the offset of the last record. I know there are still people out there using FoxPro or some variant, because over the years I occasionally have to import from it into SQL Server. As for FoxPro 2.6 being relational, we would create a cursor by opening a table and then joining it with other tables using a primeval version of SQL, and then loop between each record. Many FP developers would simply put all their columns in the same table and called it a 'database'. Indexes were stored in seperate files. When opening a table, we had to explicitly opening it's related indexes too, and if we didn't, then any records we inserted or updated would simply not be updated in the index. You knew when the indexes were not updated correctly whenever users called in the middle of the night to complain about missing data. There was no client / server engine, the tables were put out on a network file share, and every page read had to be pulled across the wire to the user's local PC. If a user shut down their PC with the application open (or if the application crashed), then whatever tables or indexes were open at the time would get corrupted and couldn't be opened without re-indexing or sometimes manually editing the table header with a HEX editor. :w00t:Processing, spooling and caching were all pulled across the wire to the user's PC, so it made more efficient use of (the server's) disk storage space. I recall that sometimes whenever a user was running a really large report, the spool tmp file would fill their local HD and crash the PC. I'll gladly shell out $1,000 of my own money for a couple more hard disks just to avoid retrograding back to ISAM databases and applications.

RE: NoSQL Is Not Everywhere

mtillman-921105 — Tue, 21 Sep 2010 12:37:52 GMT

When I stop and think about it, this all so really sad... The thought all of my precious, lonely tables never again being able to have [i]relations[/i]... you know - I don't know that I can bare it...

RE: NoSQL Is Not Everywhere

IceDread — Tue, 21 Sep 2010 11:58:23 GMT

NOSql is an interesting platform. I personally hope that one day, somehow, nosql would be usable in the common rdbms platforms. That could be interesting.

RE: NoSQL Is Not Everywhere

Steve Jones - SSC Editor — Tue, 21 Sep 2010 11:36:28 GMT

We did worry about the rep of this site using MySQL, and used that to negotiate a better license from MS at one point. :-P

RE: NoSQL Is Not Everywhere

mtillman-921105 — Tue, 21 Sep 2010 11:33:24 GMT

[quote][b]Eric Russell 13013 (9/21/2010)[/b][hr][quote][b]mtillman-921105 (9/21/2010)[/b][hr]To what extent were relational DBMS created because drive space wasn't cheap and processing power was weak? Big, flat tables weren't helpful in either case years ago. Now I've staked my career on SQL Server, so don't just jump all over me. But I've seen this point brought up elsewhere and wondered how bad it would be to just use flat tables. All the other software may not be dead, but it [i]is[/i] bloated! Why not databases too? Summary tables could be created on the side too, which of course helps performance. Video graphics cards work in a brute force, but kind of a dumb way. But they do work - right?Someone respectfully prove me wrong. Please. (Because frankly, I'm a little worried.) :w00t:[/quote]I've still got a shrink wrapped copy of FoxPro 2.6 for DOS, if you're willing to trade for a SQL Server 2008 Enterprise license.[/quote]That is funny, but FoxPro was relational believe it or not. (I was a FoxPro programmer and have actually worked with that version.) In fact, I think that even dBase III+ was relational, for that matter.

RE: NoSQL Is Not Everywhere

Eric M Russell — Tue, 21 Sep 2010 11:00:15 GMT

[quote][b]mtillman-921105 (9/21/2010)[/b][hr]To what extent were relational DBMS created because drive space wasn't cheap and processing power was weak? Big, flat tables weren't helpful in either case years ago. Now I've staked my career on SQL Server, so don't just jump all over me. But I've seen this point brought up elsewhere and wondered how bad it would be to just use flat tables. All the other software may not be dead, but it [i]is[/i] bloated! Why not databases too? Summary tables could be created on the side too, which of course helps performance. Video graphics cards work in a brute force, but kind of a dumb way. But they do work - right?Someone respectfully prove me wrong. Please. (Because frankly, I'm a little worried.) :w00t:[/quote]I've still got a shrink wrapped copy of FoxPro 2.6 for DOS, if you're willing to trade for a SQL Server 2008 Enterprise license.

RE: NoSQL Is Not Everywhere

Eric M Russell — Tue, 21 Sep 2010 10:54:53 GMT

[quote][b]Steve Jones - Editor (9/21/2010)[/b][hr]... We discarded MySQL and PostgreSQL for this site early on, even when funds were tight, because we weren't familiar with them and they were still young in 2001. I think that's a valid choice.[/quote]For this site? Had word got out that SQLServerCentral.com was powered by MySQL, then Sun Micro would have had a field day.

RE: NoSQL Is Not Everywhere

Steve Jones - SSC Editor — Tue, 21 Sep 2010 10:47:43 GMT

[quote][b]David.Poole (9/21/2010)[/b][hr]I almost feel that SQL Server should have a NoSQL provision in its storage engine. Being able to define a database in zero recovery mode so the overhead of the 2phase commit is eliminated. SQL Server already uses file based error logging for the SQL Server Agent and MSSQLSERVER service.[/quote]I agree here. There are lots of places where we don't need 2 phase commit.

RE: NoSQL Is Not Everywhere

SQLRNNR — Tue, 21 Sep 2010 10:23:31 GMT

[quote][b]David.Poole (9/21/2010)[/b][hr]I've done a lot of reading around NoSQL. First of all it stands for Not Only SQL and not NO SQL!There are a number of NoSQL solutions who claim to be web scalable but when you dig under the covers not so many are. It's worth looking at Simon Munro's blog on the subject. Simon spoke well at the SQL Bits conference in London earlier this year.The premise of NoSQL is that for applications where data loss is not the end of the world then the overhead of an RDBMS is overkill. They work on the principle of BASE rather than ACID (Basically Available, Soft state, Eventually Consistent).Quest software are even releasing a version of TOAD that can talk to cloud DBs and some NoSQL contenders.Where NoSQL is particularly useful is for complex web forms processing where a customer can ping back and forth through the user journey to suit themselves. You really wouldn't want to be trying that with a relational schema as it would require loads of nullable fields and a lot of update statements.Until the form is completed there is no need to commit the data to an RDBMS. Service interruptions are rare enough that a missed form is not the end of the world. If we committed data into an RDBMS then such service interruptions would result in orphaned data and other data quality issues.At present the NoSQL solutions I have seen are very much programmers databases rather than everymans databases though TOAD should at least be a step on the right path.[/quote]Thanks - that is highly informative.

RE: NoSQL Is Not Everywhere

David.Poole — Tue, 21 Sep 2010 10:07:26 GMT

Actually there is a lot of value in using NoSQL in ecommerce sites.One of the biggest headaches facing DBAs today is IO. Processing power is cheap, memory is cheap, disk space is cheap but until large, cheap, reliable SSD or better technology becomes available IO is the bottleneck.NoSQL offers a potential solution so as DBAs we should be embracing the possibilities it offers while scrutinising its use in our role as data professionals.Lets consider the eCommerce site. We are adding things to a shopping basket, as are many tens of thousands of people, but until we commit to pay for it the world won't stop if there is a hiccup in the shopping basket service. Provided at the point of commit we have full resilience and robustness a NoSQL solution could save a lot of IO.The reality is that hardware is very reliable and very high uptime achievable so the glitches that a proper RDBMS will handle via 2 phase commit are few and far between. Obviously I wouldn't trust sensitive financial transactions to any existing NoSQL solution but as a tool to act as a transient store NoSQL offers a lot of advantages.I almost feel that SQL Server should have a NoSQL provision in its storage engine. Being able to define a database in zero recovery mode so the overhead of the 2phase commit is eliminated. SQL Server already uses file based error logging for the SQL Server Agent and MSSQLSERVER service.Imagine a reindex process where the overhead of TempDB was halved! Obviously the reindex process would have to be sensitive to glitches even if all the reindex process did was abort in the event of a problem.

RE: NoSQL Is Not Everywhere

mtillman-921105 — Tue, 21 Sep 2010 09:52:13 GMT

[quote][b]Steve Jones - Editor (9/21/2010)[/b][hr][quote][b]mtillman-921105 (9/21/2010)[/b][hr]To what extent were relational DBMS created because drive space wasn't cheap and processing power was weak? Big, flat tables weren't helpful in either case years ago. [/quote]Perhaps that was some of the driving force, though I think that the ACID properties weren't available in flat files either. These days we're going back to XML sometimes and that's almost like scanning a flat file in many instances.Not sure about graphic cards. I know there have been some improvements over time, moving to new algorithms, but not sure about the brute force storing all pixels inside them.[/quote]At least as far as I know, there is no earth-shaking new technology in video cards. There is parallel processing and running multiple cards at once, but that's really just more processing power. I think the emphasis in that area lately has been more cards, more memory and faster processors.

RE: NoSQL Is Not Everywhere

Steve Jones - SSC Editor — Tue, 21 Sep 2010 09:45:15 GMT

[quote][b]Joe Johnson-482549 (9/21/2010)[/b][hr]Hi Steve,On the comment you made about hoping an architect doesn't disclude something because it is new ... we do this all of the time due to risk. You analyze the new-ness of a technology and make the determination (hopefully using a standardized methodology). If it is too far outside your comfort level or provides complexities that timing will not allow, you use a "more trusted" technology.[/quote]Fair points, Joe, and I would disagree that someone isn't discarding it because it's "new", but because they aren't familiar with it. We discarded MySQL and PostgreSQL for this site early on, even when funds were tight, because we weren't familiar with them and they were still young in 2001. I think that's a valid choice.

RE: NoSQL Is Not Everywhere

Steve Jones - SSC Editor — Tue, 21 Sep 2010 09:43:10 GMT

[quote][b]mtillman-921105 (9/21/2010)[/b][hr]To what extent were relational DBMS created because drive space wasn't cheap and processing power was weak? Big, flat tables weren't helpful in either case years ago. [/quote]Perhaps that was some of the driving force, though I think that the ACID properties weren't available in flat files either. These days we're going back to XML sometimes and that's almost like scanning a flat file in many instances.Not sure about graphic cards. I know there have been some improvements over time, moving to new algorithms, but not sure about the brute force storing all pixels inside them.

RE: NoSQL Is Not Everywhere

mtillman-921105 — Tue, 21 Sep 2010 09:27:40 GMT

To what extent were relational DBMS created because drive space wasn't cheap and processing power was weak? Big, flat tables weren't helpful in either case years ago. Now I've staked my career on SQL Server, so don't just jump all over me. But I've seen this point brought up elsewhere and wondered how bad it would be to just use flat tables. All the other software may not be dead, but it [i]is[/i] bloated! Why not databases too? Summary tables could be created on the side too, which of course helps performance. Video graphics cards work in a brute force, but kind of a dumb way. But they do work - right?Someone respectfully prove me wrong. Please. (Because frankly, I'm a little worried.) :w00t:

RE: NoSQL Is Not Everywhere

Eric M Russell — Tue, 21 Sep 2010 09:10:20 GMT

[quote][b]David.Poole (9/21/2010)[/b][hr]The premise of NoSQL is that for applications where data loss is not the end of the world then the overhead of an RDBMS is overkill. They work on the principle of BASE rather than ACID (Basically Available, Soft state, Eventually Consistent).[/quote]I guess it's not the end of the world if a teenage girl's latest facebook post gets lost in the nightly data defrag shuffle (although she'd certainly disagree), but this BASE concept doesn't sound like something the financial, healthcare, or eCommerce industries would find beneficial to their core operations.

RE: NoSQL Is Not Everywhere

Joe Johnson-482549 — Tue, 21 Sep 2010 08:40:08 GMT

Hi Steve,On the comment you made about hoping an architect doesn't disclude something because it is new ... we do this all of the time due to risk. You analyze the new-ness of a technology and make the determination (hopefully using a standardized methodology). If it is too far outside your comfort level or provides complexities that timing will not allow, you use a "more trusted" technology.The Digg video points out that some of their go live issues were with the Cassandra code. It doesn't point out that they came up due to the newness of the technology, but if the system had been implemented in a larger instance or had been stress tested to a higher capacity, it would probably have shown some of the inconsistencies with data corruption. They made a choice in their implementation to be on the bleeding edge and they paid for it with 1000 cuts.Regards,Joe

RE: NoSQL Is Not Everywhere

David.Poole — Tue, 21 Sep 2010 08:10:42 GMT

[quote]When was the last time anyone wrote a DOS batch file? [/quote]About 10 minutes ago. I use them quite a bit for tweaking continuous integration and deployment scripts.

RE: NoSQL Is Not Everywhere

nelsonj-902869 — Tue, 21 Sep 2010 07:57:22 GMT

Steve:As an ex-COBOL analyst and IBM360 JCL writer, I appreciate your final comments in the article. It has been said many, many times that COBOL is old hat and will die....Long live COBOL....even tho I've cut the cord and gone with SQL, VB.Net and many other client server applications and tools in my current labors. While it is great to be able to use or try new products, there are times when the "cost" of learning a new technology is many times more expensive than doing the same task with aged processes or languages. When was the last time anyone wrote a DOS batch file? For me, about two months ago to handle a remote process on another server. Some technologies will never die.

RE: NoSQL Is Not Everywhere

Tobar — Tue, 21 Sep 2010 05:40:41 GMT

Steve, if you worked with some of the people I have in my job history you would be disappointed a lot.The truth is any "great next best think" is only as good as the people who construct it. Classic garbage in garbage out. There are times when the product is poor, but there are many more times when the people are poor.

RE: NoSQL Is Not Everywhere

David.Poole — Tue, 21 Sep 2010 02:09:59 GMT

I've done a lot of reading around NoSQL. First of all it stands for Not Only SQL and not NO SQL!There are a number of NoSQL solutions who claim to be web scalable but when you dig under the covers not so many are. It's worth looking at Simon Munro's blog on the subject. Simon spoke well at the SQL Bits conference in London earlier this year.The premise of NoSQL is that for applications where data loss is not the end of the world then the overhead of an RDBMS is overkill. They work on the principle of BASE rather than ACID (Basically Available, Soft state, Eventually Consistent).Quest software are even releasing a version of TOAD that can talk to cloud DBs and some NoSQL contenders.Where NoSQL is particularly useful is for complex web forms processing where a customer can ping back and forth through the user journey to suit themselves. You really wouldn't want to be trying that with a relational schema as it would require loads of nullable fields and a lot of update statements.Until the form is completed there is no need to commit the data to an RDBMS. Service interruptions are rare enough that a missed form is not the end of the world. If we committed data into an RDBMS then such service interruptions would result in orphaned data and other data quality issues.At present the NoSQL solutions I have seen are very much programmers databases rather than everymans databases though TOAD should at least be a step on the right path.

RE: NoSQL Is Not Everywhere

Nakul Vachhrajani — Mon, 20 Sep 2010 22:37:05 GMT

I believe "NoSQL" can be used for "Service-only" offerings (Examples can be application/widgets that simply gather data by crawling across the web and display the same - the WeatherBug and Exchange Rate reference boards for example) that process data, but do not store it. If they store data it may not be SQLServer or RDBMS (may be just a flat-file), but they will then have some Query language involved in data extraction and loading.I would say that where data is involved, some query language is also involved.

NoSQL Is Not Everywhere

Steve Jones - SSC Editor — Mon, 20 Sep 2010 21:01:48 GMT

Comments posted to this topic are about the item [B]NoSQL Is Not Everywhere[/B]