October 15, 2016 at 12:09 pm
Comments posted to this topic are about the item Yet Another Database
October 16, 2016 at 6:22 am
Sometimes I feel you are reading my mind. 😛
I'm struggling pretty hard with what would be a great solution for the analytical database. I would love feedback if anyone has any for my case too.
I've had similar thoughts on whether or not yet another database is needed for my use cases. You know, those thoughts like, "Is SQL Server the best tool for this particular job?" and "Does SQL Server solve these problems?" Sometimes it's hard to find the answer to those questions, especially when you're data is large, granular and has to support BIG READS over BIG WRITES.
The most common answers I get is to partition, to summarize, and to standardize. But, what happens if you can't do that? What happens if you have to support queries that read hundreds of millions of granular records for constant reprocessing of large sets of data that popular user reporting? SQL Server can struggle to support that across billions of records unless you constantly scale up.
So, the quest for the best analytical database comes into play. I get a lot of NoSQL answers to my questions too. Yet, my data is not highly unstructured. I'm not looking for real-time analytics. My data is highly structured and batch. It's just very large and needs to be read differently than you normally canned reporting. Thus, I've been looking in yet another database to support those requirements. MongoDB came up recently because it's NoSQL, has new support for JOIN's, has actual indexing on documents, new aggregate functions and distributed processing across nodes.
Suddenly, SQL Server doesn't feel like the one true answer anymore. But, that's just on the outside. Still have to find time to get on the inside and really see.
October 17, 2016 at 8:01 am
If the queries aren't too intensive we just use production. For those over-the-top queries we use a recent backup in some staging type area, knowing that the data may not be up-to-the-minute.
October 17, 2016 at 2:28 pm
I suggest checking out Analytics Platform System (APS) for on-prem, or Azure SQL Data Warehouse[/url] for a cloud solution.
October 20, 2016 at 9:54 am
Iwas Bornready (10/17/2016)
If the queries aren't too intensive we just use production. For those over-the-top queries we use a recent backup in some staging type area, knowing that the data may not be up-to-the-minute.
I find that this is quite a common first step (and in some cases final step).
Gaz
-- Stop your grinnin' and drop your linen...they're everywhere!!!
October 23, 2016 at 9:56 am
I follow that stuff also for nearly 30 years... and I want to add "why you don't need another programming language".
in my opinion people waste lot of enery and time in inventing "revolutionary" database concepts and languages, thereby neglecting that especially C/C++ (and previously PASCAL) had the idea of being extensible for solve any future programming challenges. Someone who's a full bred programmer can dissemble any problem and code that in C++ - more or less elegant - and it is possible to store everything in a row oriented database.
Over past 30 years colum/row oriented databes (with or without using the "relational thing" were dominant, because nearly all data was provided in that way. And before we had the object oriented world an object was just a "Struct" that could contain other structs or even functions / methods, didn't even need C++ for that. Other things will come and they will go... I'd guess MS SQL server and C++ / C# will still exist even in 30 Years, but everything else will be buried in the /NUL device.
October 24, 2016 at 6:28 am
andreasg 72151 (10/23/2016)
I follow that stuff also for nearly 30 years... and I want to add "why you don't need another programming language".in my opinion people waste lot of enery and time in inventing "revolutionary" database concepts and languages, thereby neglecting that especially C/C++ (and previously PASCAL) had the idea of being extensible for solve any future programming challenges. Someone who's a full bred programmer can dissemble any problem and code that in C++ - more or less elegant - and it is possible to store everything in a row oriented database.
Over past 30 years colum/row oriented databes (with or without using the "relational thing" were dominant, because nearly all data was provided in that way. And before we had the object oriented world an object was just a "Struct" that could contain other structs or even functions / methods, didn't even need C++ for that. Other things will come and they will go... I'd guess MS SQL server and C++ / C# will still exist even in 30 Years, but everything else will be buried in the /NUL device.
Well, it's the same reason we have other spoken and written languages, dialects and accents of those languages. People use what they are native to and what they feel is clear communication in their area. We all can't move to English, Spanish or something else. In our case, it's what the business users use. However, in most cases, it's just using what make sense for the job.
For example, it does not make sense to use C++ to query data. It's a very complex language and we shouldn't have to write our own binary trees and functions to store, sort and fetch data from. The same may apply to the Python. I can surely use C++ to do everything that I'm doing in Python, but I wouldn't use C++ to prototype a new enterprise application. It's too much work and I accomplish similar objects in C++ with Python with far less code and time.
As I mentioned in previous post, SQL Server is struggling on my end. It's easier to keep scaling up and up with SQL Server. Adding more memory, adding more CPU and other resources to meet my requirements. Sure, there are ways on the application level to distribute process the data. SQL Server also has ways to configure multiple SQL instances to simulate this as well. That's a lot of overhead and extremely expensive on enterprise levels when I can just us another database engine that may be open source and does it more natively.
All are tools that have good uses. Some can pretty much do anything. It doesn't mean it's the right tool for the job though. C++ by far is not the only language we need for every programming project.
Viewing 7 posts - 1 through 6 (of 6 total)
You must be logged in to reply to this topic. Login to reply