A Swiss Army Knife Might Not Make the Best Shovel

, 2018-03-01

You might have noticed that I’ve been pretty quiet as of late. We’re working on a super top secret internal project here at my company, and we’ve got the need to ingest a LOT of data around the clock for some analytics work. My preferred DBMS is, of course, Microsoft SQL Server, and like a lot of DBAs, we want to make this swiss army knife of a relational DB platform do everything we can dream up. Thankfully, it can perform most of the tasks we throw at it pretty well. But, the pragmatist in me asks – “Is this the best tool for the job?”. Because we’re just starting this project, we can step back a bit and look at all of our options.

For our project, we do not want to deal with a datacenter of our own. Yes, we’re known as on-prem virtualization enthusiasts, and there in certainly many reasons for keeping things on-prem for some time to come, but cloud is the right choice for us for this project. We’re working on the cloud platforms just as much as we are on-prem these days, and we’re seeing the shift occurring in the industry.

Take a look at the costs of SQL Server licensing in the cloud. To design a SQL Server that can consume upwards of a few million data points a minute, we’re likely to need to spend quite a bit of capital on this platform. It’s just overkill for a straightforward ingestion then export platform. Then, we need to accommodate high availability, disaster recovery, reporting, and analytics needs.

Hmm.

Cloud brings some differences that might be advantageous here. We’re partial to MS Azure as a cloud platform for our company internally, so what does Azure have that can help us?

Azure Cosmos DB.

Now, it’s not as simple as that. Cosmos DB is a collection of APIs for different database types under the hood.

Each one are used differently, and all of the options include many differences in operation and architecture. Of the five listed platform APIs, which should we use? That’s a good question. For this particular project, we want the ability to store tons of inbound data and then will be pulling it out for analysis. Azure Table API seems to work best for this purpose.

SO! Over the next few months, expect a number of blog posts from me here exploring Azure Table on Cosmos DB and the questions, challenges, and experiences we have on ramping up on this new platform.

Rate

Share

Share

Rate

Related content

Database Mirroring FAQ: Can a 2008 SQL instance be used as the witness for a 2005 database mirroring setup?

Question: Can a 2008 SQL instance be used as the witness for a 2005 database mirroring setup? This question was sent to me via email. My reply follows. Can a 2008 SQL instance be used as the witness for a 2005 database mirroring setup? Databases to be mirrored are currently running on 2005 SQL instances but will be upgraded to 2008 SQL in the near future.

2009-02-23

1,567 reads

Networking - Part 4

You may want to read Part 1 , Part 2 , and Part 3 before continuing. This time around I'd like to talk about social networking. We'll start with social networking. Facebook, MySpace, and Twitter are all good examples of using technology to let...

2009-02-17

1,530 reads

Speaking at Community Events - More Thoughts

Last week I posted Speaking at Community Events - Time to Raise the Bar?, a first cut at talking about to what degree we should require experience for speakers at events like SQLSaturday as well as when it might be appropriate to add additional focus/limitations on the presentations that are accepted. I've got a few more thoughts on the topic this week, and I look forward to your comments.

2009-02-13

360 reads