Click here to monitor SSC
SQLServerCentral is supported by Red Gate Software Ltd.
 
Log in  ::  Register  ::  Not logged in
 
 
 
        
Home       Members    Calendar    Who's On


Add to briefcase 12»»

Message Queues in Software Expand / Collapse
Author
Message
Posted Tuesday, January 15, 2013 11:01 PM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Today @ 4:10 PM
Points: 33,095, Visits: 15,202
Comments posted to this topic are about the item Message Queues in Software






Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1407605
Posted Wednesday, January 16, 2013 5:38 AM


SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Tuesday, June 17, 2014 11:56 PM
Points: 129, Visits: 183
Your link "10 uses for a message queue" should have read: "10 reasons to use message queues". I also would have liked a few examples.

5ilverFox
Namakwa Sands
South Africa
Post #1407781
Posted Wednesday, January 16, 2013 6:11 AM


SSCertifiable

SSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiableSSCertifiable

Group: General Forum Members
Last Login: Today @ 9:07 AM
Points: 5,200, Visits: 2,831
Coming from more of a programming background I have utilized message queues many times. Often MSMQ but also various other technologies including Service Broker. As always, great when applied appropriately. Asynchronous, distributed system design is often overlooked in this day of asynchronous programming!!!

Gaz

-- Stop your grinnin' and drop your linen...they're everywhere!!!
Post #1407795
Posted Wednesday, January 16, 2013 6:45 AM
Forum Newbie

Forum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum NewbieForum Newbie

Group: General Forum Members
Last Login: Tuesday, May 27, 2014 9:43 AM
Points: 8, Visits: 131
I used queues extensively a long...long...time ago when I developed manufacturing plant floor applications using PASCAL on a CTOS (Convergent Technologies Operating System) platform...we used the Queue Manager built into the platform that drove all the printers...worked great! We could create workflows and de-couple the slower processes to make sure they performed well on these speedy 80186 cpus...CTOS was great for that - but then again - it was purely message based and distributed to begin with! I've been eyeing Service Broker for some time now...and may use it in the future...I've been using SharePoint lists as a sort of queue for not-so-realtime applications (SSIS) to grab the data and bubble it up to the right people thru SharePoint...makes a good way to collect data from the an extranet where the users cannot write into the SQL db - but can write to SharePoint lists.
Post #1407811
Posted Wednesday, January 16, 2013 7:03 AM
SSC-Enthusiastic

SSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-EnthusiasticSSC-Enthusiastic

Group: General Forum Members
Last Login: Today @ 10:46 AM
Points: 141, Visits: 1,737
It would be nice to have more familiarity with the tools to troubleshoot service broker issues.
Post #1407823
Posted Wednesday, January 16, 2013 9:27 AM
Valued Member

Valued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued MemberValued Member

Group: General Forum Members
Last Login: Today @ 10:34 AM
Points: 62, Visits: 753
tl;dr - I can't see the point in Service Broker; it doesn't appear to do what it says on the tin and is too convoluted to use...
_________________
On the surface, it looks like an ideal way to get SQL Server to 'annouce' to your application layer that something has happened (rather than the application layer polling the database every so often to look for changes). I have admittedly not delved far into its workings but, when I did, I wrote it off as a non starter. I'm sure others have done the same.

For me, the problems are: obscurity, complexity, coupling and a lack of userland documentation.

Service Broker is a bit too hidden-away for most of my developers to even think about. This obscurity means that it is not a first choice solution for any problem. Not saying it should matter but, realistically, how useful is Management Studio for promoting/configuring/managing Service Broker. I can see an tree node for a Service Broker Endpoint (but can't do anything with it -- an empty Reports option and a Refresh option is not Management in anyone's Studio). Admittedly, the SSMS-side of things is much better in 2008/R2 but, if this feature was expected to be so useful in 2005, its tools should have been there from the get-go.

Secondly, the complexity. XML is not fun to work with. XML is especially not fun to work with in SQL Server. Sure, anyone can write a basic XML document; I'm sure most of us can also put something together with a nice set-based FOR XML statement and some of us can even get that data back out with the functions built into the XML datatype but that still leaves a lot of people (good, upstanding, professionals in their field) whose eyes just glaze over at the very sight of it. Besides, good XML (with schemas and transforms and validation and such) is a pain in the rear that even the most capable of us usually want to avoid.

Even for all its XML-ness (surely originally designed as a common 'language' for interoptability in data transfer), it still feels like the only thing that can talk to a Service Broker is...another Service Broker. So you have SQL Server talking to another SQL Server. Great, but we can already do that well-enough with Linked Servers -- at least we get to talk SQL to those (even if badly-written queries can lead to vast swathes of data going back and forward across the wire).

Despite all the talk about decoupling and being able to provide interfaces for another process to communicate with, it all still feels so ostensibly coupled. If it is supposed to be a decoupling messaging framework, surely there should be some example out there of how you can listen for these messages from your own appplications (or are you just expected to throw WCF/RabbitMQ/some other messaging system into a SQLCLR if you want to do that?).

There also doesn't appear to be any usable abstraction to work with this stuff in a way that allows portability to another major platform. I know, deep down, that is the point (i.e. to tie you to SQL Server) but most other features can be abstracted away in your app layer -- you don't usually have to rewrite the entire data access layer to switch from SQL Server to Oracle, but you would have to reimplement Service Broker in your new target platform (any real parallels to Oracle Queues?).

Finally, there just doesn't seem to be that much practical information out there about why you should use it. The Microsoft documentation is all 'this is how you do it' (and can be terse, at best), the evangelist blog writers are all 'this is how you do it' (and pretty much just blog about their particular implementation) but noone is really talking about what problems it can overcome and why you might want to consider it.

In the face of that, the howtos all cover the same sort of ground: read data from your table, transform it into a message in a given format and send it somewhere vs. listen for a message in a given format, read the data from it and store/process it. And that seems like an awful lot of work when you could just replicate the data between two servers and have a SQL Agent job poll the replicated table for new rows.

Ironically, when my developers have implemented solutions that would benefit from some sort of messaging, they've ultimately ended up storing those 'messages' in tables, and polling their own service apps. Not saying that it was right, just saying it was so. They understand them, have been using them for some time, and don't necessarily require any glorified string manipulation to get anything usable out of them.

At best, I can see it allow some decoupling between disparate SQL Server databases, whereby you can transform your data, stored in your table layout to a given common format, and they can transform their data, stored in their table layout to the same common format and both can just get along.

Maybe that is the point of Service Broker (and I've only just this second got it!).
Post #1407929
Posted Wednesday, January 16, 2013 9:46 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Today @ 2:45 PM
Points: 2,267, Visits: 1,324
The idea is great. The implementation has been challenging. We have explored the decoupling and writing almost everything ourselves and for the most part that has worked without much work and complete control by our developers. I have written and been involved with writing a number of asynchronous processes over the past decade and am well versed in web services and asynchronous batch processes that run on triggers or timers. They are very handy.

One or more posters also mentioned that there is the XML issue to deal with. And I agree that it is an issue. However, with a little research and experimentation you can find some very rich tools for working XML into a friend and not a problematic foe. Consuming web service output is not as bad as the wrap some give it. XML is different but it is not that hard. Creation of appropriate schema for large data collections over a large consortium of users is much more of a challenge. But once the schema is decided the use of it is not that bad.

That all said, the use of service brokers is good, and if you use something like MSMQ or write a similar process yourself it can give you a great advantage in how your processes perform.

M.


Not all gray hairs are Dinosaurs!
Post #1407947
Posted Wednesday, January 16, 2013 10:16 AM
SSC Rookie

SSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC RookieSSC Rookie

Group: General Forum Members
Last Login: Friday, July 18, 2014 1:46 PM
Points: 32, Visits: 85
I agree with what you have here, Miles. When I looked into the concept of using queueing for a problem I wanted to solve, it sounded like a perfect fit. Researching the implementation details left me feeling like the effort and complexity weren't worth the trouble in my case. This seems to be one of those places where a little something of the Microsoft touch is missing that would have made this incredibly useful to me. Maybe in the sea of additional capabilities queues seem to have I'm missing something, but I would have expected a basic sproc interface to get me up and running in a few minutes with minimal code, leaving all the tuning and tweaking up to those with more time invested (such as future me after I've come to know and love them):

1)define the queue
create queue MyQueue (@var1 int, @var2 int) /*here's my interface, too*/

2)define a listener as a sproc
create queueListener My Listener
@var1 int,
@var2 int
as
/*do some stuff*/

3)use the queue like an insert
insert into MyQueue values(1, 2)


As someone who works in SQL every day, I'm now able to take advantage of a queued architecture using just two concepts (ddl create and insert) I already know how to use. Give me that and I'm on board.
Post #1407985
Posted Wednesday, January 16, 2013 10:56 AM


SSC-Dedicated

SSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-DedicatedSSC-Dedicated

Group: Administrators
Last Login: Today @ 4:10 PM
Points: 33,095, Visits: 15,202
The XML part is definitely challenging for beginners. It's the same with Extended Events and a number of other systems introduced in R2/2012. I don't think it's that hard, but it's certainly off-putting. A little work to return a table of some sort would seem to me, to be a good direction for the SQL Server team.

In terms of explaining this as similar to linked servers, I think it's much more reliable and stronger. Linked Servers suffer from maintaining a connection and being able to reliably ensure that you can query through the LS. A queue works differently, and it provides guaranteed delivery, once the connectivity is up. This means it can work in disconnected scenarios.

It's really an architectural re-thinking of how you build an application.







Follow me on Twitter: @way0utwest

Forum Etiquette: How to post data/code on a forum to get the best help
Post #1408003
Posted Wednesday, January 16, 2013 11:20 AM
SSCrazy

SSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazySSCrazy

Group: General Forum Members
Last Login: Today @ 2:45 PM
Points: 2,267, Visits: 1,324
Steve Jones - SSC Editor (1/16/2013)
The XML part is definitely challenging for beginners. It's the same with Extended Events and a number of other systems introduced in R2/2012. I don't think it's that hard, but it's certainly off-putting. A little work to return a table of some sort would seem to me, to be a good direction for the SQL Server team.


I would agree that it is challenging for beginners. It is a different paradigm and requires some creativity and imagination about data. By taking an XML document and converting it to a in memory dataset you can gain access to the data without much problem. You just have to change your perspective. It is challenging and does take some time and effort. But you can wade through it and make it work.

M.


Not all gray hairs are Dinosaurs!
Post #1408011
« Prev Topic | Next Topic »

Add to briefcase 12»»

Permissions Expand / Collapse