March 21, 2003 at 12:00 am
Comments posted to this topic are about the content posted at http://www.sqlservercentral.com/columnists/sjones/howdoyoureviewaproduct.asp
March 24, 2003 at 2:50 am
Steve,
How about posting a "to be reviewed" notice on a product then those of us who use the product can submit our experiences.
Keep it as a submissions only i.e. only you and Brian can read them, until there is a substantial body of information on the product over and above your own review.
Pick out the best bits and write the review.
If 10K DBA's say the product sucks then the vendor ought to be told. This would be a valuable service to the vendor.
Even if we don't say it sucks the fact that there is a potentially large user base to give feedback should be an attractive selling proposition to any vendor.
I think your comment that one DBA can rave about a tool yet that tool can leave another cold is important. If this situation occurs I think it is an important review point WHY there was a discrepancy.
As an aside Douglas Adams (Hitch hikers guide to the galaxy) said that there is technology and there are useful tools. They can be distinguished because the former needs a manual!
March 24, 2003 at 6:39 am
Steve,
I think David's idea for getting us out here in the DBA community to review products is a good idea too. It would have the advantage of taking some of the load and time consuming tasks off you guys, and it would be transparent and impartial, as none of us are in the quandry that you are in with regard to advertising revenue. It would also make the site more attractive to vendors / potential advertisers, as you would be able to process the reviews of products as soon as they come onto the market, rather than maybe months later.
In addition, customers would have even more confidence in reviews as there is no doubt about advertiser influence.
You could ask for purely written reviews from external reviewers, which would be useful, but might actually increase your workload -or- have a voting form with marks from 1 to 10 (radio buttons) for categories like performance, bugginess, price, value for money, ease of installation, documentation etc. You could also have a comments space available for those who wanted to add more detailed opinions.
Obviously you'd have to make sure people could only vote once for this, maybe only allow members to vote, to make it more difficult for a potentially unscrupulous vendor to nobble the votes.
I also agree with David's comment about telling a vendor if a product sucks. If it does, and they are told before the review gets published, then they fix it before it gets re-reviewed (perhaps with a note that it initially had problems), and fix it well, that at least indicates vendor responsiveness and an interest in getting things sorted, which is always an issue when buying software. Some small vendors are excellent at this, some global ones abysmal - we ought to know before shelling out cash, whether personal or corporate. It might even have the side effect of encouraging vendors to tighten up their testing before releasing a product.
Another thought : How about combining the above idea with 'group tests', like the PC magazines do when comparing different motherboards, graphics cards, etc ? This would give those of us thinking of purchasing a product a better indicator of how good products are relative to each other, rather than in isolation. Perhaps difficult as often only a few tools of each type exist, but may be worth considering.
Also, how about mailing out to all members asking if they would like to review products, and selecting 10-15 at random for each product? From a vendors point of view, this might have to be a 30-day evaluation version, as they'd be giving a complete version away to the very people they're trying to sell to. Obviously anyone connected with the vendor might have to be excluded, or at least declare their connection (aside: I work in a software house which is soon to release a product which we would very much like SQLServerCentral.com to review, so please take any of my comments in light of this - I can see it from a vendor's and a customer's point of view).
Yet another idea : How about making more of promoting the 'Rated Product' logos on vendor sites? It helps vendors sell their products by linking back to SQLServerCentral's review pages, which in turn helps customers decide what/whether to buy, which increases traffic to the site. This helps to support advertising costs, as you can point to more people hitting the site when selling ad space to vendors.
Finally, it might be worth looking at the Amazon book-review model, which works well and always seems to give a good indication of how good a product is. Never been disappointed as a result of buying anything which I've had my eye on, which numerous others have also recommended from experience. Guess what we're talking about here is automating, centralising and speeding up the 'word of mouth' experience - probably the best way to recommend anything.
Edited by - jonreade on 03/24/2003 07:19:30 AM
Jon
March 24, 2003 at 8:27 am
Jon and David,
The reader review is a great idea, and something that quite honestly, I hadn't thought of. I know that I often read printed reviews in PCMag and like places with a jaded eye as they aren't necessarily written to test the product as I'd like to. And it's something I've struggled with when writing reviews.
I'm wary of the reader vote model. To me, and I've seen it on smaller sites, though not Amazon, that authors or people with a vested interest spend time to skew the pool to their side. We've debated about making votes mandatory for the polls, and we still may. Not sure we could do it for products and what's to stop a small vendor from having 5 people take a few hours, register new names, and get a few hundred votes? Keep in mind that 1,000 votes on the sidebar poll is a good day, so a vendor could easily swing a poll (either way).
We'll talk about that and touch base with some vendors to see what they think.
And keep the ideas coming. As well as any skepticisms or criticisms of reviews (mine or in general).
Thanks
Steve Jones
March 24, 2003 at 9:14 am
Someone got caught on Amazon hyping their own book so the skewing issue is real.
I think that user reviews have to be
It depends on what the vendors are looking for.
If it is free sales collateral then they are going to get peed off with criticism. Arguing as devil's advocate, the knowledge that SSC can drop hand-grenades makes a positive review from SSC much more worth while.
If they are after honest feedback for future releases then they will have struck gold!
I wouldn't be adverse to a vendor saying "This is the problem we were trying to address; this is why we think we suceeded". Provided it isn't total BS then we can say offer a safe opinion.
I think we would have to make an effort to present criticism in a positive light. A diplomat can tell you to go to hell and have you looking forward to the trip! A review needs to be clear when there is criticism but be fair.
Make the vendor feel good about what they have achieved but let them know where they need to pick up a few points.
March 24, 2003 at 11:03 am
March 24, 2003 at 12:01 pm
One suggestion:
why don't you make a scale of values, applicable to all packages.When DBA's test they will have a guideline as what to look for in every package
as a small note ==I have downloaded NGSSquirrell but I could not used it fully because the functionality is of course reduced for free downloads. So my impression about is was chunked.
respectfully
LMT
March 24, 2003 at 12:36 pm
Steve,
I liked the article and can understand the constraints. I like the "brutally honest" review with the understanding that some reviews will not be posted because they are too harmful for the vendor. In reality, I really do my own reviews anyway before I purchase a product but, I use the site to see what kind of information you (SSC admins) have already gathered on the product. If it is not there I either assume that the product has not been reviewed or that there was too much negative to be posted.
So, my recommendation is to leave it alone and keep on doing what you are doing. It has been working great for me and it appears that the site has done pretty well with the current format as well.
My $0.02.
David
David
@SQLTentmaker“He is no fool who gives what he cannot keep to gain that which he cannot lose” - Jim Elliot
March 24, 2003 at 1:40 pm
Be Professional
Check out http://www.tomshardware.com/. These guys test computer hardware and have a knack for being "Brutally honest" without being overly harsh. They perform detailed tests under fair conditions and stick to the facts, sometimes comparing different products and explaining the differences between them. Tom's is pretty well regarded in the do-it-yourself community by the DIY-ers and the manufacturers; in fact, it is not unknown for their articles to generate change in a manufacturers design and manufacturing processes. Once the vendors come to see the quality and integrity of SSC they won’t send anything for review unless it is polished.
Be Selective
Don't let the general member pool do the reviews. I don't mean to sound picky but I think you should limit the people that do the reviews to those that have published good articles at SSC. That gives people an incentive to increase the SSC content as well as gauge their ability to present a well-written review. There is also the time required to adequately test the product. I would love to do some reviews of SQL Server products but if I had the time to do that I would have written at least two articles for SSC and would post to the forums a whole lot more. This should give you a broad enough user base to spread out the review work while keeping the quality at the high level I have come to expect at SSC.
Bryant E. Byrd, MCDBA
SQL Server DBA/Systems Engineer
Intellithought, Inc.
Edited by - Tatsu on 03/24/2003 1:41:10 PM
[font="Tahoma"]Bryant E. Byrd, BSSE MCDBA MCAD[/font]
Business Intelligence Administrator
MSBI Administration Blog
March 24, 2003 at 3:08 pm
Good points, though I don't know how Tom grew the site. If we lost too many advertisers, the site would get shut down. Just too expensive.
Still, it's an interesting perspective and something we will be talking about here at SSC.
Steve Jones
March 25, 2003 at 2:15 am
I think Steve's article raised some great points, and some excellent ideas from those who have posted comments too.
I like the idea of those who make the most contributions reviewing new products, they should have enough experience to make valuable comments to people like me who are new to SQL Server. At the bottom rungs of the SQL Server knowledge ladder, I rely on more experienced DBAs to give me honest answers and opinions.
So - please keep reviews brutally honest. There are too many products out there with huge marketing budgets behind them, which are promoted as being professional, useful tools. It often turns out that this is not the case and the marketing is better than the product. I have installed and immediately uninstalled too many products, purely because they are badly thought out, badly designed and unfortunately pointless.
Also, if a selection of DBAs from different environments test a product, it will be interesting to see how it works in each of their circumstances. If the contributors who review products could give us a brief description of their SQL Server environments and how a product works or doesn't work on small or large systems, this would be both interesting and beneficial to vendors and potential customers alike.
March 25, 2003 at 8:06 am
Does anyone on this site work for a vendor?
If so, could you chip into this discussion?
March 25, 2003 at 12:22 pm
We are contacting vendors to let them know about this thread already. Most of our advertisers read the site, so I'm sure they are aware.
As an FYI, I work for JD Edwards, Andy for a small company in Florida and Brian for Alltel.
Steve Jones
March 26, 2003 at 12:20 am
I would have thought some sort of combination model may be appropriate. Let vendors submit their own "informational/brochure-ware" details about their product - SSC would then classify the product into some sort of heirarchy, with (perhaps) subsequent versions listed also - this would allow us to check things like dependancies (needs Windows 2000 Server) etc.
It would also allow us to see what products exist in the same marketplace (i.e. what alternatives are there). Personally I'd rather see a complete list of products than a selection based on someone's opinion. As you said, your requirements are likely to be completely different to mine.
Then, on top of that, registered/regular members would be allowed to submit reviews about the product. As suggested, these could probably be editted by yourselves to ensure bias and/or ambush is prevented.
I sometimes think a general rating is often useless (e.g. 4 out of 5). I'd like to see the main features rated. For example, a product may have a swish GUI interface but really suck in one of the key areas it promotes as being a "leader" in. I've seen this time and time again. However, if I'm looking for a specific feature, I may be happy to trade off other areas to get what I'm really after.
This approach may offer more advertising opportunities where vendors can really tart up their "brochure ware" or synergenic products might like to advertise with a specific partner.
Anyhow, just a thought.
March 27, 2003 at 6:08 am
I like the complete listing idea. Sometimes it's hard enough just to figure out what tools/options there are in a given space.
To follow up on Steve's comment in the interest of full disclosure, I work for Hancock Information Group.
Andy
Viewing 15 posts - 1 through 15 (of 15 total)
You must be logged in to reply to this topic. Login to reply