Spasearch Ratings....Real or not?

[stuart]

I took a quote from a post that Vanguard did on another subject because this is a subject that I’m questioning myself and would like everyone’s input

That is correct; the new "Certified" rating is based off independent research and customer satisfaction.  This really is the first true independent rating.  This one cannot be bought.

So what your saying is the ratings cannot be skewed? I've been part of Spasearch pretty much from the beginning and I have always questioned how the ratings were done even though my brands always make it in the top.
 
I'm currently waiting today or tomorrow to run some of these questions by the founder of the program....
 
When they say this is a customer rating where did the customers come from?
 
Did they ask the manufactures for dealers and the dealers for customers?
 
If I'm a dealer do I give just customers that I know will give a good rating?
 
Were the customers people that submitted their info through spasearch to get a magazine and then bought a spa? If so have they even been delivered and have they had the spa long enough to know if it’s good? So is the rating then on ownership of these spas or shopping experience?
 
Are the customers counted by sheer #'s or by percentage of satisfied customers to spas sold by that manufacture?
 
Would most manufactures really give an outside source a true number of spas sold?
 
IMO, if you sent out a general questionnaire to spa owners you would find that just about everyone likes their spa. That being said I would further say that the concept of the magazine is by numbers of people responding by the brands rated. Those people are, most commonly, from the more intensely sold brands so they are more attached to the purchase than many others and would be more likely to respond.
 
It's a confusing thing...Here is an example;
Do you think after the rebate program and fake homeshow scams Master Spa used to sell the majority of their spas you would have more customers speaking favorably about them than many other brands?

I’m not saying that the ratings are wrong or “fixed”…I’m questioning the process used to derive what they are deeming as fact? We all should question these things yet we draw the battle lines and everyone rated defends while everyone not rated accuses and often it's falsely without fact (on both sides). Lets try and get to some fact and if it isn’t a real rating than lets stop supporting it.

Knowing the editor of spasearch I want to ask the questions rather than assume it’s all about the money. He seems to truly believe in the process used but I’m sure that there is a certain point where you have to look the best way to make the program profitable to publish and the more companies you exclude from the best rating the less of them support and buy the ads and magazines.

As industry pro's lets try to not just be myopic about our owns brands position but rather how the rating really works!

[Reese]

Like I said in the other thread, I'm skeptical that an organization that used to sell their "ratings" just like any other advertising, and published them in a manner that misled unsuspecting consumers, has suddenly found independence.  Even if their intentions are pure, they have a long way to go to gain credibility.

[Dr. Spa™ Ret.]

Well, I'm less skeptical as my thoughts are there hasn't really been a way to truely rate spas before. There may still not be a way, but this "seems" a bit better".

There's a similar type of rating program in my area for businesses. The way they work is that the rating company comes INTO your business and pulls names out of your files. It's a bit more difficult to "stuff the ballot box" this way.

[Tom]

I took a quote from a post that Vanguard did on another subject because this is a subject that I’m questioning myself and would like everyone’s input

I can only provide information relevant to our company, but perhaps it will help.

So what your saying is the ratings cannot be skewed?
I think you mean manipulated; it may be possible but IMO it is unlikely

I'm currently waiting today or tomorrow to run some of these questions by the founder of the program....
We'll look forward to hearing David Woods' comments!
 
When they say this is a customer rating where did the customers come from?
Arctic used a newsletter subscriber list.  Since this is an open newsletter to which anyone in the world can subscribe, the list likely included customers both satisfied and unsatisfied, prospects, and interested parties such as competitors.  Dealers were removed from the list for obvious reasons.  While it hardly represents our entire customer base, it was a broad sample that we felt was both representative and random.  
The manufacturers did not contact the customers nor did we deal with the responses; it was strictly third-party.  After the survey, we received a statistical analysis of the responses of our customers and a comparison of our scores to the average/high/low among the first 10 manufacturers to apply (of whom only 8 received certification)
 
Did they ask the manufacturers for dealers and the dealers for customers?
Not in our case.
 
If I'm a dealer do I give just customers that I know will give a good rating?
This was probably possible if someone were willing to go to that much work.
 
Were the customers people that submitted their info through spasearch to get a magazine and then bought a spa?
I have no way have knowing, but in our case that was extremely unlikely!

If so have they even been delivered and have they had the spa long enough to know if it’s good? So is the rating then on ownership of these spas or shopping experience?
Good question.  In our case, we asked a supplementary survey question about how long the respondent had owned their Arctic spa. This question was for our information only and was not part of the certification process. Over a third of our respondents had owned their spa for up to a year, just under a third have owned for two years, about 16% three years or more, the rest undetermined.  As you point out elsewhere, this skew may just be due to newer owners being more apt to respond to the survey.
 
Are the customers counted by sheer #'s or by percentage of satisfied customers to spas sold by that manufacture?
Neither.  There was a rating system that emphasized extreme (high or low) responses in order to spread out the scores.  You can probably find details on the spasearch site.
 
Would most manufactures really give an outside source a true number of spas sold?
Hmm, I think what you're asking is whether the rating accurately compares a company that sells 10,000 units annually with one that sells 100,000 units a year.  IMO, the rating method correctly treated this as irrelevant.
 
IMO, if you sent out a general questionnaire to spa owners you would find that just about everyone likes their spa.
Probably true, and a wonderful thing, too!

I would further say that the concept of the magazine is by numbers of people responding by the brands rated.
The statistical analysis was more sophisticated than that.

Those people are, most commonly, from the more intensely sold brands so they are more attached to the purchase than many others and would be more likely to respond.
An insightful comment.  Any survey suffers from this same inherent bias because it is based only on the people who are willing to respond.
 
I’m not saying that the ratings are wrong or “fixed”…I’m questioning the process used to derive what they are deeming as fact? We all should question these things...
Fair enough.

I’m sure that there is a certain point where you have to look the best way to make the program profitable to publish and the more companies you exclude from the best rating the less of them support and buy the ads and magazines.
You've kind of proved your own point!  The stated purpose of the program was to identify the top 10% of manufacturers. This means there will be some 13 to 15 who qualify in the end.  Will the others boycott the magazine/site and move their advertising $$ elsewhere?  Let's ask this question:  Can they afford NOT to advertise in a major consumer market?

If you've received the most recent Spasearch magazine, you'll note the pull-out ballot.   This is evidently an ongoing survey.

As industry pro's lets try to not just be myopic about our owns brands position but rather how the rating really works!
Okay, you've got my input.  The spasearch program may not be perfect but I think it's a step in the right direction.
Tom

[stuart]

Tom,
Thanks for your input!

You are correct, I did mean "Manipulated" and there is no doubt in any reasonable person's mind that it is possible, as far as it being "unlikely" that's what we should be trying to determine!

So this "newsletter subscriber letter" you refer to that you used...was that how spasearch received your customer info?

Was it your companies newsletter or did it come from spasearch?  

Do you have a copy of it online that you could share with us?

Did Arctic control the output and input of the survey in any way?

[Tom]

Was it your companies newsletter or did it come from spasearch?  Do you have a copy of it online that you could share with us?

It's ours, and you can subscribe or see past issues on our web site at http://www.arcticspas.com/index.php/en/show_content/68/68/

Did Arctic control the output and input of the survey in any way?

Our only input was in providing an emailing list for the survey (as well as all the documentation required for the other part of the certification - proof of membership in the BBB, the Pool & Hot Tub Council of Canada, all that).  As I mentioned, we also removed dealer store email addresses from the survey list, thinking that only fair. Individual store owners and sales people who had a subscription in their own names were left.  The survey was administered and the results analyzed neither by us nor by spasearch; a third party  company was used in each case.  

[Pathfinder]

Tom, do you know what percentage of this this was contacted?

[Vanguard]

Tom gave great answers.

From what I understand, the lists of customers were to be random.  I spoke with my new buddy the Hot Spring rep and asked him about it.  He said he wasn't exactly sure how Watkins created the list, but knows the list was random.  There was a check in place to make sure the list wasn't doctored.  It would have been very easy for Watkins to doctor the list, but he said it was totally random.

That is why this certification holds more value than others.  The survey was independent, the customer lists are random.  

I do suppose some less reputable brands could still try to doctor, but I have a feeling they'd get found out.  As everyone knows, Haven Spas will probably be "Certified" soon because they are above board and have "great" customer service.  

[stuart]

Still haven't been able to talk with David, I should talk with him today and I will let you know the outcome.

I'm very interested in ownership longivity although I'm not sure many of the companies rated where making spas the same for longer than 6 to 7 years.

[Tom]

Tom, do you know what percentage of this was contacted?

That information is not available to me at this time.   All I know is that we met or exceeded the minimum required number of respondents.  I would assume that the minimum was set to ensure a pre-determined level of statistical accuracy.

A certain percentage of the list would be incorrect email addresses.  Of those contacted, only a small fraction would reply (I believe that typical surveys consider 10% a good response rate).

[Tom]

Tom,
You are correct, I did mean "Manipulated" and there is no doubt in any reasonable person's mind that it is possible, as far as it being "unlikely" that's what we should be trying to determine!

You'll want to consider the motivation and the cost/benefit ratio.  Would the marketing value of the certification be worth the costs involved in manipulating the results?  I consider it highly unlikely.

In 2004, Arctic ran a six-month Customer Service follow-up program where we telephoned new Arctic Spa purchasers  (as many as we could reach) to ask about the quality of the sales and delivery processes using a structured interview format.  We did this in order to evaluate dealer performance and determine what improvements could be made to our sales and technical training.  This program was time-consuming and costly, obviously thorough, and of great benefit.

The spasearch survey gave us an ideal chance for a two-year follow-up to our original project.  The spasearch questions were similar to those used in our phone interviews.  It might answer questions about our progress and the results of changes to our dealer training -- all done by a third party at reasonable cost to us.   We leapt at the chance.    And since we wanted honest answers, we did our best to provide honest information.  

[Dr. Spa™ Ret.]

stuart, would I be correct in assuming that 1 particular manufacturer that made the list, has your feathers in an uproar (I'm just curious)?

[stuart]

stuart, would I be correct in assuming that 1 particular manufacturer that made the list, has your feathers in an uproar (I'm just curious)?

Well to be honest I had a pretty good idea of how they did the ratings up until the new rating came out. Know the service history on Master spas and on Jacuzzi I have strong doubts that they would either one get such a high rating.

Mind you, Jacuzzi probably has a much higher rating than Master and for the most part everything Master does for marketing is a bit suspect.

I also question whether some of the spas rated at the top have maintained any consistency in manufacturing long enough to be viable.

Although it really doesn’t matter my opinion of any of them but rather that I have to question them because of my experience and knowledge of industry history. I can’t be the only one that questions.

I also have a problem using anything as a sales tool that doesn’t have a extreme level of validity.

[East_TX_Spa]

[stuart]

Sooo let me ask this in a different light....

The magazine eludes to the fact that Master, Jacuzzi, Arctic, D1, Marquis, Sundance, HotSpring and Caldera are all equivalent and the rest of the brands are not quite as good.

Do most of you agree with this?