Chiwah,

It's preferable to ask the question in a neutral way: "Consider your experience with this product. Did you find it:

Alternatively, you could ask them to rate the usability from 0-5 or 0-10 etc.

Otherwise, you do indeed risk biasing the responses. A really good book to read on the topic is 'Improving Survey Questions: Design & Evaluation' http://www.amazon.com/Improving -Survey -Questions -Evaluation -Research /dp /0803945833

Regards

Steve

2008/5/20 chiwah liu chiwah.liu at gmail.com:

Hello, If I want to do a survey to participant about, for example, usability of a product. Is it better to formulate questions in a positive way ? "Is this product is easy to use ?" or rather in a negative way "Did you experience difficulties using this product ?" Or maybe I should ask them both, because a user could experience difficulties in a product but still find it easy to use. What do you think about that ? chiwah Welcome to the Interaction Design Association (IxDA) ! To post to this list [trim]

Steve 'Doc' Baty B.Sc (Maths) , M.EC, MBA
Principal Consultant
Meld Consulting
M: +61 417 061 292
E: stevebaty at meld.com.au

UX Statistics: http://uxstats.blogspot.com

Member, UPA - www.upassoc.org
Member, IA Institute - www.iainstitute.org
Member, IxDA - www.ixda.org
Contributor - UXMatters - www.uxmatters.com

More importantly, the question you that should always be asked if you want to get any insights that will help you improve your product is WHY they think your product is (un) usable.

On Tue, May 20, 2008 at 11:21 AM, Steve Baty stevebaty at gmail.com wrote: Chiwah, It's preferable to ask the question in a neutral way: "Consider your experience with this product. Did you find it: - very difficult - difficult - average - easy - very easy" Alternatively, you could ask them to rate the usability from 0-5 or 0-10 etc. Otherwise, you do indeed risk biasing the responses. A really good book to read on the topic is 'Improving Survey Questions: Design & Evaluation' http://www.amazon.com/Improving -Survey -Questions -Evaluation -Research /dp /0803945833 Regards Steve 2008/5/20 chiwah liu chiwah.liu at gmail.com: Hello, If [trim]

2008/5/20 Steve Baty stevebaty at gmail.com:

Chiwah, It's preferable to ask the question in a neutral way Thank you a lot. That's exactly what I was looking for. Regards, Chiwah

Can asking the computer the right question, be a positive step in understanding semantic representation and processing?

http://www.dusteddesign.com/blog /semantic -representation -and -processing /

In the our blog article, I questioned whether we are asking the right question in search engines and if not how can computer scientists and interaction designers/web developers improve the user experience and address the issues on searching for the right information online using natural language.

2008/5/20 chiwah liu chiwah.liu at gmail.com
I am thinking about bipolar scale. For example to ask users to rate a bipolar scale between "attractive" vs "un attractive"

What do you think ?

I am also thinking; would bipolar scales diminish the probability of positivity bias? I mean we could ask for example if a product is, for example, professional. And ask participant to rate it from strongly agree to strongly disagree. Participants would have a tendency to agree.

But if I ask participants to choose between professional and amateur, they could not be agree of disagree, they just have to choose. And they have a better understanding of what is being measured.

Is my hypothesis right ?

Regards,

Chiwah

Just a quick note: Your hypothesis sounds great. With Likert scales (even if they're using words instead of numbers to rate the participants response) try to use an even number of options. Four is nice. With four options, your participant must to decide between the two poles, but still has room to express the degree to which they agree.

So instead of attractive/unattractive, you could have: attractive, somewhat attractive, somewhat unattractive, unattractive (This is just as you were saying in your hypothesis.)

I go to a lot of websites that don't necessarily sparkle, but they aren't blaze orange with a looping midi of a Christmas carol. So I know that I'm always grateful for a little bit of room in the middle.

One downside of being a Tech Comm graduate student, I have in fact had nightmares about survey reports. I got to breathe, eat, and sleep this stuff for a semester last year.

Hope this helps,
Christine

What you are prosing is called semantic differential. Think very carefully about the terms you use... it it not as simple as you might think. getting those terms right is the single hardest part of this technique and has the potential to radically skew your results. There is quite a bit of information out there now that you know what to call it.

The problem with most lickert scale surveys (not like it, 1 - like it, 10) is that the survey will be pre bias towards an aggregate score of seven. It is really hard to get around that when people administer the survey and people are taking it.

Mark

On Tue, May 20, 2008 at 9:12 AM, chiwah liu chiwah.liu at gmail.com wrote: 2008/5/20 chiwah liu chiwah.liu at gmail.com I am thinking about bipolar scale. For example to ask users to rate a bipolar scale between "attractive" vs "un attractive" What do you think ? I am also thinking; would bipolar scales diminish the probability of positivity bias? I mean we could ask for example if a product is, for example, professional. And ask participant to rate it from strongly agree to strongly disagree. Participants would have a tendency to agree. But if I ask participants to choose between professional and amateur, they could not be agree of disagree, [trim]

We actually used a 5 point scale in our rating system (Very Dissatisfied — Very Satisfied) specifically to give users a "neutral" option, and not force them to show a bias where none exists.

Some people really just dont care, or have factors evenly weighted enough that they cancel out.

On May 20, 2008, at 8:51 AM, Christine Neidley wrote: Just a quick note: Your hypothesis sounds great. With Likert scales (even if they're using words instead of numbers to rate the participants response) try to use an even number of options. Four is nice. With four options, your participant must to decide between the two poles, but still has room to express the degree to which they agree. So instead of attractive/unattractive, you could have: attractive, somewhat attractive, somewhat unattractive, unattractive (This is just as you were saying in your hypothesis.) I go to a lot of websites that don't necessarily sparkle, but they aren't blaze orange [trim]

--

Jeff Gimzek | Senior User Experience Designer

jeff at glassdoor.com | www.glassdoor.com

2008/5/21 Jeff Gimzek listserv at jdgimzek.com:

We actually used a 5 point scale in our rating system (Very Dissatisfied — Very Satisfied) specifically to give users a "neutral" option, and not force them to show a bias where none exists. Some people really just dont care, or have factors evenly weighted enough that they cancel out.

I don't know if I am right, but for me, the "neutral" option depends on the number of users :

What do you think?

From: "chiwah liu" chiwah.liu at gmail.com I don't know if I am right, but for me, the "neutral" option depends on the number of users :

You ought to allow users to have the opinions that they have - even if those opinions include 'don't know' or 'don't care' (or both) .

The answer options you offer should depend solely on the answers that your users want to give - not upon how many users there are.

If you don't know what answers your users want to give, then interview them to find out before running your survey. And by the way - you should do that anyway (i.e., interview some users first) if you want anything like good results from your survey.

There's a longer version of my views at:
http://www.usabilitynews.com/news/article1269.asp

Best Caroline Jarrett
caroline.jarrett at effortmark.co.uk

Caroline makes some very good points. Questionnaire design is complex and there are hundreds of articles debating the use of mid-points, the meaning of a mid-point, and other topics like how the order of questions influences answers. For many surveys, a Don't Know, Don't Care, or I Don't want to Answer (say to salary surveys or personal information) are all items that should be considered. If you are writing a questionnaire for a survey on a topic that you don't know well, doing some research beforehand to create the response categories is quite important so you don't have a lot of answers to your "Other" response category.

There are several excellent books that delve into the issues of bias and the many design issue that you need to consider. I would recommend:

Robson, C. (2002) . Real-world research (Second edition) . Malden, MA: Blackwell Publishing. This book describes many methods for gathering data including an excellent section on scale and questionnaire design. The book has a short, but excellent description, for example about how to develop Likert items.

Sudman, S., Bradburn, N. M., & Schwarz, N. (1996) . Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass. Thinking About Answers explores cognitive issues associated with survey methods. These issues include: context effects in surveys, order effects, event dating, counting and estimation, and autobiographical memory. The final chapter summarizes implications of cognitive research for survey design, administration, and interpretation.

Dillman, D. A. (2007) . Mail and internet surveys: The tailored design method 2007 update with new internet, visual, and mixed-mode guide. New York, NY: Wiley. This book is the third by Dillman who has written the most general book of survey guidelines.

Aiken, L. R. (2002) . Attitudes and Related Psychosocial Constructs: Theories, assessment, and research. Thousands Oaks, CA: Sage Publications. There are many books in social psychology that get into scale development. It is worth getting a book like Aiken or another book to understand the issues with Likert scaling, Semantic Differential scales, odd versus even scales, whether to label each scale point or only the end points.

Chauncey

No. I think the phrase 'force them to choose' shows exactly why this is a bad idea. You ought to allow users to have the opinions that they have - even if those opinions include 'don't know' or 'don't care' (or both) . The answer options you offer should depend solely on the answers that your users want to give - not upon how many users there are. If you don't know what answers your users want to give, then interview them to find out before running your survey. And by the way - you should do that anyway [trim]

I just want to emphasize strongly that you have to be very careful in constructing questions so that you're asking what you think you're asking. What does that mean? Well, my newest favorite question is "When you finished your transaction did you believe that the sales person successfully imparted his knowledge to you?" [no, really, they asked that. It was so bizarre I actually wrote it down.] My first (and continuing) reaction was that I had been more knowledgeable than the salesperson was when we started, and I now felt like he had succeeded in deleting knowledge from my brain (though I still knew more than he did) and I wasn't sure whether that was a (7) Completely successful or a (1) Completely unsuccessful.

I make it a point when I have to construct surveys to submit the questions to a couple of the crankiest people I know in terms of language and willfully attributing meaning literally when you were thinking more figuratively and vice versa. Any question that does not survive that process I rewrite until it passes. Yes, I user test my user testing. sigh.

kt --

Katie Albers
User Experience Consulting & Project Management katie at firstthought.com

2008/5/22 Chauncey Wilson chauncey.wilson at gmail.com:

Sudman, S., Bradburn, N. M., & Schwarz, N. (1996) . Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass. Thinking About Answers explores cognitive issues associated with survey methods. These issues include: context effects in surveys, order effects, event dating, counting and estimation, and autobiographical memory. The final chapter summarizes implications of cognitive research for survey design, administration, and interpretation. Dillman, D. A. (2007) . Mail and internet surveys: The tailored design method 2007 update with new internet, visual, and mixed-mode guide. New York, NY: Wiley. This book is the third by [trim]

Thank you for the books you recommended me. Is there a book that is particularly valuable? (Because I am not sure I could buy all these books.)

I already have some knowledge about survey and psychometrics so I prefer like a book that go into detail.

Best,

Chiwah

I would consider Dillman to be the best overall set of guidelines for survey and questionnaire design and implementation. Dillman includes the processing of writing cover letters, recruiting respondents, and other issues.

Chauncey

On Thu, May 22, 2008 at 6:47 AM, chiwah liu chiwah.liu at gmail.com wrote: 2008/5/22 Chauncey Wilson chauncey.wilson at gmail.com: Sudman, S., Bradburn, N. M., & Schwarz, N. (1996) . Thinking about answers: The application of cognitive processes to survey methodology. San Francisco, CA: Jossey-Bass. Thinking About Answers explores cognitive issues associated with survey methods. These issues include: context effects in surveys, order effects, event dating, counting and estimation, and autobiographical memory. The final chapter summarizes implications of cognitive research for survey design, administration, and interpretation. Dillman, D. A. (2007) . Mail and internet surveys: The tailored design method 2007 update with new internet, visual, and mixed-mode guide. New York, NY: Wiley. [trim]

2008/5/22 Chauncey Wilson chauncey.wilson at gmail.com:

I would consider Dillman to be the best overall set of guidelines for survey and questionnaire design and implementation. Dillman includes the processing of writing cover letters, recruiting respondents, and other issues. Chauncey Thank you, I think I am going to buy this book. Chiwah