This is an abstract from the paper that was submitted and presented at ESOMAR Congress in Istanbul on September 24th, 2013 and was awarded best methodological paper. *(Copyright on the full paper is held by ESOMAR.)
In survey research, we rely on scales to help understand people’s responses – whether it be to understand how people perceive brands, their various interests, intentions or attitudes. Understanding the differences in opinions is central to market research; however, are these differences really significant? (See Scales are a cornerstone of market research graphic, top right.)
A respondent’s surroundings (sometimes altered by his or her mood), the culture or even the actual nature or content of the researcher’s question may cause differences.
Another factor that is often forgotten is that people have a tendency to respond to questions in different ways, irrespective of the content. This is particularly true when people respond using rating scales. Paulhus defined this phenomenon as, “A person’s tendency to systematically respond to questionnaire items in a certain way regardless of content.”
Extreme Response Styles
The ERS is characterized by consistently responding at either end of the scale. For a given group of statements, respondents will give a consistent, highly positive or highly negative response. The ARS is a subset of ERS, but is identified by a consistently positive score. The MRS is when respondents consistently give a score that is in the middle of the scale.
Our analysis approach derived an index for each (on an individual level) – essentially measuring the propensity to follow a particular response style within a set of scaled responses. Specifically, this paper investigates the presence of these response styles using a scaled response across three different modes: mobile, Web and social media.
- Are there differences in response style when scales are administered using different mobile modes?
- Are response styles different for traditional online surveys versus mobile phone surveys?
We wanted to understand how different scale executions affect response style, so the mobile study looked at standard grid questions on a mobile Web browser, sliders on a mobile Web browser, and sliders used in an app.
For the mobile element of the study, we interviewed 1,500 mobile respondents in the U.S. We used five-point Likert scales (agreement) for all statements and looked at a variety of different statement types (with reported reliability):
- Health environment sensitivity
- Personal health responsibility
- Attitude toward helping others (AHO)
- Material values scale (MVS 9)
- Attitude toward advertising in general (AAG)
- Online privacy concerns
- Lie acceptability scale
Additionally, within each group, we included statements that were reverse-scaled. This means that certain statements were worded in a positive fashion while others were worded with a negative slant.
If we then compare the mobile results – slider in app, Web grid and Web slider – we see there are differences between the different measures. However, they are not systematically attributable to the different ways in which we have measured the response.
Looking at the results for the ARS, we see a similar situation. There are directional differences, but none that can be attributed to the way in which the scale was related to the question. (See the Acquiescence Response Style – ARS – Index graphic on page 77.)
For the MRS, we see that the mobile slider tends to produce lower mid-point responding when compared to the other modes (statistically significant for three out of six scale types). (See the Midpoint Response Style – MRS – Index graphic on page 77.)
- Response styles remain stable across modes.
- There is no concrete evidence to suggest big differences in how response styles manifest themselves in mobile compared to Web.
- There is evidence to suggest that using a Web slider on a mobile device will cause a decrease in the tendency to select the mid-point.
Although inconclusive, in our view, the findings reinforce the need to consider respondent experience when designing a survey for mobile data collection.
Additionally, the study gave us an opportunity to look at the impact that different scale lengths would have on response style.
We designed an online study which was administered in 10 countries with sample sizes of n=2,000 in each country.
The survey was seven minutes in length independently of scale type. We used four, five, seven and ten--point scales and also looked at labeled vs. unlabeled five-point scales.
The purpose of this design was to provide rigor and confidence in our data.
Looking across the age bands, we saw some small differences in the response style indices. In general, ERS and MRS seemed to increase with age up until 55‑64, before declining again. This offers some support for the hypothesis that response style varies with age.
Looking across countries – our third hypothesis – the data showed some differences, which may not be surprising to some of us.
For example, respondents from Brazil and Mexico had the highest tendency to give extreme responses. Additionally, and consistent with findings from other academic studies, the Japanese are most likely to give a mid-point score.
We also wanted to look at the impact of the number of response options available, i.e. the length of the scale. ERS (the dark blue line) is fairly consistent across the different scale lengths, though it does dip for a five-point, labeled scale. (See the RQ3 graphic on page 78.)
The MRS is more interesting. It also dips for a five-point labeled scale (clearly the labels are having an impact on all response styles); however, the tendency to select a mid-point also drops with the more extensive scales.
Furthermore, we saw indications that a labeled, five-point scale can be used as a way to reduce the ERS: full labeling of the scale tells respondents what each point means.
We explored three aspects:
- Are there differences in the way men and women express sentiment when posting online? Are the different genders more/less prone to use extreme sentiment?
- Are we able to observe differences in the tendency to express a certain level of sentiment online based on geography?
- Does the age of the online contributor make a difference to the level of sentiment expressed?
Naturally, people do not talk in scales, but it is possible to convert the language we use to express ourselves into a ”strength of sentiment”. This is exactly what we did.
Using Conversition’s Evolisten software, we made a random sample of one year of publicly-available online comments from across the Internet and categorized them based on a five-point scale, from strongly negative through neutral to strongly positive.
The software capabilities only allowed us to work with English-speaking countries, so we included Australia, Canada, Ireland, the U.K. and the U.S. The set-up and quality control required a certain amount of reflection as it was necessary to define boundaries for the software and to help resolve any confusion.
Where possible, we included the author’s age and gender and combined it with their geographic location to make the data set more extensive.
While it was not possible to assign an age to all the authors, we do have some data. For those who provided this information, we looked at the decade in which they were born. The results show small differences; however, it seems that older and younger Internet users are least likely to post more positively. (See RQ7, Decade of Birth graphic on page 79.)
With country, our sample groups ranged from just over 9,000 in Ireland to 960,000 in the U.S. As with age, we found that the differences among country groups were not huge. People based in the U.K. and the U.S. post more comments that are considered to be extremely negative, compared to those in other countries. Looking at the overall positive vs. negative, it seems that Australians are most likely to post positively. (See RQ7, English-Speaking Countries graphic on page 79.)
The key implication is that, even in social media, cultural differences can have a tendency to cause people to rate things differently – cultural differences must therefore be taken into account in all country comparisons.
- Women tend to be more positive
- Those who are older and younger tend to be more negative in their online posts
- Australians are more positive
- For mobile devices, carefully consider the use of grids.
- We found the observable differences in response style between mobile and computer responses to be inconclusive.
- We also found that, compared to mobile grids, mobile sliders produce slightly higher levels of extreme response and lower levels of mid-point response.
- Caution is advised when comparing scale responses from different countries.
- There is little variation in response style by age or gender; however, response styles do vary by country.
- Use seven- or ten-point scales to minimize mid-point response style.
- Increasing the number of response options reduces the tendency to give consistent mid-point responses.
- Use labels on each point of your five-point scales to minimize extreme response style.
- Scale labeling impacts extreme responses and mid-point responses for five-point scales.
- When working with online sentiment, be certain that the differences are truly significant.
We found differences in terms of gender, age and nationality:
- Women are more positive online
- Australians are more positive online
- Paulhus, Delroy L. (1991). Measurement and Control of Response Bias. In Robinson, J.P., Sawyer, P.R. and Wrightman, L.S. (eds.), Measures of Personality and Social Psychological Attitudes, Vol. 1. San Diego CA: Academic Press.
- Baumgartner, Hans and Jan-Benedict E.M. Steenkamp (2001) “Research Styles in Marketing Research: A Cross-National Investigation”: Journal of Marketing Research.