TL;DR: UEQ is a 26-item questionnaire that measures reliably the Pragmatic Quality and Hedonic Quality of a product. Five other sub-dimensions have been proposed but literature is torn in their validity. UEQ provides a benchmark for comparing a system scores with normative data for a collection of other systems.

---

Defining UX has been, and to some extent still is, tricky. In their 2006 publication, Laugwitz, Schrepp, and Held introduced the User Experience Questionnaire (UEQ), originally developed in German (and later adapted into English). As a working definition of UX, the UEQ builds on Hassenzahl’s theory that users perceive a product through Pragmatic and Hedonic qualities. Pragmatic Quality emphasises how well a product supports users in achieving task-related goals with effectiveness and efficiency, whereas Hedonic Quality emphasises the product’s role in psychological well-being, stimulation, and identity expression. UEQ has shown promising psychometric properties, but more research is needed to evaluate the dimensions it is measuring.

Setting up the UEQ

The questions of the UEQ

The UEQ consists of 26 items, presented as pairs of opposite adjectives on a 7-point semantic differential scale ranging from 1 to 7, with 4 representing neutrality. Each item asks participants to rate the product somewhere between two extremes. Some items are positively worded (where higher scores indicate better UX), while others are negatively worded (where higher scores indicate worse UX).

The labels of the 26 semantic differential items are:

1. Annoying – Enjoyable
2. Not understandable – Understandable
3. Creative – Dull
4. Easy to learn – Difficult to learn
5. Valuable – Inferior
6. Boring – Exciting
7. Not interesting – Interesting
8. Unpredictable – Predictable
9. Fast – Slow
10. Inventive – Conventional
11. Obstructive – Supportive
12. Good – Bad
13. Complicated – Easy
14. Unlikable – Pleasing
15. Usual – Leading edge
16. Unpleasant – Pleasant
17. Secure – Not secure
18. Motivating – Demotivating
19. Meets expectations – Does not meet expectations
20. Inefficient – Efficient
21. Clear – Confusing
22. Impractical – Practical
23. Organized – Cluttered
24. Attractive – Unattractive
25. Friendly – Unfriendly
26. Conservative – Innovative

The translations and some adaptations can be found on the UEQ website.

Calculating the UEQ score

The rules

Positively worded items are: 

1, 2, 6, 7, 8, 11, 13, 14, 15, 16, 20, 22, and 26.

Negatively worded items are: 

3, 4, 5, 9, 10, 12, 17, 18, 19, 21, 23, 24, and 25.

The overall score of the UEQ (the product’s score)

There is no single “overall UEQ score”. Instead, the questionnaire is interpreted through three overarching qualities and six sub-dimensions. However, if we are really after an overall score we can use the score of the Attractiveness scale, which also should be approximately (Pragmatic + Hedonic) / 2.

We will assume in the calculation formulas below that all replies are in the range of 1 to 7.

The product’s Attractiveness score

To get the Attractiveness score for each participant:

1. Add items 1, 14, and 16 
2. From the number 24, subtract items 12, 24, and 25 
3. Add the two numbers from the previous steps, divide by 6, and subtract 4

Or in one formula:

\text{participant\_attractiveness\_score} = \frac{item_{1} + item_{14} + item_{16} + 24 – (item_{12} + item_{24} + item_{25})}{6} – 4

To get the overall Attractiveness score, calculate the mean of all participants’ scores.

The product’s Pragmatic Quality score

To get the Pragmatic Quality score for each participant:

Or in one formula:

1. Add items 2, 8, 11, 13, 20, and 22
2. From the number 48, subtract items 4, 9, 17, 19, 21, 23
3. Add the two numbers from the previous steps, divide by 12, and subtract 4

\text{participant\_pq\_score} = \frac{item_{2} + item_{8} + item_{11} + item_{13} + item_{20} + item_{22} + 48 – (item_{4} + item_{9} + item_{17} + item_{19} + item_{21} + item_{23})}{12} – 4

To get the overall Pragmatic Quality score, calculate the mean of all participants’ scores.

The product’s Hedonic Quality score

To get the Hedonic Quality score for each participant:

1. Add items 6, 7, 15, and 26
2. From the number 32, subtract items 3, 5, 10, and 18
3. Add the two numbers from the previous steps, divide by 8, and subtract 4

Or in one formula:

\text{participant\_hq\_score} = \frac{item_{6} + item_{7} + item_{15} + item_{26} + 32 – (item_{3} + item_{5} + item_{10} + item_{18})}{8} – 4

To get the overall Hedonic Quality score, calculate the mean of all participants’ scores.

The five sub-dimensions of the UEQ

The Pragmatic and Hedonic dimensions are further broken down into 5 sub-dimensions. Even though these dimensions were reported in the initial publications, recent research has shown mixed results in replicating them (e.g. research by Schankin et. al). The five sub-dimensions were introduced to help pin-point which aspects of the product need to be improved, for the time being, we suggest interpreting these dimensions with caution.

Pragmatic Quality dimensions:

Perspicuity (ease of learning and clarity):

To get the Perspicuity score for each participant:

1. Add items 2, and 13
2. From the number 16, subtract items 4, and 21
3. Add the two numbers from the previous steps, divide by 4, and subtract 4

Or in one formula:

\text{participant\_perspicuity\_score} = \frac{item_{2} + item_{13} + 16 – (item_{4} + item_{12})}{4} – 4

To get the overall Perspicuity score, calculate the mean of all participants’ scores.

Efficiency (task completion and productivity): 

To get the Efficiency score for each participant:

1. Add items 20, and 22
2. From the number 16, subtract items 9, and 23
3. Add the two numbers from the previous steps, divide by 4, and subtract 4

Or in one formula:

\text{participant\_efficiency\_score} = \frac{item_{20} + item_{22} + 16 – (item_{9} + item_{23})}{4} – 4

To get the overall Efficiency score, calculate the mean of all participants’ scores.

Dependability (predictability, control, and trust):

To get the Dependability score for each participant:

1. Add items 8, and 11
2. From the number 16, subtract items 17, and 19
3. Add the two numbers from the previous steps, divide by 4, and subtract 4

Or in one formula:

\text{participant\_dependability\_score} = \frac{item_{8} + item_{22} + 16 – (item_{17} + item_{19})}{4} – 4

To get the overall Dependability score, calculate the mean of all participants’ scores.

Hedonic Quality dimensions:

Stimulation (excitement, motivation, and engagement):

To get the Stimulation score for each participant:

1. Add items 6, and 7
2. From the number 16, subtract items 5, and 18
3. Add the two numbers from the previous steps, divide by 4, and subtract 4

Or in one formula:

\text{participant\_stimulation\_score} = \frac{item_{6} + item_{7} + 16 – (item_{5} + item_{18})}{4} – 4

To get the overall Stimulation score, calculate the mean of all participants’ scores.

Novelty (innovation and creativity):

To get the Novelty score for each participant:

1. Add items 15, and 26
2. From the number 16, subtract items 3, and 10
3. Add the two numbers from the previous steps, divide by 4, and subtract 4

Or in one formula:

\text{participant\_novelty\_score} = \frac{item_{15} + item_{26} + 16 – (item_{3} + item_{10})}{4} – 4

To get the overall Novelty score, calculate the mean of all participants’ scores.

Understanding the UEQ’s score

Benchmarks

The UEQ provides a benchmark dataset based on 21175 evaluations of 468 products across industries. These benchmarks can be used to compare the results from the product we are evaluating with the products in the dataset and help put a “label” to our numbers. The authors of the UEQ have provided a percentile rank (similar to SUPR-Q) that can be interpreted per subscale as shown on the tables below:

Benchmark for Attractiveness

Benchmark for Perspicuity

Benchmark for Efficiency

Benchmark for Dependability

Benchmark for Stimulation

Benchmark for Novelty

For example a product that has an attractiveness score of 1.21, is perceived as having better attractiveness than 50% of the products in the dataset, and worse than 25%. We can label it then as “Above Average” regarding its attractiveness.

The Excel file provided by UEQ can be used both as a shortcut to the calculation and to the benchmarks.

Factors that affect UEQ

TL;DR: SUPR-Q is a measurement of a website’s UX that is broken down into 4 aspects: Usability, Trust, Loyalty, and Appearance. It uses 8 questions and has initially shown to be correlated with the SUS and that it has a good factor structure.

---

Talking about web design, user experience (UX) is a crucial factor that determines a website’s success. Usability can be measured in many ways, but what about UX overall? To answer this question about measuring UX and its aspects effectively, the Standardized User Experience Percentile Rank Questionnaire (SUPR-Q) was developed. Introduced in 2015 by Jeff Sauro, the SUPR-Q comprises eight questions that measure four key aspects of UX: usability, trust, appearance, and loyalty. It demonstrates high reliability (α = 0.86) and high concurrent validity with the System Usability Scale (r = 0.87).

Setting up the Standardized User Experience Percentile Rank Questionnaire

The questions of SUPR-Q

The eight questions of the questionnaire are:

1. The website is easy to use.
2. It is easy to navigate within the website.
3. I feel comfortable purchasing from the website (or The information on the website is credible.)
4. I feel confident conducting business on the website (or The information on the website is trustworthy).
5. How likely are you to recommend this website to a friend or colleague?**
6. I will likely return to the website in the future.
7. I find the website to be attractive.
8. The website has a clean and simple presentation.

The Responses of SUPR-Q

All questions are scored on a 5-point Likert Scale (1= strongly disagree and 5 = strongly agree), except for item 5 which is scored on an 11-point Scale (0 = not at all likely and 10 = extremely likely) to align with how Likelihood to Recommend is calculated.

Calculating the SURP-Q Score

The Holistic Score

To calculate the SUPR-Q score of each participant:

1. Sum all questions except question 5
2. Multiply question 5 by 0.5 and add the result to the total sum
3. Divide by 8

To derive the overall SUPR-Q score of the website, simply take the average of all individual SURP-Q scores.

Or, in one formula: overall SUPR-Q = AVERAGE(SUM( (Q1 + Q2 + Q3 + Q4 + 0.5*Q5 + Q6 + Q7 + Q8) / 8))

You can easily calculate all the SUPR-Q scores in the (free) IX Calculator.

The Aspect Specific Scores

The SURP-Q also facilitates the evaluation of scores for each of its four measured aspects:

Usability Score:

1. For each participant, sum questions 1 and 2 then divide by 2.
2. For the overall usability score take the average of all individual scores.

Trust Score:

1. For each participant, sum questions 3 and 4 then divide by 2.
2. For the overall trust score take the average of all individual scores.

Loyalty Score:

1. For each participant, multiply question 5 by 0.5 and sum the result with question 6, then divide by 2.
2. For the overall loyalty score take the average of all individual scores.

Appearance Score:

1. For each participant, sum questions 7 and 8 then divide by 2.
2. For the overall appearance score take the average of all individual scores.

Understanding the scores of SURP-Q

Probabilistic approach

Jeff Sauro has shared the mean and standard deviation of the SUPR-Q and its aspects scores. Since data seems to generally follow the normal distribution, we can compare the score of the evaluated website with the infographic below to understand how the evaluated system compares to the 100 websites tested.

For example, a website with an overall SURP-Q score of 4.25 positions it in the top 25% for best UX. On the other side, a website with a 3.1 overall SURP-Q score will have a worse UX than 99% of the websites tested.

Score	Mean	Standard Deviation
SUPR-Q	3.93	0.29
Usability	4.06	0.29
Trust	3.8	0.52
Loyalty	3.91	0.46
Appearance	3.88	0.25

Rank Approach

Another way to understand the SURP-Q score of a website is to express it as a rank-score. This is more of a technicality for how the SUPR-Q should be calculated (as hinted by the name of the questionnaire). We can express the overall SUPR-Q score as the rank of the normally distributed probability function. To do that:

1. Find how many standard deviations away is the score from the mean
2. Calculate the standard normal cumulative distribution function for that number

For example, a site with an overall SUPR-Q score of 4.3 will have a Rank score of 90. That would mean that it has a higher User Experience than 90% of the websites tested.

You can also use this Google Sheet to perform the above calculation.

Limitations and Considerations

The original SUPR-Q study was conducted with 100 websites and 4,000 users. For more up-to-date and comprehensive results, consider using the author’s updated dataset (available for a fee). Additionally, further studies are needed to fully validate the SUPR-Q.