Category: Quantitative Usability

We have compiled a collection of informative articles covering a range of topics related to usability testing, user-centered design, and more, with a focus on quantitative data and analysis.

Each article is centering around a concept of usability, describing it from a quantitative data and analysis approach. A list of resources for anyone involved in creating digital products and services with valuable insights and practical tips to enhance the user experience.

  • Self-Assessment Manikin (SAM): A UX Practitioner’s Guide

    TL;DR: The Self-Assessment Manikin (SAM) is a valid and reliable picture-based measurement of perceived emotion. Emotion is measured in three dimensions (Pleasure, Arousal, Dominance), but the Dominance dimension is not usually needed.


    But how do we measure emotion? That’s the question the Self-Assessment Manikin (SAM) is answering. More specifically, SAM measures perceived emotion. Emotion measurement dates considerably before Human-Computer Interaction’s existence; naturally, SAM has its roots in psychology and emotion measurement in a psychological context. It was introduced in 1980 by Peter J. Lang. As a matter of fact, a large dataset of studies utilising SAM to measure perceived emotion has used pictures, audio, and nouns (words). However, in measuring aspects of Interaction Experience (IX), we find it useful to also measure perceived emotion (for example).

    Setting up the SAM

    The questions of SAM

    The Self-Assessment Manikin is different from the questionnaires we normally use in IX measurement in that it consists of pictures. Five pictures and a blank space in between them that represents the in-the-middle values are organised in a 3-row format; each line is one dimension: Pleasure, Arousal and Dominance. There is also a variant (the original version) that includes only 5 pictures per line, without the in-the-middle boxes.

    • The first line measures pleasure (valence): from (left) feeling very happy, to (right) feeling very unhappy. (This is the initial ordering of items, although some publications use the reverse order)
    • The second line measures arousal (how strong the feeling is): from (left) a very tense feeling to (right) a very calm feeling. (Once again, this is the initial ordering of items, although some publications use the reverse order)
    • The third line measures dominance: from (left) submissiveness to (right) dominance

    The 5 items SAM. No in-between the manikins replies.
    5-item SAM. Picture adapted from Betella and Verschure.
    The 9 items SAM. With in-between the manikins replies.
    9-item SAM. Picture adapted from Betella and Verschure.

    The responses of SAM

    Participants can rate their affective state (emotion) by placing an X in one box per line.

    In analysis, the leftmost manikin is coded with the number -8 and moving to the box on its right adds 2. So the equivalent numerical values for each dimension (with in-the-middle values) are: -8, -6, -4, -2, 0, 2, 4, 6, and 8.

    • For the first line, pleasure, -8 represents a very happy feeling, 0 represents feeling neither happy nor unhappy, and 8 represents a very unhappy feeling
    • For the second line, arousal, -8 represents a very tense feeling, 0 represents a feeling neither tense nor calm, and 8 represents a very calm feeling
    • For the third line, dominance, -8 represents a very submissive feeling, 0 represents neither a dominant nor a submissive feeling, and 8 represents a very dominant feeling

    The participants don’t see the numbers; we only use them to code the responses for analysis. 

    The same image as above, SAM 9-items, but superimposed with numbers to help visually explain the above paragraph.

    Calculating the SAM score

    Given that there is no single SAM score, we interpret the three replies of each participant as a triple that describes the perceived affective state in a three-dimensional plane. 

    So, calculating the score for the emotional response that an interaction with a system, product, or service elicits is a matter of averaging all replies for each of the dimensions, not forgetting to also state standard errors. We can also use statistical analysis tools to automate the process (for example, IX Calculator offers both the original three-dimensional, and the current two-dimensional version). Then we interpret the results using the equivalence between the numbers and the affective state for each dimension (above).

    For example, if we have the replies of 50 participants after an interaction with a product, and we calculate the averages as: average(Pleasure) = -6.4, average(Arousal) =  -4.5 and average(Dominance) = 1.0, we can interpret it as that the product, on average, was very pleasurable to use, arousing, and that participants felt dominant during their use.

    The Dominance dimension is not needed

    Since 1980, there has been substantial development in how emotion, or more accurately, core affect, can be represented. Although SAM was based on Russell’s and Mehrabian’s theory of Pleasure, Arousal, Dominance (PAD), later Russell clarified that core affect can be represented fully by just two dimensions (Pleasure and Arousal), which is also the case in frontier emotion construction theories.

    So, the SAM can be administered and analysed by just the first two lines measuring Pleasure and Arousal, dropping completely the Dominance dimension.

    Reliability, validity, and properties of SAM

    In the last almost half a century, SAM has shown good psychometric properties. Test-retest reliability was very high (r = .96–.98). At the same time, Pleasure and Arousal dimensions showed high correlations with the Semantic Differential Scale and demonstrated high agreement between participants. 

    The distribution of emotions in a two-dimensional space

    Even though all numerical combinations of Pleasure–Arousal are theoretically possible, in practice, measured emotion appears as a U-curve when plotted on a Pleasure-Arousal space and can be modelled with a quadratic function. That highlights the difficulty of the appearance of emotions that are felt as either very happy and very calm, or very unhappy and very calm. At the same time, emotions that are considered very calm tend to be neutral in terms of pleasure.

    A plot on a 2-axis space (Valence-Arousal), the datapoints cluster in a U-shape curve
    Image courtesy of Philipp Kanske & Sonja A. Kotz.
  • The User Experience Questionnaire (UEQ): A Practical Semantic Differential

    TL;DR: UEQ is a 26-item questionnaire that measures reliably the Pragmatic Quality and Hedonic Quality of a product. Five other sub-dimensions have been proposed but literature is torn in their validity. UEQ provides a benchmark for comparing a system scores with normative data for a collection of other systems.


    Defining UX has been, and to some extent still is, tricky. In their 2006 publication, Laugwitz, Schrepp, and Held introduced the User Experience Questionnaire (UEQ), originally developed in German (and later adapted into English). As a working definition of UX, the UEQ builds on Hassenzahl’s theory that users perceive a product through Pragmatic and Hedonic qualities. Pragmatic Quality emphasises how well a product supports users in achieving task-related goals with effectiveness and efficiency, whereas Hedonic Quality emphasises the product’s role in psychological well-being, stimulation, and identity expression. UEQ has shown promising psychometric properties, but more research is needed to evaluate the dimensions it is measuring.

    Setting up the UEQ

    The questions of the UEQ

    The UEQ consists of 26 items, presented as pairs of opposite adjectives on a 7-point semantic differential scale ranging from 1 to 7, with 4 representing neutrality. Each item asks participants to rate the product somewhere between two extremes. Some items are positively worded (where higher scores indicate better UX), while others are negatively worded (where higher scores indicate worse UX).

    The labels of the 26 semantic differential items are:

    1. Annoying – Enjoyable
    2. Not understandable – Understandable
    3. Creative – Dull
    4. Easy to learn – Difficult to learn
    5. Valuable – Inferior
    6. Boring – Exciting
    7. Not interesting – Interesting
    8. Unpredictable – Predictable
    9. Fast – Slow
    10. Inventive – Conventional
    11. Obstructive – Supportive
    12. Good – Bad
    13. Complicated – Easy
    14. Unlikable – Pleasing
    15. Usual – Leading edge
    16. Unpleasant – Pleasant
    17. Secure – Not secure
    18. Motivating – Demotivating
    19. Meets expectations – Does not meet expectations
    20. Inefficient – Efficient
    21. Clear – Confusing
    22. Impractical – Practical
    23. Organized – Cluttered
    24. Attractive – Unattractive
    25. Friendly – Unfriendly
    26. Conservative – Innovative

    The translations and some adaptations can be found on the UEQ website.

    A screenshot of the google forms (link in the caption) containing the UEQ.
    The UEQ presented in Google Forms.

    Calculating the UEQ score

    The rules

    Positively worded items are: 

    1, 2, 6, 7, 8, 11, 13, 14, 15, 16, 20, 22, and 26.

    Negatively worded items are: 

    3, 4, 5, 9, 10, 12, 17, 18, 19, 21, 23, 24, and 25.

    The overall score of the UEQ (the product’s score)

    There is no single “overall UEQ score”. Instead, the questionnaire is interpreted through three overarching qualities and six sub-dimensions. However, if we are really after an overall score we can use the score of the Attractiveness scale, which also should be approximately (Pragmatic + Hedonic) / 2.

    We will assume in the calculation formulas below that all replies are in the range of 1 to 7.

    The product’s Attractiveness score

    To get the Attractiveness score for each participant:

    1. Add items 1, 14, and 16 
    2. From the number 24, subtract items 12, 24, and 25 
    3. Add the two numbers from the previous steps, divide by 6, and subtract 4

    Or in one formula:

    \text{participant\_attractiveness\_score} = \frac{item_{1} + item_{14} + item_{16} + 24 – (item_{12} + item_{24} + item_{25})}{6} – 4

    To get the overall Attractiveness score, calculate the mean of all participants’ scores.

    The product’s Pragmatic Quality score

    To get the Pragmatic Quality score for each participant:

    Or in one formula:

    1. Add items 2, 8, 11, 13, 20, and 22
    2. From the number 48, subtract items 4, 9, 17, 19, 21, 23
    3. Add the two numbers from the previous steps, divide by 12, and subtract 4
    \text{participant\_pq\_score} = \frac{item_{2} + item_{8} + item_{11} + item_{13} + item_{20} + item_{22} + 48 – (item_{4} + item_{9} + item_{17} + item_{19} + item_{21} + item_{23})}{12} – 4

    To get the overall Pragmatic Quality score, calculate the mean of all participants’ scores.

    The product’s Hedonic Quality score

    To get the Hedonic Quality score for each participant:

    1. Add items 6, 7, 15, and 26
    2. From the number 32, subtract items 3, 5, 10, and 18
    3. Add the two numbers from the previous steps, divide by 8, and subtract 4

    Or in one formula:

    \text{participant\_hq\_score} = \frac{item_{6} + item_{7} + item_{15} + item_{26} + 32 – (item_{3} + item_{5} + item_{10} + item_{18})}{8} – 4

    To get the overall Hedonic Quality score, calculate the mean of all participants’ scores.

    The five sub-dimensions of the UEQ

    The Pragmatic and Hedonic dimensions are further broken down into 5 sub-dimensions. Even though these dimensions were reported in the initial publications, recent research has shown mixed results in replicating them (e.g. research by Schankin et. al). The five sub-dimensions were introduced to help pin-point which aspects of the product need to be improved, for the time being, we suggest interpreting these dimensions with caution.

    Pragmatic Quality dimensions:

    Perspicuity (ease of learning and clarity):

    To get the Perspicuity score for each participant:

    1. Add items 2, and 13
    2. From the number 16, subtract items 4, and 21
    3. Add the two numbers from the previous steps, divide by 4, and subtract 4

    Or in one formula:

    \text{participant\_perspicuity\_score} = \frac{item_{2} + item_{13} + 16 – (item_{4} + item_{12})}{4} – 4

    To get the overall Perspicuity score, calculate the mean of all participants’ scores.

    Efficiency (task completion and productivity): 

    To get the Efficiency score for each participant:

    1. Add items 20, and 22
    2. From the number 16, subtract items 9, and 23
    3. Add the two numbers from the previous steps, divide by 4, and subtract 4

    Or in one formula:

    \text{participant\_efficiency\_score} = \frac{item_{20} + item_{22} + 16 – (item_{9} + item_{23})}{4} – 4

    To get the overall Efficiency score, calculate the mean of all participants’ scores.

    Dependability (predictability, control, and trust):

    To get the Dependability score for each participant:

    1. Add items 8, and 11
    2. From the number 16, subtract items 17, and 19
    3. Add the two numbers from the previous steps, divide by 4, and subtract 4

    Or in one formula:

    \text{participant\_dependability\_score} = \frac{item_{8} + item_{22} + 16 – (item_{17} + item_{19})}{4} – 4

    To get the overall Dependability score, calculate the mean of all participants’ scores.

    Hedonic Quality dimensions:

    Stimulation (excitement, motivation, and engagement):

    To get the Stimulation score for each participant:

    1. Add items 6, and 7
    2. From the number 16, subtract items 5, and 18
    3. Add the two numbers from the previous steps, divide by 4, and subtract 4

    Or in one formula:

    \text{participant\_stimulation\_score} = \frac{item_{6} + item_{7} + 16 – (item_{5} + item_{18})}{4} – 4

    To get the overall Stimulation score, calculate the mean of all participants’ scores.

    Novelty (innovation and creativity):

    To get the Novelty score for each participant:

    1. Add items 15, and 26
    2. From the number 16, subtract items 3, and 10
    3. Add the two numbers from the previous steps, divide by 4, and subtract 4

    Or in one formula:

    \text{participant\_novelty\_score} = \frac{item_{15} + item_{26} + 16 – (item_{3} + item_{10})}{4} – 4

    To get the overall Novelty score, calculate the mean of all participants’ scores.

    Understanding the UEQ’s score

    Benchmarks

    The UEQ provides a benchmark dataset based on 21175 evaluations of 468 products across industries. These benchmarks can be used to compare the results from the product we are evaluating with the products in the dataset and help put a “label” to our numbers. The authors of the UEQ have provided a percentile rank (similar to SUPR-Q) that can be interpreted per subscale as shown on the tables below:

    Benchmark for Attractiveness

    PercentileMeanLabel
    < 25%–Bad
    25%0.69Bellow Average
    50%1.18Above Average
    75%1.58Good
    90%1.84Excellent

    Benchmark for Perspicuity

    PercentileMeanLabel
    < 25%–Bad
    25%0.72Bellow Average
    50%1.2Above Average
    75%1.73Good
    90%2Excellent

    Benchmark for Efficiency

    PercentileMeanLabel
    < 25%–Bad
    25%0.6Bellow Average
    50%1.05Above Average
    75%1.5Good
    90%1.88Excellent

    Benchmark for Dependability

    PercentileMeanLabel
    < 25%–Bad
    25%0.78Bellow Average
    50%1.14Above Average
    75%1.48Good
    90%1.7Excellent

    Benchmark for Stimulation

    PercentileMeanLabel
    < 25%–Bad
    25%0.5Bellow Average
    50%1Above Average
    75%1.35Good
    90%1.7Excellent

    Benchmark for Novelty

    PercentileMeanLabel
    < 25%–Bad
    25%0.16Bellow Average
    50%0.7Above Average
    75%1.12Good
    90%1.6Excellent

    For example a product that has an attractiveness score of 1.21, is perceived as having better attractiveness than 50% of the products in the dataset, and worse than 25%. We can label it then as “Above Average” regarding its attractiveness.

    The Excel file provided by UEQ can be used both as a shortcut to the calculation and to the benchmarks.

    Factors that affect UEQ

    FactorInfluenceStudies
    GenderPossibly no significant influenceInitial data from Schrepp et al. and initial data for UEQ-S Kollmorgen et al.
    Usage frequencyPossibly some influence on Attractiveness, PQ and HQInitial data for UEQ-S from Kollmorgen et al.
    Self-reported knowledge of the productPossibly some influence on Attractiveness, PQ and HQInitial data for UEQ-S from Kollmorgen et al.
    Duration of usePossibly some influence on Attractiveness, PQ and HQInitial data for UEQ-S from Kollmorgen et al.

  • The SUPR-Q: Assessing Website User Experience and Its Aspects

    TL;DR: SUPR-Q is a measurement of a website’s UX that is broken down into 4 aspects: Usability, Trust, Loyalty, and Appearance. It uses 8 questions and has initially shown to be correlated with the SUS and that it has a good factor structure.


    Talking about web design, user experience (UX) is a crucial factor that determines a website’s success. Usability can be measured in many ways, but what about UX overall? To answer this question about measuring UX and its aspects effectively, the Standardized User Experience Percentile Rank Questionnaire (SUPR-Q) was developed. Introduced in 2015 by Jeff Sauro, the SUPR-Q comprises eight questions that measure four key aspects of UX: usability, trust, appearance, and loyalty. It demonstrates high reliability (α = 0.86) and high concurrent validity with the System Usability Scale (r = 0.87).

    Setting up the Standardized User Experience Percentile Rank Questionnaire

    The questions of SUPR-Q

    The eight questions of the questionnaire are:

    1. The website is easy to use.
    2. It is easy to navigate within the website.
    3. I feel comfortable purchasing from the website (or The information on the website is credible.)
    4. I feel confident conducting business on the website (or The information on the website is trustworthy).
    5. How likely are you to recommend this website to a friend or colleague?**
    6. I will likely return to the website in the future.
    7. I find the website to be attractive.
    8. The website has a clean and simple presentation.

    The Responses of SUPR-Q

    All questions are scored on a 5-point Likert Scale (1= strongly disagree and 5 = strongly agree), except for item 5 which is scored on an 11-point Scale (0 = not at all likely and 10 = extremely likely) to align with how Likelihood to Recommend is calculated.

    Calculating the SURP-Q Score

    The Holistic Score

    To calculate the SUPR-Q score of each participant:

    1. Sum all questions except question 5
    2. Multiply question 5 by 0.5 and add the result to the total sum
    3. Divide by 8

    To derive the overall SUPR-Q score of the website, simply take the average of all individual SURP-Q scores.

    Or, in one formula: overall SUPR-Q = AVERAGE(SUM( (Q1 + Q2 + Q3 + Q4 + 0.5*Q5 + Q6 + Q7 + Q8) / 8))

    You can easily calculate all the SUPR-Q scores in the (free) IX Calculator.

    The Aspect Specific Scores

    The SURP-Q also facilitates the evaluation of scores for each of its four measured aspects:

    Usability Score:

    1. For each participant, sum questions 1 and 2 then divide by 2.
    2. For the overall usability score take the average of all individual scores.

    Trust Score:

    1. For each participant, sum questions 3 and 4 then divide by 2.
    2. For the overall trust score take the average of all individual scores.

    Loyalty Score:

    1. For each participant, multiply question 5 by 0.5 and sum the result with question 6, then divide by 2.
    2. For the overall loyalty score take the average of all individual scores.

    Appearance Score:

    1. For each participant, sum questions 7 and 8 then divide by 2.
    2. For the overall appearance score take the average of all individual scores.

    Understanding the scores of SURP-Q

    Probabilistic approach

    Jeff Sauro has shared the mean and standard deviation of the SUPR-Q and its aspects scores. Since data seems to generally follow the normal distribution, we can compare the score of the evaluated website with the infographic below to understand how the evaluated system compares to the 100 websites tested.

    Infographic showing the data available on the table below
    SUPR-QUsabilityTrustLoyaltyAppearanceBetter than
    3.033.162.192.493.110.1 %
    3.263.392.592.843.31 %
    3.453.582.943.153.475 %
    3.563.693.133.323.5610 %
    3.653.783.293.463.6417%
    3.733.863.453.603.7125 %
    3.934.063.83.913.8850 %
    4.134.264.154.224.0575 %
    4.214.354.324.374.1384 %
    4.34.434.474.504.290 %
    4.414.544.664.674.2995 %
    4.64.73––4.4699 %
    4.834.96554.6599.9%
    SUPR-Q scores and the percentage of websites they are better.

    For example, a website with an overall SURP-Q score of 4.25 positions it in the top 25% for best UX. On the other side, a website with a 3.1 overall SURP-Q score will have a worse UX than 99% of the websites tested.

    ScoreMeanStandard Deviation
    SUPR-Q3.930.29
    Usability4.060.29
    Trust3.80.52
    Loyalty3.910.46
    Appearance3.880.25
    The scores of SUPR-Q, the recorded mean and SD based on the study of 4000 users and 100 websites by Jeff Sauro

    Rank Approach

    Another way to understand the SURP-Q score of a website is to express it as a rank-score. This is more of a technicality for how the SUPR-Q should be calculated (as hinted by the name of the questionnaire). We can express the overall SUPR-Q score as the rank of the normally distributed probability function. To do that:

    1. Find how many standard deviations away is the score from the mean
    2. Calculate the standard normal cumulative distribution function for that number

    For example, a site with an overall SUPR-Q score of 4.3 will have a Rank score of 90. That would mean that it has a higher User Experience than 90% of the websites tested.

    You can also use this Google Sheet to perform the above calculation.

    Limitations and Considerations

    The original SUPR-Q study was conducted with 100 websites and 4,000 users. For more up-to-date and comprehensive results, consider using the author’s updated dataset (available for a fee). Additionally, further studies are needed to fully validate the SUPR-Q.

  • UMUX-Lite: A detailed insight on the UMUX’s kid

    TL;DR: The UMUX-Lite is a short 2-items questionnaire that incorporates the TAM. It has good psychometric properties and can be correlated with the System Usability Scale.


    The UMUX-Lite is really a direct descendant of UMUX, it assumes that user acceptance of a new technology is based on its perceived usefulness (Item 2 of the UMUX) and perceived ease of use (Item 3 of the UMUX). Similar to the Technology Acceptance Model.

    The Questions: Assessing Usability for User Experience Lite

    In order to gauge user acceptance effectively, the UMUX-Lite questionnaire utilises the two positively worded questions of the UMUX. Each question is rated on a 7-point Likert scale, ranging from strongly disagree to strongly agree. The questions used in UMUX-Lite are as follows:

    • Usefulness: This system’s capabilities meet my requirements
    • Ease of Use: This system is easy to use

    The subsequent calculation of the UMUX-Lite score allows for a quantitative evaluation of user experience.

    Calculating the UMUX-Lite Score

    The UMUX-Lite score serves as a quantitative measure to assess the usability of a system based on user feedback. To calculate the UMUX-Lite score for each user, the following steps are followed:

    • Add the scores of the two questions provided by the user.
    • Subtract 2 from the total obtained in step 1.
    • Multiply the result by 100/12 or 8.33 to normalise the score (bring it on a scale from 0 to 100).

    To obtain the UMUX-Lite score for the entire system, the average of all user UMUX scores is calculated. Alternatively, you can use this free tool to automatically calculate the UMUX score.

    This system score offers a comprehensive overview of the overall usability from the collective user perspective.

    Correlation to SUS: Linking UMUX-Lite and SUS

    The UMUX-Lite metric has demonstrated a notable correlation with the widely used System Usability Scale (SUS). Multiple studies have consistently reported correlation coefficients between UMUX-Lite and SUS ranging from 0.74 to 0.83, indicating a strong association between the two metrics.  This is quite useful since the SUS, being one of the most thoroughly researched usability questionnaires, comes with a wide variety of benchmarks.

    The initial regression formula

    Initially, a regression formula was introduced as it seemed to exhibit a stronger correlation with the SUS. The regression formula is calculated as: 

    UMUX-Liter = 0.65 * (Item1 – Item2 – 2) * (100 / 12) + 22.9. 

    However, subsequent evidence emerged contradicting the initial claim that the regression formula correlates better with the SUS compared to the original UMUX-Lite calculation.

    Multiple studies (for example, Lah et al. and Schrepp et al.) have argued against the predictive quality of this formula; thus, the initial regression formula shouldn’t be preferred over the original UMUX-Lite calculation.

    Which formula to use to correlate UMUX-Lite with the SUS

    Correlating to SUS using just the Ease question

    In this approach, the SUS score is obtained through the equation: (Item2 – 1) * (100 / 6). The idea behind just using the ease-of-use item comes from the initial research of UMUX, where this question was a rephrasing of the SUS’s third question.

    Correlating to SUS using the regression equation of the Ease question

    The other alternative is to use a regression equation specifically derived from the Ease question. The UMUX-Lite score can be obtained using the equation: 19.2 * ((Item2 – 1) * (4 / 6) + 1) – 2.279. This approach has the highest predictive quality.

    Evaluating the Approaches

    Both the simple correlation equation and the regression equation exhibit improved results in correlating with the SUS. This is supported by MeasuringU’s unpublished studies, encompassing a diverse range of systems and involving nearly 500 combined participants, and later research by Schrepp et al. Nonetheless, just using the overall UMUX-Lite score as a proxy for the SUS is a straightforward alternative that gives good results.

    Alternatives to the first question of UMUX-Lite

    In a separate investigation conducted by Lewis and Sauro, alternative formulations were explored to simplify the first question of UMUX-Lite, “This system’s capabilities meet my requirements.” The following alternative statements can be considered as viable alternatives to the original question:

    • “{Product}’s functionality meets my needs.”
    • “{Product}’s features meet my needs.”
    • “{Product}’s functions meet my needs.”
    • “{Product} does what I need it to do.”

    These alternative statements aim to provide flexibility for picking the question to suit specific research contexts and user populations.

    Presenting the UMUX-Lite: Communicating the Metric Effectively

    The UMUX-Lite metric can be effectively presented using two distinct approaches: a graphical representation and a grading system.

    Graphical Way: Visualising the UMUX-Lite

    Not to state the obvious, as with all one-score questionnaires, the system’s overall UMIX-Lite score can be presented using a graph. However, UMUX-Lite offers a unique visualisation opportunity compared to other usability questionnaires. Since it is built to follow the TAM, each question can be presented in its axis to mark the overall usability of the system in the two-dimensional world —quite handy for comparison of different systems.

    An orthogonal axis system with the x axis representing usability and the y axis representing ease of use. Two systems are presented as dots in the axis system. System 1 with a UMUX-Lite score of 33 and Q1=Q2=3. System 2 with a UMUX-Lite score of 58 and Q1 = 4, Q2 = 5

    While nicely presented, there is no evidence that each question is reliably capturing one aspect of usability, so use with caution.

    Grading System: Categorising the UMUX-Lite Scores

    Since we can approximate an application’s SUS score from the UMUX-Lite, we can use the two grading scales for SUS scores, and all other known benchmarks of SUS scores (e.g., the average SUS score of approximately 500 products in the wild is 68). 

    A visualisation idea is to calculate the SUS score corresponding to the measured UMUX-Lite score based on one of the correlation formulas on top (or just by using the UMUX-Lite score directly). Then plot the system’s grade based on a grading scale.

    The Bangor et. al scale for SUS scores. Score less than 25 is considered "worst imaginable", around 38 "poor", around 52 "OK", around 72 "Excellent", around 98 "Best imaginable".
  • Understanding the UMUX: A guide to the short but accurate questionnaire

    TL;DR: The UMUX is a short, 4 items questionnaire that correlates to the System Usability Scale. It is a valid and reliable way of measuring perceived usability. Its short form makes it a good asset in a researcher’s toolbox of UX Metrics that can be administered in-the-wild or on user testings.


    There are a lot of definitions of usability; the ISO 9241-11 defines usability as a combination of effectiveness, efficiency, and satisfaction. There are also a lot of ways to measure usability, in regards to quantitative usability measurement questionnaires are frontrunners, with the System Usability Scale being the most commonly used and researched. However, the SUS consists of 10 questions and can be difficult to administer in large, in-the-wild, studies. Kraig Finstad understood that and in 2010 introduced the Usability Metric for User Experience (UMUX) with 4 questions.

    Setting up the UMUX

    The questions of the UMUX

    The UMUX consists of 4 questions, these questions measure the perceived usability as described in the relevant ISO. As in the SUS the tone of the questions changes between positive and negative.

    1. [This system’s] capabilities meet my requirements.
    2. Using [this system] is a frustrating experience.
    3. [This system] is easy to use
    4. I have to spend too much time correcting things with [this system].

    Example of questions with changed system’s wording

    The design of the questionnaire allows for changing the content of the brackets ([This system’s]) to the actual name of the system, or a more suitable word for the application.

    • This application’s capabilities meet my requirements.
    • Using Google Forms is a frustrating experience.

    etc.

    The responses of the UMUX

    The UMUX can be administered either verbally or through some written format (including digital questionnaires). Each question is scored on a Likert scale, from 1 (strongly disagree) to 7 (strongly agree).

    A sample Google Form for administering the UMUX.

    Calculating the UMUX score

    The rules:

    • Positively worded questions are 1 and 3
    • Negatively worded questions are 2 and 4

    Scoring each question:

    1. For positively worded questions, take the number of the reply and subtract 1 (e.g. if the answer to the question 1 is Slight Agreement—5, then the question is scored: 5 – 1 = 4)
    2. For negatively worded questions, take the number of the reply and subtract it from 7 (e.g. if the answer to the question 2 is Slight Agreement—5, then the question is scored: 7 – 5 = 2)

    Find the participant score:

    We have the score of each question, to calculate the UMUX score for each participant:

    1. Add all the questions’ score
    2. Multiply by 4.1667 (or 100 / 24)
    3. You will get a score between 0 and 100, this is the UMUX score

    To sum up, the formula for calculating the participant’s UMUX score is:

    UMUX = ((Item_1 − 1) + (Item_3 − 1) + (7 − Item_2) + (7 − Item_4)) * (100/24)

    The overall score of the UMUX (the system’s score)

    Often one would use the UMUX to “put a number” to the usability of the whole system / application in question. In that case, the System’s score (or overall UMUX score) is the average of all the participant’s scores.

    Presenting the UMUX results

    Except for presenting the mean UMUX score of the system, usually the Standard Deviation and participants volume (sample size) are presented as well.

    How many participants should someone include to get significant results?

    To the point of the writing (will update if new research is published), unlike in the SUS, there are no general guidelines of the minimum number of participants required to achieve accurate results. Finstad created the questionnaire in a study with 42 participants, while Berkman and Karahoca validated it with 3 studies of 363, 42, and 151 participants — to give some general numbers. When working with relatively low numbers of participants, Laplace’s rule of succession is your friend.

    Is the UMUX a good metric for measuring User Experience?

    UMUX is a reliable metric. It is a great metric of usability, which can be seen as a part of User Experience. It has been validated numerous times. Berkman and Karahoca investigated deeply the structure, validity, and reliability of the UMUX, they also found that the UMUX scores correlate with the scores of the System Usability Scale (SUS) and the Computer System Usability Questionnaire (CSUQ).

    Nonetheless UMUX is a usability questionnaire. If one wants to evaluate a system they can use UMUX and obtain similar results to other validated questionnaires. The age and genre of the participants shouldn’t affect the UMUX scores.

    Is the UMUX affected by participant’s characteristics?

    As with other usability questionnaires, the previous experience of the participant with the system affects the resulting score. Participants that have used the system in the past would tend to rate the system higher, than participants who haven’t. Berkman and Karahoca also reported that non-native English speakers gave slightly different UMUX scores than native English speakers.

    Translations of the UMUX

    UMUX has been translated and validated in a couple of languages:

    The Arabic UMUX

    The adaptation can be found in the original paper by Khalid S. Al-Tahat.

    The Chinese UMUX

    1. 该系统的功能能够满足我的要求.
    2. 使用该系统是令人沮丧的经历.
    3. 该系统易于使用.
    4. 我必须花大量时间去适应该系统.

    Adapted by Yuhui Wang, Jing Chai & Tian Lei.

    The Greek UMUX

    1. Οι δυνατότητες [αυτού του συστήματος] ανταποκρίνονται στις απαιτήσεις μου.
    2. Η χρήση [αυτού του συστήματος] είναι μία απογοητευτική εμπειρία.
    3. [Αυτό το σύστημα] είναι εύκολο στη χρήση.
    4. Πρέπει να αφιερώσω πολύ χρόνο για να διορθώσω πράγματα με [αυτό το σύστημα].

    Adapted by Christos Katsanos, Georgios Melissourgos, Nikolaos Tselios.

    The Slovene UMUX

    1. Zmogljivosti [] izpolnjujejo moje zahteve.
    2. Uporaba [] me frustrira.
    3. [] je enostaven za uporabo.
    4. Preveč časa porabim za popravljanje napak ob uporabi [].

    Adapted by Urška Laha, James R. Lewis, and Boštjan Šumak.

    The Turkish UMUX

    The adaptation can be found in the original paper by Mİ Berkman, Ş Şahin.

  • The positive SUS: as good as the original

    TL;DR: The positive SUS, designed by Sauro and Lewis in 2011, is as good at measuring perceived usability as the original System Usability Scale designed by Brooke in 1996. The positive SUS could help in studies in which cognitive load is a concern, while the original SUS ability to detect acquiescence bias could be used in more error-prone studies.


    The System Usability Scale is arguably one of the most used perceived usability questionnaires. However, since its original publication by Brooke in 1996 there have been several criticisms, one of the most common being the alternation between negatively and positively worded items. In the SUS questionnaire, all even questions are negatively worded, and all odd are positively worded; this design tries to mitigate acquiescence bias. Nonetheless, there are four main lines of reasoning against that wording alternation:

    1. Negatively worded questions increase the cognitive load
    2. They reduce the psychometric qualities of the questionnaire, as measured by Cronbach’s coefficient
    3. They increase the likelihood of replies miscoding
    4. They do not help that much in addressing the acquiescence bias

    The positive SUS

    In 2011 Jeff Sauro and James R. Lewis introduced the positive System Usability Scale (positive SUS), which is a modified version of the original SUS with even items having positive wording. They read as follows:

    1. I think that I would like to use the website frequently.
    2. I found the website to be simple.
    3. I thought the website was easy to use.
    4. I think that I could use the website without the support of a technical person.
    5. I found the various functions in the website were well integrated.
    6. I thought there was a lot of consistency in the website.
    7. I would imagine that most people would learn to use the website very quickly.
    8. I found the website very intuitive.
    9. I felt very confident using the website.
    10. I could use the website without having to learn anything new.
    The positive SUS questions in a google form
    Google form of sample positive SUS https://forms.gle/7Y9jQ74sCUUV6ARR7

    Calculation of the positive SUS

    Calculation of the positive System Usability Scale is more straightforward than the original SUS, that’s part of the reason it was created after all. In short:

    Scoring each question:

    • Take the number of the answer and subtract 1 (e.g. if the answer was disagree, the score is 2 – 1 = 1)

    Calculating the participant’s score

    1. Add all 10 questions’ scores
    2. Multiply that number by 2.5
    3. You will get a number between 0 and 100, which is the SUS score

    Calculating the mean (total) score

    Most of the time, when stating the SUS score the total usability of a system is the mean of all the participants’ scores.

    Mean SUS = (score of participant 1 + score of participant 2 + … + score of participant n) / n

    There are free utilities that will perform this calculation just by adding the raw score, for example our (free) positive SUS calculator.

    The same interpretation of the score applies using the scale developed by A. Bangor, P.T. Kortum, and J.T. Miller

    The Bangor et. al scale for SUS scores. Score less than 25 is considered "worst imaginable", around 38 "poor", around 52 "OK", around 72 "Excellent", around 98 "Best imaginable".

    Is the positive SUS equivalent to the original SUS?

    The positive SUS was introduced alongside two studies that justified its validity and correspondence to the original System Usability Scale. Philip Kortum et al. in 2020 compared the two questionnaires by surveying retrospectively a short of 1000 participants, they found that both questionnaires are measuring perceived usability similarly. In general, they can be considered equivalent and the same principles should apply when examining SUS scores.

    The positive SUS is as valid as the original SUS

    In the same study, when measuring internal reliability the positive SUS showed constantly higher measures of validity compared to SUS. However, the reasoning behind that is not understood. The higher internal reliability coefficient could be due to participants being less careful in their replies —tending to agree with all the questions. At last, we can say that both questionnaires show the same validity.

    Should we all switch to using the positive SUS?

    For the time being, the answer is no. Both questionnaires are equivalent and usability practitioners can use them independently based on the specific study attributes and personal experience. The positive SUS could help in studies in which cognitive load is a concern, while the original SUS ability to detect acquiescence bias could be used in more error-prone studies.

  • Presenting quantitative usability test results

    Presenting quantitative usability test results

    Setting up the test

    It probably doesn’t come as a surprise, for presenting quantitative test results we need to define and set up the actual test. There are a ton of ways to approach gathering usability results. A way to gather quantitative data is questionnaires.

    Measuring usability quantitatively 

    Usability questionnaires have gathered the attention of researchers for years. Almost whichever we end up picking it will result in some sort of scale (a number) from “completely unusable” to “completely usable”. Picking the best questionnaire is something that we will only slightly discuss here, as it deserves a post for itself. But to sum up, there are different questionnaires that try to work around an inevitable truth “one cannot measure everything with one question”. 

    The System Usability Scale is pretty accurate at identifying the usability of a given system, whether the 10 questions it needs to measure that are a lot, depends on the research context. UMUX-Lite just uses 2 questions, but lacks a bit in accuracy [2]. Other questionnaires, like Net Promoter Score, come with their own trade offs.

    In this example we will use the System Usability Scale (SUS), as it is one of the most researched [1], and others usability questionnaires scores can be projected to its scale with good accuracy [2].

    Understanding the optimal number of participants

    Depending on the questionnaire chosen, different guidelines exist. In case of SUS we can follow the diagram below .

    Comparison of accuracy of different questionnaires based on sample size. For SUS: 
6	35%
7	55%
8	75%
9	78%
10	80%
11	90%
12	100%
    Image courtesy of Thomas S. Tullis and Jacqueline N. Stetson

    Define goals

    There is a relation of cost-to-accuracy in each usability test we want to perform. Getting from 75% to 100% requires 4 more users (50% increase in cost). At times, 75% percent would be enough; others 100% accuracy is needed. Still even if we use 75%, there are ways to present the information with 100% accuracy. More on that later.

    Uncover limitations

    Using a questionnaire to assess usability of a system doesn’t come without its pitfalls. A usability score doesn’t show where the problem lies. Scores are comparable between one system, and even if a system has way higher usability than another; the only comparison is about which of the two systems is more usable, not by how much. The size of the system plays a role as well, small systems tend to create stricter scores [4].

    Presenting the results

    So, we have defined the process we want to test and created the experiment (instructions, mockups, redirections, etc.). Found the participants, they have gone through it, and they have answered the questionnaire as well.

    Presenting based on target accuracy

    If the end accuracy is 100%, then presenting is straight forward. However, if the target accuracy was some other number below 100%, or we didn’t get enough participants (to reach 100%), let’s discuss two ways to present the results.

    Present in ranges to improve accuracy

    Eventually what a score with low accuracy means is: “we are not sure about the exact usability score, but it should be around x”. The problem that we want to solve then is, how to find the “how around x it is”. 

    To calculate that, let’s imagine that we wanted to achieve 100% accuracy in our SUS questionnaire, but we got 10 participants. We know that 10 participants resulted in 80% accuracy (see image above). The SUS score that we got is 63.5. So, “we are not sure about the exact usability score, but it should be around 63.5”.

    In order to reach 100% accuracy we need 2 more people, in total 12. Let’s first examine the worst case scenario: 

    Two more participants appeared —magically— and they both rated the system as “completely unusable”, both giving a 0 out of 100 score.

    One the bright side, the best case scenario would be:

    Two more participants appeared and they both rated the system as “completely usable”, both giving a score of 100 out of 100.

    In order to calculate SUS score with 100% accuracy, we just need to find out how much the score is in the worst case scenario and in the best case scenario. To calculate the worst case scenario score we: 

    1. Calculate each participant’s score
    2. Add them all together
    3. Subtract by 12

    That will give 53, in our example.

    To calculate the worst case scenario score we:

    1. Calculate each participant’s score
    2. Add them all together and add (12 – 10) * 100
    3. Subtract by 12

    That will give 70 in our example.

    This can work regardless of the number of participants or target accuracy. Some examples:

    If we have 9 participants and we want to figure out the range with 100% accuracy

    Worst case scenario:

    1. Calculate each participant’s score
    2. Add them all together
    3. Subtract by 12

    Best case scenario:

    1. Calculate each participant’s score
    2. Add them all together and add (12 – 9) * 100
    3. Subtract by 12
    If we have 6 participants and we want to figure out the range with 80% accuracy

    For 80% accuracy, we need 10 participants.

    Worst case scenario:

    1. Calculate each participant’s score
    2. Add them all together
    3. Subtract by 10

    Best case scenario:

    1. Calculate each participant’s score
    2. Add them all together and add (10 – 6) * 100
    3. Subtract by 12

    The Laplace’s Rule of Succession

    Another way to present the results is answering the question “What is the probability that the next person, who will use the system, will rate it with at least good usability?”

    Let’s try to answer that question:

    For a system to have good usability that means that its SUS is more than 73. So:

    1. Go through all SUScores and assign as a “success” if the score is more than 73.
    2. Count the number of “successes” and add 1
    3. Divide that number by the total amount of participants plus two

    To talk numbers, let’s say that we have these 10 scores:

    78SUCCESS
    62,5FAIL
    36FAIL
    75SUCCESS
    83SUCCESS
    39FAIL
    67FAIL
    37FAIL
    78SUCCESS
    81SUCCESS

    5 of them are a success. That means the probability of the system to be rated with good usability is (5 + 1) / (10 + 2) = 50%.

    With the same logic, the probability of the system to be rated with OK usability (SUS > 52) is 9 / 12 = 75%.

    Card that includes: 
SUS Score: 53-70
Accuracy: 100%
Total Participants: 10
Mean System Grade: D
Probability of Good: 50%
Maximum Acceptability: Marginal
    A card for presenting the usability results

    Discussion

    Calculating ranges

    Ranges are an inevitable truth of low accuracy. Essentially getting a score, let’s say 63, with an accuracy of 75% means uncertainty about the results. We are not sure that the actual usability of the system is 63, it can be 75, or 58, or something in between. 

    Now, a lot of times stakeholders will prefer a single number, rather than an interval. The interval may look like it doesn’t give that much information — what does it mean that the system has between 52 and 70 usability? At the same time, presenting a single number with a low accuracy contains the exact same (or the lack of) information. In a way, the interval is a visualisation of the accuracy. 

    How large the interval is, correlates with how uncertain we are. If we have great uncertainty, then the interval will be large. If we are very certain, then the interval will be very small. And of course, if we are 100% certain then the interval is the smallest possible —a single value.

    6 Participants: 32 - 80 SUS
8 Participants: 37 - 75 SUS
10 Participants: 53 - 70 SUS
12 Participants: 65 SUS
    As participants number lowers, the accuracy lowers as well thus the interval becomes larger.

    We can leverage the visualisation of the accuracy (the interval) to get valuable information. Some insights that we might gather is whether the higher or lower end of the interval is acceptable or not. If the higher end is not (or marginally) acceptable, this suggests that we are dealing with a low usability system, even if we don’t know the exact number. If the lower end is acceptable, the system in question probably has good usability. The same apply with adjective ratings and grade scales.

    Eventually someone may stumble upon the thought that not all scores are equally likely. For example, the likelihood of a participant giving a score of 0 when eight already gave an average of 63.5, is unlikely. Let alone 4 of them giving zeros. The same applies for 100. 

    This would be a correct statement. Indeed, the people who would have come in reality would most probably give scores close to the ones we already saw. This opens a whole new discussion, as mathematicians tried for years to understand and predict the behaviours of systems like this. Gradient noises, like Perlin noise, were introduced to create random data linked to the previous value. Then the discussion starts to include sensitivity of systems and extreme values, which we will leave for another time.

    Talking about probabilities, there is something that we can calcualte…

    The Laplace’s Rule of Succession

    From the previous discussion a logical question that might arise is: “Okay, we know that the system has a relatively good usability, but how likely is it that the next person will find it actually good?”

    Essentially, we can measure the probability of anything on the SUS scale: 

    • What is the probability the next person, who will use the system, will rate it with at least excellent/good/poor etc. ?
    • What is the probability the next person, who will use the system, will rate it with at least being acceptable ?
    • What is the likelihood of dealing with an A grade system?
    • …

    The method stays the same as we discussed before, and once again we are trying to express our uncertainty. 

    The probability of a specific question will be based on two factors:

    1. The number of participants. This is intuitive, the more participants we have the better we can calculate a probability. If we have the amount of participants for the target 100% accuracy, we don’t even need to calculate the probability; we are certain.
    2. If the assumption is at the lower end or the higher end of the interval. For example, if we try to find the probability of the system being at least poor, in the example with the 8 participants above, it will be 66%. The probability of it being at least the worst imaginable is 91%. This happens because of the simple fact that both the probability and the interval measure the same things —uncertainty.

    If this whole discussion sparked your curiosity, 3Blue1Brown has a video that introduces these concepts with very good explanations, and more broad examples.

    Plot of the function p = (s + 1) / 14
As the s becomes large the probability converges to 1.
    As success number increases, the probability of a given question increases as well

    References

    [1] https://uxpajournal.org/sus-a-retrospective/ 

    [2] https://www.researchgate.net/profile/James-Lewis-8/publication/262344995_UMUX-LITE_when_there%27s_no_time_for_the_SUS/links/5bb2daf892851ca9ed33c239/UMUX-LITE-when-theres-no-time-for-the-SUS.pdf 

    [3] https://www.researchgate.net/profile/Ginanjar-Wiro-Sasmito-2/publication/339130765_Usability_Testing_based_on_System_Usability_Scale_and_Net_Promoter_Score/links/5f92db3a299bf1b53e3dab1b/Usability-Testing-based-on-System-Usability-Scale-and-Net-Promoter-Score.pdf 

    [4] http://tecfa.unige.ch/tecfa/maltt/ergo/1415/UtopiaPeriode4/articles/JUS_Kortum_November_2013.pdf 

  • System Usability Scale 101 — An extensive insight in quantitative usability

    TL;DR: SUS is a very reliable measurement of a system’s perceived usability. It uses 10 questions, graded from strongly disagree to strongly agree, to measure perceived usability.


    A large amount of effort is attributed to uncovering usability issues of a system. The System Usability Scale is a quantitative measure of the usability of the system. Although it doesn’t pinpoint specific attributes that affect usability, it is shown to be one of the most powerful ways to put a number on the system’s perceived usability.

    The purpose of this document is to provide an extensive insight into the current state of the System Usability Scale, with the most recent information. Updated regularly.

    Setting up the System Usability Scale (SUS)

    The SUS is a questionnaire that can be administered either verbally in a moderated environment or written in an unmoderated environment. It was initially introduced in 1986 by John Brooke as an undimensional, quick-and-dirty way of measuring usability and was validated numerous times.

    The questions of the SUS

    The SUS consists of 10 questions, interchanging positive and negative wordings. See below the original questions with the number 8 modified by Finstad for non-native English speakers

    1. I think that I would like to use this system frequently.
    2. I found the system unnecessarily complex.
    3. I thought the system was easy to use.
    4. I think that I would need the support of a technical person to be able to use this system.
    5. I found the various functions in this system were well integrated.
    6. I thought there was too much inconsistency in this system.
    7. I would imagine that most people would learn to use this system very quickly.
    8. I found the system very cumbersome/awkward to use.
    9. I felt very confident using the system.
    10. I needed to learn a lot of things before I could get going with this system.

    The responses of the SUS

    Each question is supposed to be answered on a scale from 1 (Strongly Disagree) to 5 (Strongly Agree). Participants shouldn’t dwell on any specific questions, but instead respond with their first thoughts. If the test is administered verbally and the participant doesn’t answer a specific question, it is scored as a 3.

    Sample SUS – https://docs.google.com/forms/d/e/1FAIpQLSf3rR2baAwftb8PQN2ot3hWDhuDfcK-OF-ZLYeUSDS9ZMwOmQ/viewform

    Calculating the score of SUS

    The rules:

    • Each question’s score ranges between 0 and 4
    • Positively worded questions are 1, 3, 5, 7 and 9
    • Negatively worded questions are 2, 4, 6, 8 and 10

    Scoring each question:

    1. For positively worded questions, take the number of the answer and subtract 1 (e.g. if the answer to the first question was neutral (3), then the score of the question is 3 – 1 = 2)
    2. For negatively worded questions, take the number of the answer and subtract it from 5 (e.g. if the answer to the second question was agree (4), then the score of that question is 5 – 4 = 1)

    After that, the overall score of the SUS is calculated as follows:

    1. Add all 10 questions’ scores
    2. Multiply the number by 2.5
    3. You will get a number between 0 and 100, which is the SUS score

    If you want to save the manual work, you can just enter the scores of each question and get all relevant results in our free SUS calculator.

    Extracting learnability measurements

    Although there was some evidence by James R. Lewis and Jeff Sauro that SUS measures two dimensions of one’s system: Learnability and Usability, this is not considered the case in most studies. Instead, Lewis and Sauro reviewed 9000 questionnaires to find a factor structure of negatively vs positively worded questions, rather than Usability and Learnability. However, there is still some connection between questions 4 and 10 that, in some cases, may be strong enough to form a separate factor.

    If in the specific field of the study, learnability is a strong factor (most of the time not the case), questions 4 and 10 form a separate factor (specifically measure learnability), the SUS’s learnability score can be extracted as follows:

    1. Add the scores of questions 4 and 10
    2. Multiply the result by 12.5
    3. You will get a number between 0 and 100, which corresponds to the learnability of the system

    Understanding the score of SUS

    A score of 0 translates to “it is impossible to perform the required task”, whereas a score of 100 means that the required task is performed easily. However, most systems are evaluated by testing a flow with numerous tasks; thus, realistically, it is very difficult for a score to be close to 0. In general, it is suggested to use the System Usability Score as a relative measurement of the system tested. A system with a higher SUS than another means that it is more usable, although how much more usable is not up to the SUS to show.

    One way to understand the score of the SUS is to compare the evaluated application’s SUS score with a pool of other SUS scores. Two common ways of qualifying the SUS scores are the grading scales below.

    Adjective Rating Scale

    Aaron Bangor, Philip Kortum, and James Miller have correlated the numerical score of the SUS with a perceived adjective meaning. By asking users to both rate an application with the SUS and replying to the question “Overall, I would rate the user-friendliness of this product as: Worst Imaginable -> Best imaginable” they created a qualitative meaning of each score range (for example, a SUS score larger than 50.9 and less than 71.4 means that users perceive the usability of the application as “Good”). They then created a grading scale from F (SUS score < 60) to A (SUS Score >90) with all grades in the middle spanning 10 points. Presenting the grade instead (or alongside) a SUS score gives a qualitative understanding and comes in handy for setting KPIs.

    The Bangor et. al scale for SUS scores. Score less than 20 is considered "worst imaginable", around 36 "poor", around 51 "OK", around 71 "Good" around 86 "Excellent", around 100 "Best imaginable".
    AdjectiveMean SUS ScoreStandard Deviation
    Worst Imaginable12.513.1
    Awful20.311.3
    Poor 35.712.6
    OK 50.9 13.8
    Good 71.4 11.6
    Excellent 85.5 10.4
    Best Imaginable 90.9 13.4

    Curved Grading Scale

    Sauro and Lewis analysed 446 SUS studies (including the data from Bangor et al. above) and noted that the average of the SUS score from approximately 500 products is 68. They then created a curved grading scale that has 68 as the centre and provided with percentiles for each grade to enable comparison of a SUS score with that dataset of studies (for example, a SUS score of 79 is considered an A- which is assigned to a percentile range of 85-89. That means that the evaluated application with SUS score 79 has a better usability than at least 85% of the studies in the dataset; see also Quantifying the User Experience: Practical Statistics for User Research, Sauro and Lewis: 204).

    SUS Score RangeGradePercentile Range
    84.1–100A+96–100
    80.8–84.0A90–95
    78.9–80.7A-85–89
    77.2–78.8B+80–84
    74.1–77.1B70–79
    72.6–74.0B-65–69
    71.1–72.5C+60–64
    65.0 –71.0C41–59
    62.7–64.9C-35–40
    51.7–62.6D15–34
    0.0–51.6F0–14

    How many participants should be included in a SUS test

    The power of SUS is also shown in the sample size needed to achieve statistical significance. Thomas S. Tullis and Jacqueline N. Stetson showed that for 12 participants or more, the SUS result remains the same.

    Image courtesy of Thomas S. Tullis and Jacqueline N. Stetson

    Can you change the word “system” in the SUS questions?

    SUS is quite resilient to changes in the name of the “system”. Bangor et al. changed the word “system” to “product” and didn’t see a significant change in the results. This suggests that probably other, more specific, synonyms of “system” could be used safely.

    Internationalised SUS vs non-native English speakers

    The SUS is tested with non-native English speakers, and its reliability stays the same. However, several researchers have translated and validated SUS into local languages. If the participants are native speakers of a language other than English, it is worth considering using the local version of the SUS.

    Does perceived Usability affect loyalty?

    There exists initial data to support a link between the perceived usability of a system and users’ loyalty. Users who are promoters, measured with NPS, rate the system with substantially higher usability than passives and detractors. That could potentially indicate that improving usability could increase the likelihood of a user becoming a promoter.

    What about the positive SUS?

    Sauro and Lewis developed an all-positively worded version of the System Usability Scale. It performs as well and is as reliable as the original SUS.

    Validated translations of the System Usability Scale

    The Chinese System Usability Scale

    1. 我愿意经常使用这个系统.
    2. 我发现这个系统太复杂.
    3. 我认为这个系统使用起来很容易.
    4. 我认为我需要技术人员的帮助才能使用这个系统.
    5. 我发现这个系统很好地集成了各种功能.
    6. 在使用该系统的过程中, 我发现很多操作结果和预想功能不一致.
    7. 我想大多数用户能很快学会使用该系统.
    8. 我发现这个系统使用起来很别扭.
    9. 我对熟练掌握这个系统很有信心.
    10. 在使用这个系统之前, 我需要学习很多知识.

    Adaption by Yuhui Wang, Tian Lei & Xinxiong Liu

    The Danish System Usability Scale

    1. Jeg tror godt, at jeg kunne tænke mig at bruge systemet ofte.
    2. Jeg synes, at systemet var unødigt kompliceret at bruge.
    3. Jeg synes, at systemet var nemt at bruge.
    4. Jeg tror, at jeg ville få brug for support fra en tekniker for at kunne bruge systemet.
    5. Jeg synes, at mange af systemets funktioner var godt integreret med hinanden.
    6. Jeg synes, at systemet var usammenhængende.
    7. Jeg tror, at de fleste hurtigt kan lære at bruge dette system.
    8. Jeg synes, at systemet var meget besværligt at bruge.
    9. Jeg følte mig sikker på, at jeg kunne bruge systemet.
    10. Jeg var nødt til at lære en masse ting, for at komme i gang med at bruge systemet.

    Adaption by

    Julie Christine S. Hvidt, Lone Fisker Christensen, Christian Sibbersen, Stig Helweg-Jørgensen, Jens Peter Hansen & Mia Beck Lichtenstein

    The French System Usability Scale

    1. Je voudrais utiliser ce système fréquemment.
    2. Ce système est inutilement complexe.
    3. Ce système est facile à utiliser.
    4. J’aurais besoin du soutien d’un technicien pour être capable d’utiliser ce système.
    5. Les différentes fonctionnalités de ce système sont bien intégrées.
    6. Il y a trop d’incohérences dans ce système.
    7. La plupart des gens apprendront à utiliser ce système très rapidement.
    8. Ce système est très lourd à utiliser.
    9. Je me suis senti·e très en confiance en utilisant ce système.
    10. J’ai eu besoin d’apprendre beaucoup de choses avant de pouvoir utiliser ce système.

    Adaptation by Guillaume Gronier & Alexandre Baudet

    The German System Usability Scale

    1. Ich denke, dass ich das System gerne häufig benutzen würde.
    2. Ich fand das System unnötig komplex.
    3. Ich fand das System einfach zu benutzen.
    4. Ich glaube, ich würde die Hilfe einer technisch versierten Person benötigen, um das System benutzen zu können.
    5. Ich fand, die verschiedenen Funktionen in diesem System waren gut integriert.
    6. Ich denke, das System enthielt zu viele Inkonsistenzen.
    7. Ich kann mir vorstellen, dass die meisten Menschen den Umgang mit diesem System sehr schnell lernen.
    8. Ich fand das System sehr umständlich zu nutzen.
    9. Ich fühlte mich bei der Benutzung des Systems sehr sicher.
    10. Ich musste eine Menge lernen, bevor ich anfangen konnte das System zu verwenden.

    Translation by Rummet et al. picked by Tobias J. Brix, Alice Janssen, Michael Storck, Julian Varghese

    The Greek System Usability Scale

    1. Νομίζω οτι θα ήθελα να χρησιμοποιώ αυτό το σύστημα συχνά.
    2. Βρήκα αυτό το σύστημα αδικαιολόγητα περίπλοκο.
    3. Σκέφτηκα ότι αυτό το σύστημα ήταν έυκολο στη χρήση.
    4. Νομίζω οτι θα χρειαστώ βοήθεια από κάποιον τεχνικό για να είμαι σε θέση να χρησιμοποιήσω αυτό το σύστημα.
    5. Βρήκα τις διάφορες λειτουργίες σε αυτό το σύστημα καλά ολοκληρωμένες.
    6. Σκέφτηκα οτι υπήρχε μεγάλη ασυνέπεια σε αυτό το σύστημα.
    7. Φαντάζομαι οτι οι περισσότεροι άνθρωποι θα μάθουν να χρησιμοποιούν αυτό το σύστημα πολύ γρήγορα.
    8. Βρήκα αυτό το σύστημα πολύ περίπλοκο/δύσκολο στη χρήση.
    9. Ένιωσα πολύ σίγουρος/η χρησιμοποιώντας αυτό το σύστημα.
    10. Χρειάστηκε να μάθω πολλά πράγματα πριν μπορέσω να ξεκινήσω με αυτό το σύστημα.

    Adaptation by Christos Katsanos, Nikolaos Tselios, Michalis Xenos

    The Indonesian System Usability Scale

    1. Saya berpikir akan menggunakan sistem ini lagi.
    2. Saya merasa sistem ini rumit untuk digunakan.
    3. Saya merasa sistem ini mudah untuk digunakan.
    4. Saya membutuhkan bantuan dari orang lain atau teknisi dalam menggunakan sistem ini.
    5. Saya merasa fitur-fitur sistem ini berjalan dengan semestinya.
    6. Saya merasa ada banyak hal yang tidak konsisten (tidak serasi) pada sistem ini.
    7. Saya merasa orang lain akan memahami cara menggunakan sistem ini dengan cepat.
    8. Saya merasa sistem ini membingungkan.
    9. Saya merasa tidak ada hambatan dalam menggunakan sistem ini.
    10. Saya perlu membiasakan diri terlebih dahulu sebelum menggunakan sistem ini.

    Adaptation by Zahra Sharfina, Harry Budi Santoso

    The Malay (Mobile) System Usability Scale

    1. Saya rasa saya ingin kerap menggunakan aplikasi mudah alih ini.
    2. Saya rasa aplikasi mudah alih ini sesuatu yang rumit.
    3. Saya rasa aplikasi mudah alih ini mudah untuk digunakan.
    4. Saya memerlukan bantuan daripada orang lain untuk menggunakan aplikasi mudah alih ini.
    5. Saya mendapati fungsi-fungsi dalam aplikasi mudah alih ini saling bersepadu (berhubungkait) dengan baik.
    6. Saya rasa terdapat banyak kandungan di dalam aplikasi mudah alih ini yang tidak konsisten antara satu sama lain.
    7. Saya membayangkan bahawa kebanyakan orang akan cepat belajar menggunakan aplikasi mudah alih ini.
    8. Saya mendapati aplikasi mudah alih ini leceh untuk digunakan.
    9. Saya berasa yakin menggunakan aplikasi mudah alih ini.
    10. Saya perlu belajar terlalu banyak perkara sebelum boleh menggunakan aplikasi mudah alih ini.

    Adaptation by Muhamad Fadhil Mohamad Marzuki, Nor Azwany Yaacob, Najib Majdi Yaacob

    The Polish System Usability Scale

    1. Myślę, że często używałbym/używałabym tego systemu.
    2. Odbieram ten system jako niepotrzebnie skomplikowany.
    3. Myślę, że system jest łatwy w użyciu.
    4. Myślę, że potrzebowałbym/potrzebowałabym wsparcia asystenta, aby używać tego systemu.
    5. Uważam, że różne funkcje tego system są dobrze zintegrowane.
    6. Myślę, że w tym systemie jest za dużo niespójności.
    7. Wydaje mi się, że większość ludzi nauczyłaby się bardzo szybko używać tego systemu.
    8. Odbieram ten system jako bardzo niewygodny w użyciu.
    9. Czułem/czułam się bardzo pewnie używając tego systemu.
    10. Musiałem/musiałam nauczyć się wielu rzeczy, zanim zacząłem/zaczęłam właściwie posługiwać się tym systemem.

    Adaptation by Anna Borkowska and Katarzyna Jach

    The European Portuguese System Usability Scale

    1. Acho que gostaria de utilizar este produto com frequência.
    2. Considerei o produto mais complexo do que necessário.
    3. Achei o produto fácil de utilizar.
    4. Acho que necessitaria de ajuda de um técnico para conseguir utilizar este produto.
    5. Considerei que as várias funcionalidades deste produto estavam bem integradas.
    6. Achei que este produto tinha muitas inconsistências.
    7. Suponho que a maioria das pessoas aprenderia a utilizar rapidamente este produto.
    8. Considerei o produto muito complicado de utilizar.
    9. Senti-me muito confiante a utilizar este produto.
    10. Tive que aprender muito antes de conseguir lidar com este produto.

    Adaptation by Ana Isabel Martins,*, Ana Filipa Rosa, Alexandra Queirós, Anabela Silva, Nelson Pacheco Rocha

    The Spanish System Usability Scale

    1. Me gustaría usar esta herramienta frecuentemente.
    2. Considero que esta herramienta es innecesariamente compleja.
    3. Considero que la herramienta es fácil de usar.
    4. Considero necesario el apoyo de personal experto para poder utilizar esta herramienta.
    5. Considero que las funciones de la herramienta están bien integradas.
    6. Considero que la herramienta presenta muchas contradicciones.
    7. Imagino que la mayoría de las personas aprenderían a usar esta herramienta rápidamente.
    8. Considero que el uso de esta herramienta es tedioso.
    9. Me sentí muy confiado al usar la herramienta.
    10. Necesité saber bastantes cosas antes de poder empezar a usar esta herramienta.

    Adaptation by Magdalena Del Rocio Sevilla-Gonzalez, Lizbeth Moreno Loaeza, Laura Sofia Lazaro-Carrera, Brigette Bourguet Ramirez, Anabel Vázquez Rodríguez, María Luisa Peralta-Pedrero, Paloma Almeda-Valdes

    The Slovene System Usability Scale

    1. Menim, da bi ta sistem rad pogosto uporabljal.
    2. Sistem se mi je zdel po nepotrebnem zapleten.
    3. Sistem se mi je zdel enostaven za uporabo.
    4. Menim, da bi za uporabo tega sistema potreboval pomoč tehnika.
    5. Različne funkcije tega sistema so se mi zdele dobro povezane v smiselno celoto.
    6. Sistem se mi je zdel preveč nekonsistenten.
    7. Menim, da bi se večina uporabnikov zelo hitro naučila uporabljati ta sistem.
    8. Sistem se mi je zdel neroden za uporabo.
    9. Pri uporabi sistema sem bil zelo suveren.
    10. Preden sem osvojil uporabo tega sistema, sem se moral naučiti veliko stvari.

    Adaptation by Bojan Blažica and James R. Lewis

  • Defining User-Centred Design

    User-centred design, user-centred approach, user-driven development, user-centric organisation, are all trying to describe the same phenomenon. An entity which performs design for a product,  with high-priority consideration and constant input of the person who will use it. Furthermore, we see everyday more and more organisations converting themselves to being user-centric. So the question appears, what is user-centred design and how does it defer from other approaches on design.

    User-centred design

    In the en​​cyclopedia of  Human-Computer Interaction, Abras et al. state:

    “‘User-centered design’ (UCD) is a broad term to describe design processes in which end-users influence how a design takes shape. It is both a broad philosophy and variety of methods”

    Essentially, user-centric design puts a name to the primal need that appears when starting to develop a product: what do the people who are going to use the product want? The differentiating characteristic of the user-centred process —over some other design process e.g. process-centred design— is the emphasis on including the user from early stages, while continuing to iterate over their needs throughout the whole process. Nonetheless, as with every movement, UCD evolved to represent a more humanistic approach to product design; by transferring the focus from process and business objectives to user needs.

    User-centric design as a philosophy

    Donald A. Norman pioneered in shaping the UCD to a set of axioms, all of which focus on one singularity; maximising usability. Developing a product with that constrain —maximisation of usability— refers some involvement with the following principles, that interestingly apply to both physical and digital products.

    • Visibility: The principle that the more visible an element is, the more likely users will know about them and how to use them. 
    • Feedback: The principle of making it clear to the user what action has been taken and what has been accomplished.
    • Constrains: Limiting the range of interaction possibilities for the user to simplify the interface and guide the user to the appropriate next action.
    • Mapping: Having a clear relationship between controls and the effect they have on the world.
    • Consistency:  Having similar operations and similar elements for achieving similar tasks.
    • Affordance: An attribute of an object that allows people to know how to use it.

    Following the path carved by usability optimisation, implies a lean towards user empathy that a lot of the time goes against individual processes. This mentality reveals the fundamental humaine nature of user-centric design. 

    User-centric designs vs intuition

    Intuitive products tend to have high usability, although the opposite is not always true; there are a couple of cases when initially non-intuitive products are designed in a way to maximise usability. A generic guideline of “design intuitive products” is not enough to guide a product. 

    In spite of the fact that user-centred design calls for creation of usable products, that could very well be intuitive products, it calls for the removal of the designer’s intuition. A product designed to be intuitive is done by understanding the user’s intuition; which most of the time is different from the designer’s. Different methodologies have been proposed to help identify what can be intuitive for the user. Usability tests, first-click testing, focus-groups, task analysis, are some examples of those methodologies. 

    Criticism 

    Lately more and more organisations are turning to be user-centred. However, user-centric design lacks a lot of  business needs that might arise. A product designed just with the user-centric process doesn’t include the incentive of making money; as something might as well be extremely usable but to have almost zero business perspective. At the same time, usability is not what eludes the customers to start using a product, it is partly what keeps them. As a last note, the user-centric approach is not an equivalent of market research — it is a way of maximising usability for a product that is already conceived and validated.

    A toy grabber with a pikachu inside
    A toy grabber — very profitable, little usable. Image courtesy Melvina Mak 

    Conclusion

    User-centred design can be a very valuable process for creating products that have high usability. At the same time, it lacks business orientation. Maximisation of usability is one of the core values of user-centric design, whereas designs involve the user of a product throughout the design decisions.

WordPress Cookie Plugin by Real Cookie Banner