“User Experiences of Spanish-Speaking Latinos with the Frontier Behavioral Health Website”
by Raquel L. Dean | Xchanges 14.1, Spring 2019
Metrics
This study required a mixed-methods approach to collect data. While each participant navigated through the website and completed each task, I asked users to “think-aloud.” Think-aloud protocol encourages the test participants to use the website while continuously thinking out loud—verbalizing their thoughts as they move through the user interface. As Barnum (2011) states: “hearing from the participant while he or she is working, and learning what pleases, frustrates, confuses, confounds him or her is illuminating” (205).
I used the “see-say-do triangle” to collect data (Still & Crane, 2017): observing what users do (see), listening to what users say (say), and measuring what users do (do). For the “see” element, I observed what the users were doing while interacting with the design. The “say” data — gathered through think-aloud protocol (TAP), pre-test surveys, and post-test interviews — provided user feedback about the design. Finally, the “do” refers to performance data: time on task, mouse clicks, and error rates.
I used two metric scales when analyzing the tasks:
| Rate | Description |
|---|---|
| 0 | Completed with ease—little to no frustration or confusion |
| 1 | Completed with difficulty—struggled with navigation, locating information, or understanding what was being asked |
| 2 | Failed to complete—not able to complete the task due to a variety of factors |
Table 2: Ease of Completing Task Scale
| Rate | Category | Description |
|---|---|---|
| 1 | Catastrophe | User cannot complete task; user expresses extreme irritation; or user needs assistance |
| 2 | Serious | User is frustrated but gets through it; suggests that others may be less inclined to put up with the inconvenience |
| 3 | Cosmetic (minor) | User may hesitate or pick the wrong option but corrects it without incident; or user expresses minor irritation but completes the task |
Table 3: Error Severity Scale