Testing Interfaces with Diverse Populations / Todd Suomela

The study path introduces students to user experience testing reports, and introduces ideas from feminist HCI and psychology in order to critique and improve those tests and reports.

By Todd Suomela, Interim Assistant Director, Digital Pedagogy & Scholarship, Bertrand Library at Bucknell University

Abstract

User interfaces are one of the key domains where people interact with technology. The design process for a user interface may involve user testing. It is crucial that this testing is fair and unbiased. Including a diverse group of people in the testing process is one way to insure an interface is designed for maximum inclusiveness. The study path introduces students to user experience testing reports, and introduces ideas from feminist HCI and psychology in order to critique and improve those tests and reports.

Prerequisite Knowledge

Some background in interface design or user testing may be helpful for understanding the readings, but is not required.

Learning Objectives

  • Students will be able recall and summarize the concepts of WEIRD (Western, Educated, Industrialized, Rich, Developed) Cultures and Design Practice and describe how it is applied to human-computer interaction and system design.
  • Students will identify a user experience report and be able to summarize the methods used in the report, and critique those methods for ways they could include more diverse audiences.
  • Students will design a user testing plan that is inclusive and diverse

Activities

This study path includes two related activities:

Reflection/discussion prompt

The Sweeney case study1 is one potential starting place for discussing the testing of a user interface. Students may also seek out examples of user interface reports from their own personal or organizational experience. The key topic for discussion is evaluating user experience testing and design processes. Do those processes include diverse groups? How should diverse testing populations be reached or included in the process?

Read some user testing reports. Finding user testing reports will require some effort. One approach to this task is to solicit organizations, people, and professional groups to share completed test reports for your class or for self-study. Many reports may be confidential so obtaining a completed report could be a challenge. A second approach is to search the internet for ‘user testing reports’. This will yield some report templates, as well as standard documents for reference (Allen 2015 2 is one place to begin). A third approach is to seek out textbooks on user interface design and look for example reports, or for discussions of diversity (e.g. Norman 20083; Rogers 20124; Shneiderman et al. 20095

  • Is diversity of users discussed in the report?
  • Do the authors or testers discuss how they found the appropriate sample to test the interface?
  • Do you think that the users tested in the report are a sufficient sample of the potential users of the product that is being evaluated?

Read the resources on feminist HCI:

  • How could you incorporate principles from feminist HCI into the user testing activity or experience?
  • How would you revise the testing plan in order to reflect a more diverse community?

Find an online interface for a GLAM6 institution:

  • Is there any information about the testing or design process for the website or interface?
  • Who designed the interface? If this information is not available, ask yourself why the designers did not publish this information?
  • Would knowing the process for testing a user interface change how you interact with the system?

Assignment

This assignment asks students to design a testing plan for a user interface with a focus on identifying a diverse population. The goal is to make students aware of the challenges of designing for a diverse population.
The assignment should:

  • Describe the interface or product to be evaluated.
  • Describe the user tests that will be conducted.
  • Describe the population to be tested.
  • Suggest ways to recruit testers from this population.

If you are learning on your own, see below for some ideas on how you might think through these questions.

Assessment

  1. A written report describing the methods and procedures used in a user testing report. The written essay should include an assessment of the methods to see if they do or do not address diversity and make suggestions for improvement.
  2. Design a user testing plan that explicitly addresses the problem of diversity and offers suggestions for how diverse audiences can be reached. The student could use examples from their own experience, such as how they would perform a user test of a website that they regularly use.
  3. Interview a usability professional and ask them how they address the problem of diversity in their own work. Report back to your peers based on the information you discover in this interview.

Resources

Readings

About Feminist HCI

Bardzell, Shaowen. 2010. “Feminist HCI: Taking Stock and Outlining an Agenda for Design.” In CHI ’10 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1301–10. Atlanta, GA. http://wtf.tw/ref/bardzell.pdf.
Fiesler, Casey, Shannon Morrison, and Amy S Bruckman. 2016. “An Archive of Their Own: A Case Study of Feminist HCI and Values in Design.” In CHI ’16 Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2574–85. Santa Clara, CA. https://caseyfiesler.com/2016/02/09/an-archive-of-their-own-a-case-study-of-feminist-hci-and-values-in-design-chi-2016/.
D’Ignazio, Catherine, Alexis Hope, Becky Michelson, Robyn Churchill, and Ethan Zuckerman. 2016. “A Feminist HCI Approach to Designing Postpartum Technologies: ‘When I First Saw a Breast Pump I Was Wondering If It Was a Joke.’” In , 2612–22. ACM Press. https://doi.org/10.1145/2858036.2858460.

 

About WEIRD (Western, Educated, Industrialized, Rich, Developed) Cultures and Design Practice

Jackson, S. J., & Kang, L. (2014). Breakdown, Obsolescence and Reuse: HCI and the Art of Repair. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (pp. 449–458). New York, NY, USA: ACM. https://doi.org/10.1145/2556288.2557332

https://doi.org/10.1145/2702613.2702656

On User Testing Reports

Allen, Thelma A. 2015. “Industry Usability Reporting.” NIST. May 5, 2015. https://www.nist.gov/itl/iad/industry-usability-reporting

Norman, Kent L. 2008. Cyberpsychology : An Introduction to Human-Computer Interaction. Cambridge; New York: Cambridge University Press.

Rogers, Yvonne. 2012. HCI Theory Classical, Modern, and Contemporary. San Rafael, CA: Morgan & Claypool Publishers.

Shneiderman, Ben, Catherine Plaisant, Maxine Cohen, and Steven Jacobs. 2009. Designing the User Interface: Strategies for Effective Human-Computer Interaction. 5th ed. Addison Wesley.

 
Print This Page Print This Page
 

Footnotes

  1. Sweeney, Sarah. Terp Talks: Designing a Schema to Describe Interpretive Sign Language Videos
  2. Allen, Thelma A. 2015. “Industry Usability Reporting.” NIST. May 5, 2015. https://www.nist.gov/itl/iad/industry-usability-reporting.
  3. Norman, Kent L. 2008. Cyberpsychology : An Introduction to Human-Computer Interaction. Cambridge; New York: Cambridge University Press.
  4. Rogers, Yvonne. 2012. HCI Theory Classical, Modern, and Contemporary. San Rafael, CA: Morgan & Claypool Publishers
  5. Shneiderman, Ben, Catherine Plaisant, Maxine Cohen, and Steven Jacobs. 2009. Designing the User Interface: Strategies for Effective Human-Computer Interaction. 5th ed. Addison Wesley.
  6. Galleries, Libraries, Archives and Museums