Expanding usability testing to evaluate complex systems

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Abstract

When you think of usability testing, do you think about working with someone for an hour or two, watching and listening as they do a series of short, discrete scenarios where there is a clear ending or a correct answer for each scenario? For most of us that's a typical usability test. It's the type of usability test assumed by the Common Industry Format (ISO/IEC 25062, see http://zing.ncsl.nist.gov/iusr/). It's the type being discussed for the CIF-Formative (Theofanos and Quesenbery, 2005). It's the type that the various CUE studies have focused on (see http://www.dialogdesign.dk/cue.html).

References

Albers, M. J. (2003) Complex problem solving and content analysis, in M. J. Albers and B. Mazur (Eds.), Content and Complexity: Information Design in Technical Communication, Mahwah, NJ: Lawrence Erlbaum Associates, 263--284.

Chisnell, D., Redish, J. C., and Lee, A. (2006) New heuristics for understanding older adults as web users, Technical Communication, 53 (1), February, 39--59.

Cowley, P., Nowell, L., and Scholtz, J. (2005) Glass Box: An instrumented infrastructure for supporting human interaction with information. Proceedings of the 38th Annual Hawaii International Conference on System Sciences (HICSS'05) - Track 9 - Volume 09.

Dumas, J. S. (2003) User-based evaluations. In J. Jacko and A. Sears (Eds.), The Human-Computer Interaction Handbook. (pp. 1093--1117) Mahwah, NJ: Lawrence Erlbaum Associates, Inc.

Endsley, M. R. (2000) Theoretical underpinnings of situation awareness: A critical review, in M. R. Endsley and D. J. Garland (Eds.), Situation Awareness Analysis and Measurement. Mahwah, NJ: Lawrence Erlbaum Associates.

Endsley, M. R., Bolté, B., and Jones, D. G. (2003) Designing for Situation Awareness -- An Approach to User-Centered Design. Boca Raton, FL: CRC / Taylor & Francis.

Fischer, G. (2005) Creativity and distributed intelligence, Report of the NSF Workshop on Creativity Support Tools (pp. 71--73), downloaded from http://www.cs.umd.edu/hcil/CST.

Greenbaum, J. and Kyng. M. (1991) Design at Work: Cooperative Design of Computer Systems. Hillsdale, NJ: Lawrence Erlbaum Associates.

Grinstein, G., O'Connell, T., Laskowski, S., Plaisant, C., Scholtz, J., and Whiting, M. (2006) VAST 2006 Contest -- A Tale of Alderwood, Proceedings of the IEEE Symposium on Visual Analytics Science and Technology, pp. 215--216.

Mirel, B. (2004) Interaction Design for Complex Problem Solving -- Developing Useful and Usable Software. San Francisco: Morgan Kaufmann.

Mirel, B. (2003) Dynamic usability: Designing usefulness into systems for complex tasks, in M. J. Albers and B. Mazur (Eds.), Content and Complexity: Information Design in Technical Communication, Mahwah, NJ: Lawrence Erlbaum Associates, pp. 233--262.

Olsson, E. and Jansson, A. (2005) Participatory design with train drivers -- a process analysis, Interacting with Computers, 17, 147--166.

Olsson, E., Johansson, N., Gullikssen, J., and Sandblad, B. (2005) A Participatory Process Supporting Design of Future Work, paper from the Department of Information Technology, Human Computer Interaction, Uppsala University, Sweden, retrieved from http://www.it.uu.se/research/publications/reports/2005-018/2005-018-nc.pdf

Parush, A., Erev-Yehene, V., Straoucher, Z., Rugachev, M., Kedmi, E., and Markovitch, R. (2004) The safety implications of command and control tasks: A comparative study, Research Center for Work Safety and Human Engineering, Israel Institute of Technology, HEIS-04 (available in Hebrew from the first author at [email protected]).

Patterson, E. S. (1999) A simulation study of computer-supported inferential analysis under data overload, Proceedings of the Human Factors and Ergonomic Society, 43rd Annual Meeting, pp. 363--367.

Polson, P. G., Lewis, C., Rieman, J., and Wharton, C. (1992) Cognitive walkthroughs: A method for theory-based evaluation of user interfaces, International Journal of Man-Machine Studies, 36, 741--773.

Redish, J. C. and Scholtz, J. (2007) Evaluating complex information systems for domain experts, paper presented at a symposium on HCI and Information Design to Communicate Complex Information, Memphis, TN: University of Memphis, February. (Paper available by request to the first author at [email protected].)

Scholtz, J. (2006) Beyond usability: Evaluation aspects of visual analytic environments, Proceedings of the IEEE Symposium on Visual Analytics Science and Technology, pp. 145--150.

Scholtz, J., Morse, E., and Potts Steves, M. (2006) Evaluation metrics and methodologies for usercentered evaluation of intelligent systems, Interacting with Computers, 18, 1186--1214.

Schuler, D. and Namioka, A., (Eds.) (1993) Participatory Design: Principles and Practices. Hillsdale, NJ: Lawrence Erlbaum Associates.

Shorrock, S. T. and Kirwan, B. (2002) Development and application of a human error identification tool for air traffic control, Applied Ergonomics, 33, 319--336.

Theofanos, M. and Quesenbery, W., 2005, Towards the design of effective formative test reports, Journal of Usability Studies 1 (1), November, 28--46.

Thomas, J. J. and Cook, K. A. (Eds.) (2005) Illuminating the Path: The Research and Development Agenda for Visual Analytics. Piscataway, NJ: IEEE Press.