By William Jeffrey Rankin, Mon Jul 17 2017
A while back I got into an exchange with another designer on Twitter regarding his conduct of a usability test. It started when he "tweeted" this (these aren't the exact words, but I've captured the gist): "I'm testing with users and updating the design as issues are uncovered."
This surprised me: it didn't seem like a good idea from a methodological perspective, yet here was a fairly well-known designer (who'd written a book or two by the time of our exchange) talking about it like it was business as usual.
I replied something to the effect of "Shouldn't you change the design after you've run all the sessions (and therefore collected all the data)?" We had a brief friendly exchange following this: he didn't see the harm in changing the design as he was running the sessions. I left it at that, but wanted to put down my thoughts on why this is not a good idea.
If you change the design immediately after the session, how do you know the issue is a problem? And to what extent is it a problem? For example, pretend that you ran a series of 12 user-testing sessions, and observed several issues:
Is issue 5 really a problem? Maybe, but certainly not as big of a problem as issues 1 and 2. Had issue 5 been "fixed" early in the test sessions, you wouldn't have been able to know whether it's truly a problem (perhaps it's a methodological or other test anomaly), and you wouldn't know the magnitude of the problem (data necessary for prioritization).
Usability testing, in part (a large part), is about understanding why users are having problems with a product's design. For a given issue, you want as much data as you can get so you can understand the nature of the problem. If you're changing the design (ostensibly to fix an observed issue) as you're testing, you've lost the opportunity to learn more the problem. If you only have 1 data point for an issue, can you really address it with a high level of confidence (and again, is it really a problem)?
A side question: What happens when the mid-test "fix" introduces new issues? It seems like the design and testing sessions could go off the rails pretty quickly with this methodology.
Testing and updating in this way, it's entirely possible that the designer is "fixing" issues that aren't really problems, or are relatively trivial problems. Or, because the designer doesn't have an understanding of the nature of an actual problem, it's not addressed as well as it could've been, had there been more data.
What are your thoughts? Is this a common testing methodology? Is there a context in which it would make sense?