Hey there Joel,

I definitely agree with you that in many cases, moderated usability testing can be a good tool to directly gather information from users with the ability to articulate their process in an unbiased fashion.

On the flip-side, part of the other problem that we face is that users can tell us "what" they are doing, and "why" they're doing something, when, in actuality, they don't really know.

Take for example the stocking study by Nisbett and Wilson in 1977. Customers reported on which socks they preferred out of a lineup; A, B, C, or D, from left to right. When asked why they chose what they chose, they had a variety of reasons, but a good deal of the participants chose D, or the furthest set to the right.

Then the researchers changed the order and repeated the experiment. These stockings were of IDENTICAL quality, and, yet again, independent of which pair was in the "D" position, participants overwhelmingly chose D.

Come to find out, its not because a pair were ever any better or wose, but because most people are right-handed and its easier to point to D, than any other pair.

This hightlights a GLARING problem with the standing idea that moderated research is somehow "better," when, generally speaking, all the researcher ends up doing is contaminating their sample while unwittingly projecting their own biases.

In the wild, no one is going to ask the user why they did what they did, and when they are asked, most users can't actually tell you why with any real degree of accuracy.

This is part of the problem that we face with moderation, because the Hawthorne Effect, along with many others, has the propencity to change the outcome(s) of our data in statistically significant ways.

I would assert, based on the evidence on record we have for these biases, and with the current tools at our disposal, that user research is ultimately best conducted in a way that risks sample contamination and/or data bias the least.

I can only speak to my own experience, but I have found that unmoderated, even anonymous, methods almost always produce data that is more accurate, honest, and actionable in ways that speak to what the user has to say, rather than what the user thinks we want to hear.