who should test our HMI?
should we use people with no prior knowledge of the system or test people with system knowledge?
if you want to capture first impressions you have to test "naive" participants. if you want to test system efficiency and effectiveness you should test participants with some system knowledge.
I am not sure if you can do both. for example, in one on-road study (see video), the owner of the car had no knowledge of a navigation feature and that resulted in 3 times more eyes-of-the-road time in comparison with his 2nd attempt that he knew about the feature. significant difference on performance.
because I really love on-road studies, and because I don't want people learning the system while they are driving, I use people with system knowledge or I train them before the task. I try to capture some, limited, first impressions during system setup (e.g. phone pairing and home/work address setup).
what is your approach ? have you got the resources to run longitudinal studies?
Data collection and analysis as part of a personal study conducted in the summer of 2018. Many thanks to Scott Hodgins and Jon Ward at Tobii Pro