PUB - Publikationen an der Universität Bielefeld

In subjective speech synthesis evaluation, participants are often instructed to rate voices on absolute category or comparison scales. It has often been claimed that idiosyncratic behaviour of participants has an influence on the obtained ratings. This article studies the influence of participant specific rating behaviour on the resulting quality estimate. Using Bayesian multi-level modelling, it is confirmed that the different systems in a standard Mean Opinion Score (MOS) rating test explain more of the variance than participants’ individual rating behaviour. It is also shown that additionally collected personality traits and affect disposition data do not seem to have a tangible effect on outcome ratings, with an exception of conscientiousness, for which higher values are associated with higher MOS ratings.