Validity of Data as Precondition for Evidence: A Methodological Analysis of What is Taken to Count as Evidence in Psychotherapy Research

The evidence-based paradigm in mental health care emphasizes the use of the best available methods to provide a sound evidence-base for clinical practice. However, there is strikingly little consensus on what evidence is. Nonetheless, psychotherapy researchers conduct a vast amount of research in which the outcome is taken as evidence. To derive outcome, quantitative data are collected from samples of patient– participants by means of validated self-report measures. Clinical case vignettes from the Ghent Psychotherapy Study (Meganck et al., 2017) are used to exhibit that quantitative self-report data are hermeneutic in their basis. Although quantitative data are generally taken as straightforwardly comparable and trustworthy input for analysis of treatment effect, we show that these data can yield validity issues despite being collected by validated measures. As the gold standard methodological procedure does not prevent that these validity issues become inherent to the dataset, validity issues on the level of individual data collection form a threat to evidence on treatment effect. Therefore, we argue that validity of data is a precondition for evidence. For a sound psychotherapeutic evidence-base we need a proper definition of validity that is sensitive to actual data collection processes, and a theory of evidence that is clear on what should be evidenced to be useful and valid for psychological practice.