Evaluating the inter-rater reliability of the Scale to Assess Unawareness of Mental Disorder using the DOMENIC method

Abstract

Insight in psychosis is a complex construct, conceptualized as a continuous and multidimensional phenomenon. It comprises the awareness of having a mental disorder, treatment compliance, the ability to label unusual mental events as symptoms of the disorder, and the awareness of the social conse­quences of the disorder. Insight has an important impact on the outcome of psychosis and has been related to quality of life, psychosocial functioning, severity of symptomatology, therapeutic compliance, and number of readmissions. Thus, insight assessment is crucial in patients with psychosis. But in real-world clinical settings, a single patient being assessed and/or treated by several clinicians is a common situation, and given the complexity of the phenomenon of insight, it is difficult to ensure that all clinicians are referring to the same construct. Using questionnaires could help clinicians to deal with this situation. The Scale to Assess Unawareness of Mental Disorder (SUMD) is a widely used instrument to explore insight in clinical trials and epidemiological studies that has proved to be adequate in terms of validity and reliability with the usual statistical procedures. Nevertheless, it is not easy to use, and heterogeneity across studies can compromise the results (Dumas et al., 2013). We proposed an inter-rater reliability study to explore differences in the assessment of insight using this instrument. Procedures based on the calculation of Kappa coefficients, weighted Kappa, or interclass corre­lation coefficient (ICC) require a sample of several subjects who are evaluated by a small number of examiners and, for our purpose, this reflects an inefficient strategy. Therefore, we used the Detection of Multiple Examiners Not in Consensus (DOMENIC) method, elaborated by Cicchetti et al. (1997), which allows the inter-rater reliability for one single patient and several raters to be deter­mined. By using the DOMENIC method, we can see that items referring to insight of the negative symptoms of psychosis may cause the greatest difficulty for raters, a difficulty that, according to our data, does not seem to be significantly related to their clinical experience. Retraining clinicians in this area could be the best alternative to improve the reli­ability of the appraisals. Thus, the DOMENIC method could be a useful tool in the preparation phases of a study with this type of scale to easily identify areas of disagreement and investigators who need training to improve inter-rater reliability.