Background The Korean translated AGREE II instrument has been distrubuted to Korean medical societies in 2011. But inter-rater disagreement issues still existed. The Korean AGREE II scoring guide was therefore developed to reduce inter-rater differences. Objectives To examine effects of the Korean AGREE II scoring guide on reduction of the inter-rater differences. Methods Appraisers were randomly assigned to group A or B. The Korean AGREE II instument was provided to all participants, but the Korea AGREE II scoring guide was offered to only group A. Total of 14 appraisers participated and each gudeline was assessed by 8 appraisers. To evaluate the reliability of the Korean AGREE II scoring guide, correlation of scores among appraisers and domain-specific intra-class correlation (ICC) were calculated. Results Most scores of two groups were comparable. And scoring guide user group showed higher reliability at all guidelines. It showed higher correlation among appraisers and higher ICC values at almost all domains (from 0.272∼0.792 to 0.667∼0.888 in CPG A, from 0.273∼0.682 to 0.424∼0.815 in CPG B). Discussion Scoring guide reduces the inter-rater disagreement and improves the overall reliability of the Korean-AGREE II instrument. Implications for Guideline Developers/Users This study identified the performance and utility of the scoring guide as a useful guidance to Korean-AGREE II instrument users.
M. Oh, H. Jo, You Kyoung Lee
BMJ Quality & Safety