Intra- and Inter-Rater Reliability and Validation of Orofacial Cleft Classification in the Cleft Collective Cohort Study

Objective: Assess the level of agreement for classification of orofacial clefts within and between different raters and validate classifications. Design: Validation study. Setting: National longitudinal prospective cohort, United Kingdom. Participants: Children born with orofacial cleft (n=4211), recruited to the Cleft Collective between 2013 and 2024. Main outcome measures: Four cleft classifications of orofacial clefts were explored. Classification one comprised cleft lip, cleft palate, and cleft lip and palate. Classification two added laterality, classification three added sidedness and classification four added completeness using LAHSAL. Data on orofacial cleft classification were collected through seven different sources. Results: At least one report of cleft classification was available for 4052 Cleft Collective study children. When assessing intra-rater agreement mothers had the highest level of agreement for the simplest form of cleft classification with a Krippendorf's Alpha of 0.987. When recording LAHSAL for the same child, surgeons reported the same classification for 71% of children (Krippendorf's Alpha = 0.672). When assessing inter-rater agreement across different sources, the simplest cleft classification resulted in the highest level of agreement (Krippendorf's Alpha = 0.957) and the least agreement when using LAHSAL (Krippendorf's Alpha = 0.538). Conclusion: Our study found that the more complex a cleft classification becomes the less agreement there is between sources. Differences across sources became most apparent when reporting the sidedness or completeness of a cleft. Validation of cleft classification is important for both clinical care and research. Although LAHSAL is advocated for use both clinically and in research our data show that rigorous training is essential.