A five-stage process for establishing contextually valid systematic observation instruments: The case of Rugby Union

The purpose of this paper is to propose a five-stage process for establishing both validity and reliability in new systematic observation instruments. The process is contextualized within the working behaviors of elite level rugby union coaches within the practice setting. The sequential stages began with observer training and progressed through the identification of coaching behaviors through induction (to establish content validity), to establishing face validity through a domain-referenced test. The objectivity and reliability of the developed behavioral classifications are determined through an inter-observer agreement test while, finally, the researcher's ability to reliably re-produce data with the developed instrument is determined using a test/retest intraobserver reliability check. The developed instrument (the Rugby Union Coaches Observation Instrument: RUCOI) is deemed able to record the situationally unique behaviors arising from the nature of the sport and of the elite standard, both of which were considered to impinge upon the pedagogical process in the said context.