Validity and reliability of assessment in medical education

Assessments have become a hallmark of the quality of any educational system and with a greater understanding of learning and developments in the field of psychometrics, assessors and test developers have been held accountable for the inferences that are made on the basis of the assessment scores. This has led to validation exercises in assessment and all educational assessors have to consider validity at some point of their work.

Get Help With Your Essay
If you need assistance with writing your essay, our professional essay writing service is here to help!
Essay Writing Service

Determining the validity and reliability of assessments has been the mainstay of the validation exercises. However, in line with the developments in educational psychology and learning theories – over the last sixty years – the concept of validity has broadened1,2,3 and the research question for determining validity has moved from how valid is the instrument, to, is the inference made on the basis of this instrument valid for the group of people for which it is being made and for the purpose of assessment results?
The intent of this paper is to presents the current definition and the sources/aspects of validity and reliability vis-à-vis medical education literature published in peer reviewed journals and text books followed by detailed discussion of one of the aspects of validity evidence namely predictive validity and its utility for the admission tests and processes in medical education. The paper is organized into the following section:

Validity and sources of validity evidence
Threats to validity
Reliability and factors affecting reliability
Predictive Validity of admission tests
Discussion and conclusion

Definition of Validity
According to Vlueten and Schuwirth4 validity refers to whether an instrument actually measures what it is supposed to measure. This means that it is very important for the person developing the assessment instrument to be sure that all items of the instrument are appropriate for the purpose of measurement (assessment). Thus by-and-large the validity of an assessment method would be dependent on the “intrinsic meaning” of the items that make up the instrument which includes the content and the cognitive process that the particular assessment is trying to gauge1,3,4,5.
Downing6 adds to the understanding of the concept of validity by maintaining that validity is not a yes or no decision rather it is the degree to which evidence and theory support the interpretations of test scores for the proposed uses of tests. This lends itself to the need for a theoretical basis for interpreting the results of a test and gives due importance to the process of validating against some theory or hypothesis. Thus validity is not a quality of the instrument in itself but refers to the evidence presented to support or refute the meaning or interpretation assigned to assessment results for the specific group of test takers and the purpose of the test.
Sources of validity evidence:
According to the current understanding of the concept of validity, it requires a chain of evidence to support the interpretations which are made on the basis of the test score1,2. The evidence would help link the interpretation thus made to a theory, hypotheses and logic leading to either accepting or refusing the interpretations. Sources of evidence1 include i) evidence of the content representativeness of the test materials, ii) the response process is the statistical characteristics of the assessment questions, iii) the internal structure of the assessment, iv) correlation of assessment scores to other variable’s score (criterion measure) and v) the consequence of assessment scores for students.
The standards7 recommended to use various sources since strong evidence from one source does not preclude the need to seek evidence from other sources. Some types of assessment demand a stronger emphasis on one or more sources of evidence as opposed to other sources and not all sources of data or evidence are required for all assessments. These sources for the evidence required are briefly discussed below1,7.
1. Evidence for content validity is obtained from test blueprint or test specifications which ideally describe the subcategories and sub-classifications of content and specifies precisely the proportion of test questions in each category and the cognitive level expected to be assessed by those questions. The test specifications are reflective of the emphasis placed on content considering how essential and /or important it is for the level of student being assessed and the desired level of cognitive ability. Therefore while checking for validity evidence the researcher correlates the level of cognitive ability presumably assessed by the questions included in the test with the desired level as specified. The number of questions and their technical appropriateness also provides evidence for content-related validity. Hence, validation by subject experts and quality check by technical experts are both essential for providing evidence of content validity.
2. Evidence regarding the response process is gathered by providing evidence that all sources of error which may be associated with the administration of the test are minimized to the maximum possible. This includes evidence regarding accuracy of response keys, quality control mechanisms of data obtained from the assessments, appropriateness of methods used to obtain a composite score from scores received from different types of assessments and the usefulness and the accuracy of the score reports provided to examinees.
3. Evidence for internal structure is determined by statistical relationship between and among other measures of the same or different but related constructs or traits. The psychometric characteristics required as evidence under this head include difficulty and discrimination indices, reliability and /or generalizability coefficients etc. High reliability coefficients indicate that if the test were to be repeated over time, examinees would receive about the same scores on retesting as they received the first time. This aspect is dealt in greater detail in the section on reliability later in the paper.
4. Evidence regarding relationship of assessment scores to other variable’s (criterion measure’s) scores requires the test to be ‘validated’ against an existing, older measure with well known characteristics that is the extent to which the scores obtained on one test relate to performance on a criterion which is usually another test. The two tests can be administered in the same time period (concurrent validity) or the second may be administered at some future time (predictive validity)8.
Concurrent validity is determined by establishing a relationship (correlation) between the score on new test and the score on an old test (whose validity is already determined) administered in the same time frame. If the correlation coefficient is high that is near +1.0 the test is said to have good concurrent validity. Predictive validity on the other hand is the degree to which a test can predict how well a person will perform on a criterion measure in the future. This criterion measure can be a test for example standardized licensing examination or a performance measure such as patient satisfaction ratings during practice10.
If the tests do not correlate this demonstrates that one test is measuring a specific construct while the other test is measuring another that is they are measuring distinct constructs. This absence of correlation provides evidence of discrimination which is desirable if the two tests are claiming to test discrete constructs, while correlation of scores from two instruments which claim to measure the same construct should correlate with each other providing convergence evidence to support the validity of interpretations of scores from both instruments10.
5. Evidence regarding the impact of assessment on examinees or evidence of consequential validity of the instrument seeks to know the decisions and outcomes made on the basis of assessment score and the impact of assessments on teaching and learning. The consequences of assessments on examinees, faculty, patients and society are enormous and these consequences can be positive or negative, intended or unintended.
Threats to validity evidence
According to Downing9 validity faces two major threats, construct under representation (CU), and construct irrelevant variance (CIV). CU can be due to under-sampling (few questions, few stations, few observations), biased sampling or a mismatch of sample to domain and low reliability of scores, ratings. CIV refers to systematic error introduced by variables unrelated to the construct being measured. Such can happen if the items are flawed, items too easy/too hard/non discriminating/cheating/flawed checklists/ratings scales, variability in the performance of the standardized patient/s (due to poor training), systematic rater error, indefensible passing score, poorly trained assessors.
Reliability: Definition
According to Classical Test Theory (CTT) reliability is defined as the ratio of true score variance to the observed score variance and is represented by reliability coefficients8. In CTT the observed score is a composite of the true score and error. Thus reliability coefficients are used to estimate the amount of measurement error in assessments and is generally expressed as a coefficient ranging from 0 (no reliability) to 1 (perfect reliability). Low reliability means that the error component is large for that assessment and hence results do not hold value. Although higher reliability is always preferable, there is no fixed threshold to discriminate “reliable” from “unreliable” scores. Often 0.80 is regarded as the minimal acceptable value, although it may be lower or higher depending on the examination’s purpose. Reliability can be negatively affected by many sources of error or bias, however, adequate sampling ensures taking account of the undesirable sources of variance and increases reliability10.
A predominant condition which affects reliability of assessment is domain- or content-specificity, since competence is shown to be highly dependent on the context and content6,7. In the light of these findings reliable scores can be achieved only if the content of the subject (to be tested) is largely sampled. This has led to the assessments in medical education moving away from open ended essay questions, long cases and limited number of short cases to multiple choice questions, objective structured clinical examinations and multiple assessments of clinical performance since all of these provide opportunities of assessing students on a larger sample of test items compared to. The amount of time spent on assessment also influences reliability since larger samples of performance can be gathered. The other factors which effect reliability are involvement of larger number of examiners and (standardized or real) patients which increase the chances of variability from student-to-student and hence affect the reliability of such assessments. Examiner training, increasing use of trained standardized patients and sampling across different health conditions are steps taken to improve reliability of scores in assessment of medical students at both undergraduate and postgraduate levels. Recent studies have demonstrated that sampling is the main factor in achieving reliable scores with any instrument10.
Types of reliability
There are many types of reliability estimates and it is the specific purpose of the assessment that dictates the type of reliability estimate which is of greatest importance. The different types of reliability estimates include8,10,11
test retest – assuming that a test is testing a single construct, if the test is split into two halves, the items on one half should correlate with the other half (this only gives the reliability for half of the test and spearman brown’s prophecy formula has to be applied to get the reliability of the entire test.
internal consistency – estimates the reliability from all possible ways by splitting the test into 2 halves: [this is Cronbach’s alpha coefficient, which can be used with polytomous data (0, 1, 2, 3, 4,…n) and is the more general form of the KR 20 coefficient, which can be used only with dichotomously scored items (0, 1), such as typically found on selected-response tests.]
Inter rater reliability determined by using kappa statistics which account for the random-chance occurrence of rater agreement and is therefore sometimes used as an interrater reliability estimate, particularly for individual questions, rated by 2 independent raters.
Generalizability coefficient -GT can estimate variance components for all the variables of interest in the design: the persons, the raters and the items.
Issues of validity and reliability with respect to medical education
Schuwirth & Vleuten5 in a critical analysis of validity and reliability are of the view that although the psychometric paradigm of viewing assessment has provided tools such as reliability and validity to ensure and improve the quality of assessment, it is of limited value in the light of current developments in assessment. An important consequence of the shift in the perspective on reliability (increased sampling more important than standardization) is that there is no need for us to exclude from our assessment methods, instruments that are rather more subjective or not perfectly standardized, provided that we use those instruments sensibly and expertly. This has resulted in a change in the way we think about assessment in medical education and in quest of using instruments for assessment that are structured and standardized which took us away from real life settings into construed environments such as OSCE we are now moving back into assessment methods which are more authentic though less structured and standardized provided adequate sampling is done ensuring reliability of measurements. This view is becoming quite popular with assessors since it is lends more credibility to work-based or practice assessments tools which may not be highly standardized but are much more authentic. A summary of important points to be considered while assessing instruments for validity evidence and reliability estimates is given below:

Validity is based on a theory or hypothesis and all sources of validity evidence contribute to accepting or rejecting the hypothesis.
Validity is a property of scores and scores interpretations and not a property of the instrument itself.
Broader variety of validity evidence should be sought with greater attention to the categories of relation to other variables, consequences and response process.
Instruments using multiple observers should report inter rater reliability.

Predictive validity of admissions tests in medical education
The main purpose for conducting selection tests is to choose from a pool of applicants those who are most suitable for the course of study or for practicing the profession. In medical education this means that the entrants selected for admission to medical school or residency programs demonstrate a readiness for medical education programs and have the right kind of characteristics, assuming that students selected will stay and not leave the program and on graduation will practice medicine with professionalism. Thus national and institutional examining bodies in medical education responsible for developing and conducting admission tests have to demonstrate the predictive value of their examinations to the society. And therefore require selection criteria that are evidence-based and legally defensible. The variables that are generally investigated during the admission process include cognitive abilities (knowledge), skills and non cognitive characteristics (personal attributes). Assessment of knowledge at the entry level in medical schools has been used in many countries since a long time.
While reviewing English language literature for studies on validity evidence of selection tests the largest numbers of studies available are from North America especially from USA which has more than eighty years of history of centralized admission test in medical education. Few studies are also reported from United Kingdom and Australia. Three studies were found from South Asia. Both in United States of America (USA) and Canada admissions and licensing examinations have been extensively studied for their ability to predict performance during medical school, in licensing examinations, during residency education and specialty (Board) examinations.
The components of knowledge and skills tested differ along the continuum of medical education with undergraduate grade point average (UGPA) and medical college admission test (MCAT) scores being used for selection in medical schools while the United States Medical Licensing Examinations (USMLE) for foreign medical graduates and the National Board of Medical Examiners (NBME) taken by graduates of US medical schools scores, medical school GPA and performance scores on assessment during the medical school years used for selection to residency programs. I will review relevant studies under separate headings for medical schools and residency programs.
Studies on medical school admission tests:
Predictive validity of tests of Cognitive ability
Basco12 studied the contribution of undergraduate institutional measure to predicting basic science achievement in medical school. The undergraduate institutional measure was calculated by averaging MCAT scores attained by all students of an institution from between 1996 – 1999. the researcher found moderate correlation between Undergraduate science GPA and individual MCAT scores and between SciGPA and USMLE step 1 scores. Correlation between individual MCAT scores and USMLE step 1 was higher than that between institutional MCAT score. Jones et al13 studying the predictive validity of MCAT have reported that MCAT scores have significant predictive validity for first and second year medical school course grades and NBME part 1 examination scores. Swanson et al14 studied the predictive validity of the previous and current MCAT for USMLE Step 1 and did not find much difference between the two forms. Vancouver et al15 examined the use of MCAT scores and undergraduate GPA for predictive validity and differential predictions based on ethnic groups using NBME part 1 as a measure of medical students performance. They found that using the science GPA and composite MCAT scores were equally predictive for the minority and majority groups studied.
Violato and Donnon16 studying the predictive ability of MCAT for clinical reasoning skills reported evidence of predictive validity for performance on Part 1 of Medical Council of Canada Examination (MCCE). The verbal reasoning subset of MCAT was positively correlated with MCCE part 2. This demonstrates that items testing similar constructs have a positive correlation (convergent validity evidence).
Peskun et al17 assessed the predictive validity of medical school application components by estimating association between the components of the admission process and the ranking of students by residency programs. They found that residency rank in internal medicine was correlated significantly with GPA and non cognitive assessment while residency rank in family medicine (FM) was correlated significantly with the admissions interview and there was a trend towards significance between non cognitive assessment and FM ranking. However, there was no relationship between GPA, MCAT and FM ranking. OSCE score was correlated significantly with non cognitive assessment of admission predictor variable. Final grade in med school was correlated significantly with GPA, MCAT and non cognitive assessment of admission variable.
Residency ranking in IM was correlated significantly with OSCE, IM clerkship final grade and final grade in med school. Ranking in FM was correlated significantly with OSCE score, IM clerkship ward evaluation, FM clerkship final grade and final grade in med school.
A number of studies have reported reviews of published reports of MCAT. Mitchell et al18 have reported on studies published from 1980 – 1987 using many predictors such as total as well as science and non science subjects undergraduate GPA (uGPA), MCAT scores and institutional quality. They found that uGPA and MCAT scores predict performance in basic sciences exams and performance in earlier years of medical school. Donnon et al19 in a meta analysis of all published data of the predictive validity of post 1991 version of MCAT and its subtest domains determined the validity coefficients for performance during medical school and on medical board licensing examinations. They found that the MCAT total has medium predictive validity coefficient effect size for basic science/pre clinical (r = 0.43) and clerkship/clinical. The biological science subtest has the highest predictive validity for both basic science /preclinical & clinical/ years of the med school performance while the MCAT total has a large predictive validity coefficient for USMLE Step 1 and a medium validity coefficient for USMLE step 2. The writing sample subtest had little predictive validity for both the medical school performance and the licensing exam. Hojat et al20 have also studied the relationship between the writing sample subtest and the measures of performance during medical school and USMLE step 1. They did not find any differences amongst the high, medium and low scorers in the written test with respect to MCAT or USMLE scores. However they reported positive correlations with undergraduate non science and MCAT verbal reasoning scores of the three groups as well as in written clerkship exams, and global ratings of clinical competence and ratings of interpersonal skills. Thus it shows that although the written scores do not correlate with MCQ type of knowledge based tests they may be assessing other constructs useful in clinical practice. Andriole et al21 studied independent predictors of USMLE Step 3 performance among a cohort of U.S. medical school graduates. They analyzed Step 3 scores in association with four measures of academic achievement during medical school, including first-attempt USMLE Step 1 and Step 2 scores, third-year clinical clerkships’ grade point average (GPA), and Alpha Omega Alpha (AOA) election. They found higher third year clerkships’ GPA, higher Step 2 scores, and choosing residency training in broad-based specialties being associated with higher Step 3 scores. However they did not report on the reliability estimates for the continuous assessment forms used for clerkships in their study which is required for making inferences.
Two studies were found from UK which looked into predictive validity of medical school admission criteria. McManus22 reporting on use of A level grades for admission in medical schools has shown that they are predictive of performance in basic medical science, final clinical science as well as for part I of a postgraduate examination. He reported that use of intellectual aptitude tests as predictor of academic performance did not demonstrate any predictive validity. Yates and James23 in a retrospective study design looked into the academic records of students who struggled in the medical school. They found that negative comments in the head teachers reference letters were the only indictor for strugglers.
Two studies were found reporting on predictive validty of admission tests from Karachi, Pakistan. The study by Baig et al24 showed that the admission test scores had significantly positive weak correlation with second (p = 0.009) and third (p = 0.003) professional scores of the medical students. When the scores of High School were combined with the admission test scores, the predictive validity increased for first (p = 0.031) second (p = 0.032) and third (p = 0.011) professional examinations. Another study from Aga Khan University reported that performance on admission test is a better predictor of performance on medical college examinations that interviews25.
deSilva et al26 assessed the extent to which selection criteria used for admission in Sri Lankan medical schools predicted success later on and found that being a female and having a higher aggregate score were the only independent predictors of success for performance in medical school while A level scores which were used as the only criteria for admission had no correlation with performance in medical school.
A study by Coates27 reports the predictive validity of Graduate Medical School Admission Test (GAMSAT) which is used for admission to medical school in Australia and recently have been in UK and Ireland. They found that GAMSAT, interview and GPA showed divergent relationships, while combination of GAMSAT and GPA scores provided the best means of predicting year 1 performance.
Predictive validity of non cognitive assessment
The non cognitive (personal) characteristics are predominantly assessed through interviews, personal statements letter of support from the Head of institution studied. Albanese et al28 in a review of published literature reporting on means to effectively measure personal qualities discussed the challenges in using interviews to assess personal qualities and have come up with recommendations for an approach for assessing these. They have provided evidence that interviews provide information for admission related to students’ performance in the clinical component of medical education. They have concluded that interview ratings can discriminate between students who fail to complete medical school and those who complete as well as between those who graduate with honors and those who do not.
Eva and colleagues29 have discussed the role of multiple mini interviews (MMI) in assessing non cognitive attributes. MMI were developed to improve the subjective assessment on traditional interviews and are being studied for their predictive validity.
Skills that are assessed and whose scores are used for selection include the communication skills and self directed learning skills for admission in medical school while for residency selection a more specific set of skills coming under the domain of clinical competence are assessed. These skills in addition to communication skills include history taking, physical examination etc28.
Predictive validity of assessment in graduate medical education
Not many studies could be retrieved which discuss the predictive validity of selection processes for graduate medical education. Patterson et al30 evaluated three short listing methodologies for their effectiveness and efficacy for selection into postgraduate training in general practice in UK. They reported that clinical problem solving tests along with a newly developed situational judgment test which assessed non cognitive domains were effective in predicting performance at the selection center test which used work-relevant simulations that have been already validated.
Althouse et al31 have reported on the predictive validity of in-training evaluation (ITE) of residents for passing the General Pediatrics certification examination. They found that the predictive validity of ITE increased with each year of training being the least in year one and maximum in year three.
Discussion
Assessment methods in medical education have evolved over the last many years with increasing understanding of the underlying constructs and development of sophisticated psychological tests leading to more sophisticated techniques being used at entry, during and exit levels of medical education.
The medical school admission tests in USA and Canada have been most extensively studied. The MCAT has undergone four major modifications over the years, all of which have been researched and reported32. However, most of the studies conducted to determine the predictive value of admission tests for performance in medical school, during internship and residency education do not provide information on issues like content, consequential and construct validity in particular. Scores used for student selection have been used as predictors and performance in medical school, licensing examination or during residency education as outcomes. Two types of designs have been used while studying predictive validity; prospective studies which look at the performance of medical students on medical school examinations, licensing examinations, specialty board examination or health outcomes and retrospective studies analyzing the correlation between outcomes and predictor variables.
The conceptual framework used by the Best Evidence in Medical Education (BEME) group to study the predictive validity of assessment in medical education helps in critical evaluation of this literature. The results of the BEME systematic review indicated that analysis of the performance after graduating from the medical school is complex and cannot be measured by one type of measurement. Since clinical competence is a multifaceted entity, and the strength of relationships with medical school performance measure varies depending upon conceptual relevance of the measures taken during and after medical school. This is evident in the studies referred to earlier as we see that the preclinical GPAs yields more overlap with physicians’ medical knowledge than with physicians’ interpersonal skills33.
Consideration of aspects of validity to evaluate tests for selection of applicants for medical schools
1. Content related evidence:
McGaghie32 in a detailed overview of the MCAT from 1926 to date providing the details of the subtest categories and question types of MCAT over the years states that the definition of aptitude for medicine is what has driven the content of the MCAT. In the early years of its use from 1928 to 1946 the content was mostly dominated by biomedical knowledge and intellectual qualities assumed to be needed to succeed in medical education at that time. However we see that the content underwent revisions based on advancement of educational measurement and technology and a modified understanding of aptitude needed for medical education which in the era from 1946-1962 consisted of reduction in the subtests and inclusion of understanding modern society. This was the first time that it was felt that medical students also need to have an understanding of what is going on around them. This realization was made clearer when the 1962-1977 MCAT introduced a section on general knowledge in place of understanding modern society. The 1977-1991 evolution of MCAT resulted in discarding the general liberal arts and knowledge as a separate section however reading skills and quantitative skills were included. The latest version of MCAT which changed in 1991 does not measure liberal arts achievement or numeracy, but requires the student to write a free response essay on a current topic while the verbal reasoning section presents short comprehension passages from humanities, social and natural sciences followed by multiple choice questions.
However the few other selection tests pay a great emphasis on knowledge of biomedical subjects and quantitative skills. The admission test of AKU has biology, chemistry, physics and mathematics questions25.
The non cognitive portion of the admission process is improving now with a better understanding of the outcomes expected from a medical graduate. Interviews, personal essays, letter of reference and evidence of participation in community work are based on the content of non cognitive attributes or traits which include compassion, empathy, altruism.