Understanding Internal, External, And Statistical Conclusion Validity In Program Evaluation

Internal Validity

This is a class discussion.  Please respond to Tim by reading his response Question

1. a) Validity is an important aspect of evaluation as the validity of the evaluation contributes to the credibility of the evaluation and the results rendered from it. Working through The Pyramid of Strength as we work through our various levels of validity including Measurement Validity, Measurement Reliability, Internal Validity, External Validity, Statistical Conclusion Validity and Clear Reporting on Choices we see that measurement validity builds a strong base when it comes to being able to produce a clear reporting of choices. As identified in the Pyramid of Strength when you work your way up through the different levels of validity to the top of the pyramid your credibility improves. Validity improves credibility in the results of the evaluation which is very important. “In the end, even careful planning and reasoned decision making about both measurement and design will not ensure that all evaluations will produce perfectly credible results. There are a variety of pitfalls that frequently constrain evaluation ?ndings, as described in Chapter Twenty-Three. But clarity in reporting ?ndings and open discussion about methodological decisions and any obstacles encountered during data collection will bolster con?dence in ?ndings.” (Wholey, 2015) As the saying goes ‘failing to plan is planning to fail’. While taking the proper measures to ensure validity is key there still are potential down sides to any evaluation findings but at least the data obtained through validity can increase the confidence in the results.
2. b) Internal validity as defined is the ability to determine whether a program or intervention has produced an outcome and to also figure out the effect of said outcome. Internal validity essentially is the relationship between the program and the observed effects. An example of internal validity would be evaluating if a pay raise is directly related to someone working more effectively. External Validity is the generalizability of the results. Generalizability is when the findings can be attributed to something outside the realm of what is being studied. Relating to the previous example external validity would assess if the pay raise and possibly increased working effectiveness of an employee affected factors outside of the evaluation such as the persons the position serves. Finally, statistical conclusion validity uses statistically significant findings to relate to the samples used in the evaluation. Going back to the example of pay raises a statistical conclusion validity may be if repeated pay raises led to the continued effectiveness of an employee’s work.

By testing internal validity of an argument, we would like to find out its applicability in terms of the independent and dependent variables identified. ie is there a correlation in the variables for example it would be invalid to test teachers’ qualifications and basket-ball enrollment in an area (Dwork, Feldman,Hardt, Pitassi, Reingold, & Roth, 2015)

External validity seeks to find out whether the argument is logical in that are there any other external factors which are common in all segments of a behavior or population. For example whether the behavior between rural and urban populations is the same. On the other hand statistical conclusion validity will evaluate the validity of the entire model and the conclusions from the research for instance are we saying that increased school enrollment rates lead to higher incomes? (Nuzzo, 2014)

1. What is meant by “internal validity”, “external validity”, and “statistical conclusion validity” as they relate to  program evaluation? Please provide a public administration example to illustrate your point…

Construct validity—whether operational variables adequately represent theoretical constructs.

External validity—whether causal relationships can be generalized to different measures, persons, settings, and times.

Statistical conclusion validity—whether conclusions about statistical inferences of covariation between variables are justified. Statistical conclusion validity also provides a proactive means for improving reliability; moreover, they can be used to design products that are robust to noise factors which are hard or impossible to control. Traditionally, failure- time data have been collected; for high-reliability products, it is unlikely that failures will occur in a reasonable testing period, so the experiment will be uninformative.” (Chiao, C.-H. and Hamada, M. (2001). For an example “Public Health Interventions”, Public Health Interventions are polices that are set in place to progress mental and physical health on a population level. Public Health Interventions include obesity, drug, tobacco, and alcohol use, and the spread of infectious disease, e.g. HIV. If Public Health Interventions decided to do a new study showing depression levels among people who engage in drug, tobacco, and alcohol use during an economic recession, then you will learn that the study is used in a new questionnaire to ask individuals about their mental health over a number of years. Was it valid? Was it reliable?

It is interesting to point out the role of construct validity in statistics is to ensure that the theoretical components of the arguments are covered in a particular model. It therefore has a theoretical base to it hence the need for the section for reliance of theory in a report. On the other hand I agree that external validity provides a measure of the need for causal relationships to be identified in the way that they are. For example, why are rich men considered to be bald (Nuzzo, 2014). I am not sure that statistical conclusion validity was clear in this case. But in the case of public administration, I think that statistical conclusion validity would look at the reasonableness of a statistical conclusion for example if more funds are allocated to a program, does it increase the motivation or performance of the individuals involved (McNabb, 2015).

External Validity

This is a group discussion please Respond to Carval Lopez answer below:

What is meant by “internal validity”, “external validity”, and “statistical conclusion validity” as they relate to   program evaluation? Please provide a public administration example to illustrate your point…

Internal validity refers to the ability of the research design to rule out alternative explanations of the results (Marczyk, DeMatteo & Festinger, 2005). Internal validity may be related with experimental strategy, where the effect of an independent variable on a dependent variable is calculated under strictly controlled settings. Examples of experimental strategies would be medical research or clinical trials. Internal validity helps determine if there is a relationship between a program and an outcome.

External validity determines if the results can be generalized to similar programs. It concerns generalizability of the research conclusions. In other words, if the study is repeated in a different place or with different people, would the results be the same? An example would be a lab test of people’s behavior patterns in response to a particular medication.

Statistical Conclusions is the point to which conclusions about the relationship among variables based on the data are ‘reasonable’. The confidence level of the research findings must be statistically significant (90% – 99%). For example, determining how many years of schooling is needed as a prerequisite.

I agree that internal validity is a measure of the relationship between programs and outcomes.  For example, it would be useful to know if indeed increase in HIV awareness will reduce HIV incidences in a particular area (Dwork, Feldman,Hardt, Pitassi, Reingold, & Roth, 2015).

 In the case of external validity, it could be tested if other people of a population demonstrate characteristics which are thought to be unique to one part of the population. In the case of statistical conclusion validity. There is need to look at whether the conclusion would have been arrived at by mere observation of the phenomena I more taxes led to decrease in consumption (McNabb, 2015). 

References

Wholey, J., Hatry, H., & Newcomer, K. (2015). Handbook of Practical Program Evaluation 4^th Edition. San Francisco, USA. Jossey-Bass/Wiley Imprint

Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. L. (2015, June). Preserving statistical validity in adaptive data analysis. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing (pp. 117-126). ACM.

Nuzzo, R. (2014). Statistical errors: P values, the’gold standard’of statistical validity, are not as reliable as many scientists assume. Nature, 506(7487), 150-153.

Chiao, C.-H. and Hamada, M. (2001), Analyzing experiments with degradation data for improving reliability and for achieving robust reliability. Qual. Reliab. Engng. Int., 17: 333–344. doi:10.1002/qre.412

Nuzzo, R. (2014). Statistical errors: P values, the’gold standard’of statistical validity, are not as reliable as many scientists assume. Nature, 506(7487), 150-153.

McNabb, D. E. (2015). Research methods in public administration and nonprofit management. Routledge.

McNabb, D. E. (2015). Research methods in public administration and nonprofit management. Routledge.

Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. L. (2015, June). Preserving statistical validity in adaptive data analysis. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing (pp. 117-126). ACM.