- Evaluation approaches
Evaluation approaches are conceptually distinct ways of thinking about, designing and conducting
evaluationefforts. Many of the evaluation approaches in use today make unique contributions to solving important problems, while others refine existing approaches in some way. Classification systems intended to sort out unique approaches from variations on a theme are presented here to help identify some basic schools of thought for conducting an evaluation. After these approaches are identified, they are summarized in terms of a few important attributes.
Since the mid 1960s, the number of alternative approaches to conducting evaluation efforts has increased dramatically. Factors such as the United States
Elementary and Secondary Education Actof 1965 that required educators to evaluate their efforts and results, and the growing public concern for accountability of human service programs contributed to this growth. In addition, over this period of time there has been an international movement towards encouraging evidence based practicein all professions and in all sectors. Evidence Based Practice (EBP) requires evaluations to deliver the information needed to determine what is the best way of achieving results.
Classification of approaches
Two classifications of evaluation approaches by House ref|House and Stufflebeam & Webster ref|Stufflebeam were combined by Frisbie ref|Frisbie into a manageable number of approaches in terms of their unique and important underlying principles. The general structures of these classification systems are discussed first. The structures then are combined to present a more detailed classification of fifteen evaluation approaches.
House considers all major evaluation approaches to be based on a common
ideology, liberal democracy. Important principles of this ideology include freedom of choice, the uniqueness of the individual, and empirical inquirygrounded in objectivity. He also contends they all are based on subjectivist ethics, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is utilitarian, in which “the good” is determined by what maximizes some single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is intuitionist / pluralist, in which no single interpretation of “the good” is assumed and these interpretations need not be explicitly stated nor justified.
These ethical positions have corresponding epistemologies—philosophies of obtaining
knowledge. The objectivist epistemology is associated with the utilitarian ethic. In general, it is used to acquire knowledge capable of external verification (intersubjective agreement) through publicly inspectable methods and data. The subjectivist epistemology is associated with the intuitionist/pluralist ethic. It is used to acquire new knowledge based on existing personal knowledge and experiences that are (explicit) or are not (tacit) available for publicinspection.
House further divides each epistemological approach by two main political perspectives. Approaches can take an
eliteperspective, focusing on the interests of managers and professionals. They also can take a mass perspective, focusing on consumers and participatory approaches.
Stufflebeam and Webster place approaches into one of three groups according to their orientation toward the role of values, an ethical consideration. The political orientation promotes a positive or negative view of an object regardless of what its value might actually be. They call this pseudo-evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object. They call this quasi-evaluation. The values orientation includes approaches primarily intended to determine the value of some object. They call this true evaluation.
Table 1 is used to classify fifteen evaluation approaches in terms of epistemology, major perspective (from House), and orientation (from Stufflebeam & Webster). When considered simultaneously, these three dimensions produce twelve cells. Only seven of the cells contain approaches, although all four true evaluation cells contain at least one approach.
Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective.
Six quasi-evaluation approaches use an objectivist epistemology. Five of them—experimental research, management information systems, testing programs, objectives-based studies, and content analysis—take an elite perspective. Accountability takes a mass perspective.
Seven true evaluation approaches are included. Two approaches, decision-oriented and policy studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective. Two approaches—accreditation/certification and connoisseur studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and client-centered studies are based on a subjectivist epistemology from a mass perspective.
ummary of approaches
The preceding section was used to distinguish between fifteen evaluation approaches in terms of their epistemology, major perspective, and orientation to values. This section is used to summarize each of the fifteen approaches in enough detail so that those placed in the same cell of Table 1 can be distinguished from each other.
Table 2 is used to summarize each approach in terms of four attributes—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The
purposerepresents the desired outcome for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped into the same cell of Table 1.
Politically controlled and public relations studies are based on an objectivist epistemology from an elite perspective. Although both of these approaches seek to misrepresent value interpretations about some object, they go about it a bit differently.
Informationobtained through politically controlled studies is released or withheld to meet the special interests of the holder. Public relationsinformation is used to paint a positive image of an object regardless of the actual situation. Neither of these approaches is acceptable evaluation practice, although the seasoned reader can surely think of a few examples where they have been used.
Objectivist, elite, quasi-evaluation
As a group, these five approaches represent a highly respected collection of disciplined inquiry approaches. They are considered quasi-evaluation approaches because particular studies can legitimately focus only on questions of knowledge without addressing any questions of value. Such studies are, by definition, not evaluations. These approaches can produce characterizations without producing appraisals, although specific studies can produce both. Each of these approaches serves its intended purpose well. They are discussed roughly in order of the extent to which they approach the objectivist ideal.
Experimental research is the best approach for determining causal relationships between
variables. The potential problem with using this as an evaluation approach is that its highly controlled and stylized methodologymay not be sufficiently responsive to the dynamically changing needs of most human service programs. Management information systems (MISs) can give detailed information about the dynamic operations of complex programs. However, this information is restricted to readily quantifiable data usually available at regular intervals.
Testing programs are familiar to just about anyone who has attended school, served in the military, or worked for a large company. These programs are good at comparing individuals or groups to selected norms in a number of subject areas or to a set of standards of performance. However, they only focus on testee performance and they might not adequately sample what is taught or expected.
Objectives-based approaches relate outcomes to prespecified objectives, allowing judgments to be made about their level of attainment. Unfortunately, the objectives are often not proven to be important or they focus on outcomes too narrow to provide the basis for determining the value of an object.
Content analysisis a quasi-evaluation approach because content analysis judgments need not be based on value statements. Instead, they can be based on knowledge. Such content analyses are not evaluations. On the other hand, when content analysis judgments are based on values, such studies are evaluations.
Objectivist, mass, quasi-evaluation
Accountabilityis popular with constituents because it is intended to provide an accurate accounting of results that can improve the qualityof products and services. However, this approach quickly can turn practitioners and consumers into adversaries when implemented in a heavy-handed fashion.
Objectivist, elite, true evaluation
Decision-oriented studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close collaborationbetween an evaluator and decision-maker, allowing it to be susceptible to corruption and bias.
Policy studies provide general guidance and direction on broad issues by identifying and assessing potential costs and benefits of competing policies. The drawback is these studies can be corrupted or subverted by the politically motivated actions of the participants.
Objectivist, mass, true evaluation
Consumer-oriented studies are used to judge the relative merits of goods and services based on generalized needs and values, along with a comprehensive range of effects. However, this approach does not necessarily help practitioners improve their work, and it requires a very good and credible evaluator to do it well.
ubjectivist, elite, true evaluation
Accreditation / certification programs are based on self-study and peer review of organizations, programs, and personnel. They draw on the insights, experience, and expertise of qualified individuals who use established guidelines to determine if the applicant should be approved to perform specified functions. However, unless performance-based standards are used, attributes of applicants and the processes they perform often are overemphasized in relation to measures of outcomes or effects.
Connoisseurstudies use the highly refined skills of individuals intimately familiar with the subject of the evaluation to critically characterize and appraise it. This approach can help others see programs in a new light, but it is difficult to find a qualified and unbiased connoisseur.
ubjectivist, mass, true evaluation
The adversary approach focuses on drawing out the pros and cons of controversial issues through quasi-legal proceedings. This helps ensure a balanced presentation of different perspectives on the issues, but it is also likely to discourage later cooperation and heighten animosities between contesting parties if “winners” and “losers” emerge.
Client-centered studies address specific concerns and issues of practitioners and other clients of the study in a particular setting. These studies help people understand the activities and values involved from a variety of perspectives. However, this responsive approach can lead to low external
credibilityand a favorable bias toward those who participated in the study.
Notes and references
# House, E. R. (1978). Assumptions underlying evaluation models. "Educational Researcher". 7(3), 4-12.
# Stufflebeam, D. L., & Webster, W. J. (1980). An analysis of alternative approaches to evaluation. "Educational Evaluation and Policy Analysis". 2(3), 5-19.
# Frisbie, R. D. (1986). [http://web.ics.purdue.edu/~rfrisbie/Professional/index.htm#ContentAnalysisEvaluation The use of microcomputer programs to improve the reliability and validity of content analysis in evaluation.] Doctoral dissertation, Western Michigan University.
Wikimedia Foundation. 2010.
Look at other dictionaries:
Evaluation — is systematic determination of merit, worth, and significance of something or someone using criteria against a set of standards. Evaluation often is used to characterize and appraise subjects of interest in a wide range of human enterprises,… … Wikipedia
Evaluation (disambiguation) — Evaluation is the process of characterizing and appraising something of interest or of determining the value of an expression (mathematics). Computer science * determining the value of an expression (programming) * Eager evaluation or strict… … Wikipedia
Evaluation of machine translation — Various methods for the evaluation for machine translation have been employed. This article will focus on the evaluation of the output of machine translation, rather than on performance or usability evaluation.Before covering the large scale… … Wikipedia
Re-evaluation Counseling — Re evaluation Counseling, or RC is the best known and largest organization for Co Counseling.Fact|date=October 2007 RC today spans over 40 countries and offers many individuals an inexpensive or largely free form of counseling and personal… … Wikipedia
Impact evaluation — assesses the changes in the well being of individuals that can be attributed to a particular intervention, such as a project, program or policy. [ [http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/EXTISPMA/0,,menuPK:384336 pagePK:149018 … Wikipedia
Fire Safety Evaluation System — The Fire Safety Evaluation System (FSES) is a system used in the United States to evaluate the overall level of a building s fire safety. FSES applies to health care, prisons and jails, offices, laboratory buildings, and overnight accommodations… … Wikipedia
Web Accessibility Initiative — The World Wide Web Consortium (W3C) s Web Accessibility Initiative (WAI) is an effort to improve the accessibility of the World Wide Web (WWW or Web) for people with disabilities. People with disabilities may encounter difficulties when using… … Wikipedia
Evidence-based practice — The term evidence based practice (EBP) refers to preferential use of mental and behavioral health interventions for which systematic empirical research has provided evidence of statistically significant effectiveness as treatments for specific… … Wikipedia
Activity recognition — aims to recognize the actions and goals of one or more agents from a series of observations on the agents actions and the environmental conditions. Since the 1980s, this research field has captured the attention of several computer science… … Wikipedia
Information technology controls — In business and accounting, Information technology controls (or IT controls) are specific activities performed by persons or systems designed to ensure that business objectives are met. They are a subset of an enterprise s internal control. IT… … Wikipedia