definition of evaluation by different authors

The exploitation of research to provide impact occurs through a complex variety of processes, individuals, and organizations, and therefore, attributing the contribution made by a specific individual, piece of research, funding, strategy, or organization to an impact is not straight forward. (2006) on the impact arising from health research. One might consider that by funding excellent research, impacts (including those that are unforeseen) will follow, and traditionally, assessment of university research focused on academic quality and productivity. While looking forward, we will be able to reduce this problem in the future, identifying, capturing, and storing the evidence in such a way that it can be used in the decades to come is a difficulty that we will need to tackle. Muffat says - "Evaluation is a continuous process and is concerned with than the formal academic achievement of pupils. Published by Oxford University Press. Worth refers to extrinsic value to those outside the . The justification for a university is that it preserves the connection between knowledge and the zest of life, by uniting the young and the old in the imaginative consideration of learning. Any information on the context of the data will be valuable to understanding the degree to which impact has taken place. Standard approaches actively used in programme evaluation such as surveys, case studies, bibliometrics, econometrics and statistical analyses, content analysis, and expert judgment are each considered by some (Vonortas and Link, 2012) to have shortcomings when used to measure impacts. The REF will therefore assess three aspects of research: Research impact is assessed in two formats, first, through an impact template that describes the approach to enabling impact within a unit of assessment, and second, using impact case studies that describe the impact taking place following excellent research within a unit of assessment (REF2014 2011a). Definition of evaluation. This report, prepared by one of the evaluation team members (Richard Flaman), presents a non-exhaustive review definitions of primarily decentralization, and to a lesser extent decentralization as linked to local governance. A discussion on the benefits and drawbacks of a range of evaluation tools (bibliometrics, economic rate of return, peer review, case study, logic modelling, and benchmarking) can be found in the article by Grant (2006). 0000004019 00000 n Aspects of impact, such as value of Intellectual Property, are currently recorded by universities in the UK through their Higher Education Business and Community Interaction Survey return to Higher Education Statistics Agency; however, as with other public and charitable sector organizations, showcasing impact is an important part of attracting and retaining donors and support (Kelly and McNicoll 2011). The Value of Public Sector R&D, Assessing impacts of higher education systems, National Co-ordinating Centre for Public Engagement, Through a Glass, Darkly: Measuring the Social Value of Universities, Describing the Impact of Health Research: A Research Impact Framework, LSE Public Policy Group. Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. To understand the socio-economic value of research and subsequently inform funding decisions. Table 1 summarizes some of the advantages and disadvantages of the case study approach. As such research outputs, for example, knowledge generated and publications, can be translated into outcomes, for example, new products and services, and impacts or added value (Duryea et al. We will focus attention towards generating results that enable boxes to be ticked rather than delivering real value for money and innovative research. The Oxford English Dictionary defines impact as a 'Marked effect or influence', this is clearly a very broad definition. 0000348060 00000 n The basic purpose of both measurement assessment and evaluation is to determine the needs of all the learners. Again the objective and perspective of the individuals and organizations assessing impact will be key to understanding how temporal and dissipated impact will be valued in comparison with longer-term impact. 0000012122 00000 n At least, this is the function which it should perform for society. They risk being monetized or converted into a lowest common denominator in an attempt to compare the cost of a new theatre against that of a hospital. Collecting this type of evidence is time-consuming, and again, it can be difficult to gather the required evidence retrospectively when, for example, the appropriate user group might have dispersed. n.d.). And also that people who are recognized as authors, understand their responsibility and accountability for what is being published. An alternative approach was suggested for the RQF in Australia, where it was proposed that types of impact be compared rather than impact from specific disciplines. Other approaches to impact evaluation such as contribution analysis, process tracing, qualitative comparative analysis, and theory-based evaluation designs (e.g., Stern, Stame, Mayne, Forss, & Befani, 2012) do not necessarily employ explicit counterfactual logic for causal inference and do not introduce observation-based definitions. Perhaps, SROI indicates the desire to be able to demonstrate the monetary value of investment and impact by some organizations. n.d.). New Directions for Evaluation, Impact is a Strong Weapon for Making an Evidence-Based Case Study for Enhanced Research Support but a State-of-the-Art Approach to Measurement is Needed, The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance, Evaluation in National Research Funding Agencies: Approaches, Experiences and Case Studies, Methodologies for Assessing and Evidencing Research Impact. There has been a drive from the UK government through Higher Education Funding Council for England (HEFCE) and the Research Councils (HM Treasury 2004) to account for the spending of public money by demonstrating the value of research to tax payers, voters, and the public in terms of socio-economic benefits (European Science Foundation 2009), in effect, justifying this expenditure (Davies Nutley, and Walter 2005; Hanney and Gonzlez-Block 2011). Attempting to evaluate impact to justify expenditure, showcase our work, and inform future funding decisions will only prove to be a valuable use of time and resources if we can take measures to ensure that assessment attempts will not ultimately have a negative influence on the impact of our research. This database of evidence needs to establish both where impact can be directly attributed to a piece of research as well as various contributions to impact made during the pathway. The authors propose a new definition for measurement process based on the identification of the type of measurand and other metrological elements at each measurement process identified. Wooding et al. Clearly there is the possibility that the potential new drug will fail at any one of these phases but each phase can be classed as an interim impact of the original discovery work on route to the delivery of health benefits, but the time at which an impact assessment takes place will influence the degree of impact that has taken place. In 200910, the REF team conducted a pilot study for the REF involving 29 institutions, submitting case studies to one of five units of assessment (in clinical medicine, physics, earth systems and environmental sciences, social work and social policy, and English language and literature) (REF2014 2010). Researchers were asked to evidence the economic, societal, environmental, and cultural impact of their research within broad categories, which were then verified by an expert panel (Duryea et al. different things to different people, and it is primarily a function of the application, as will be seen in the following. Definition of Evaluation "Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education, as part of a recognised process of judging its effectiveness, its efficiency and any other outcomes it may have." Mary Thorpe 2. It is concerned with both the evaluation of achievement and its enhancement. The introduction of impact assessments with the requirement to collate evidence retrospectively poses difficulties because evidence, measurements, and baselines have, in many cases, not been collected and may no longer be available. Evaluation of impact in terms of reach and significance allows all disciplines of research and types of impact to be assessed side-by-side (Scoble et al. Many theorists, authors, research scholars, and practitioners have defined performance appraisal in a wide variety of ways. Here we outline a few of the most notable models that demonstrate the contrast in approaches available. One way in which change of opinion and user perceptions can be evidenced is by gathering of stakeholder and user testimonies or undertaking surveys. Impact has become the term of choice in the UK for research influence beyond academia. Measurement assessment and evaluation helps the teachers to determine the learning progress of the students. These techniques have the potential to provide a transformation in data capture and impact assessment (Jones and Grant 2013). There are standardized tests involved in the process of measurement assessment and evaluation to enable the students to make better use of the data available in the daily classroom. % 0000342980 00000 n 4. One notable definition is provided by Scriven (1991) and later adopted by the American Evaluation Association (): "Evaluation is the systematic process to determine merit, worth, value, or . 0000002109 00000 n It is perhaps assumed here that a positive or beneficial effect will be considered as an impact but what about changes that are perceived to be negative? The university imparts information, but it imparts it imaginatively. Describe and use several methods for finding previous research on a particular research idea or question. As Donovan (2011) comments, Impact is a strong weapon for making an evidence based case to governments for enhanced research support. Classroom Assessment -- (sometime referred to as Course-based Assessment) - is a process of gathering data on student learning during the educational experience, designed to help the instructor determine which concepts or skills the students are not learning well, so that steps may be taken to improve the students' learning while the course is 0000004731 00000 n Impact is often the culmination of work within spanning research communities (Duryea et al. In undertaking excellent research, we anticipate that great things will come and as such one of the fundamental reasons for undertaking research is that we will generate and transform knowledge that will benefit society as a whole. The advantages and disadvantages of the case study approach. From an international perspective, this represents a step change in the comprehensive nature to which impact will be assessed within universities and research institutes, incorporating impact from across all research disciplines. Despite many attempts to replace it, no alternative definition has . 2009), and differentiating between the various major and minor contributions that lead to impact is a significant challenge. In many instances, controls are not feasible as we cannot look at what impact would have occurred if a piece of research had not taken place; however, indications of the picture before and after impact are valuable and worth collecting for impact that can be predicted. 0000008591 00000 n What indicators, evidence, and impacts need to be captured within developing systems? Reviewing the research literature means finding, reading, and summarizing the published research relevant to your question. By evaluating the contribution that research makes to society and the economy, future funding can be allocated where it is perceived to bring about the desired impact. These sometimes dissim- ilar views are due to the varied training and background of the writers in terms of their profession, concerned with different aspects of the education process. 1. Concerns over how to attribute impacts have been raised many times (The Allen Consulting Group 2005; Duryea et al. What is the Concept and Importance of Continuous and Comprehensive Evaluation. Definition of Assessment & Evaluation in Education by Different Authors with Its Characteristics, Evaluation is the collection, analysis and interpretation of information about any aspect of a programme of education, as part of a recognised process of judging its effectiveness, its efficiency and any other outcomes it may have., 2. Frameworks for assessing impact have been designed and are employed at an organizational level addressing the specific requirements of the organization and stakeholders. Key features of the adapted criteria . (2005), Wooding et al. The Payback Framework has been adopted internationally, largely within the health sector, by organizations such as the Canadian Institute of Health Research, the Dutch Public Health Authority, the Australian National Health and Medical Research Council, and the Welfare Bureau in Hong Kong (Bernstein et al. Attempts have been made to categorize impact evidence and data, for example, the aim of the MICE Project was to develop a set of impact indicators to enable impact to be fed into a based system. Why should this be the case? A collation of several indicators of impact may be enough to convince that an impact has taken place. The ability to record and log these type of data is important for enabling the path from research to impact to be established and the development of systems that can capture this would be very valuable. It has been acknowledged that outstanding leaps forward in knowledge and understanding come from immersing in a background of intellectual thinking that one is able to see further by standing on the shoulders of giants. For example, following the discovery of a new potential drug, preclinical work is required, followed by Phase 1, 2, and 3 trials, and then regulatory approval is granted before the drug is used to deliver potential health benefits. In developing the UK REF, HEFCE commissioned a report, in 2009, from RAND to review international practice for assessing research impact and provide recommendations to inform the development of the REF. In endeavouring to assess or evaluate impact, a number of difficulties emerge and these may be specific to certain types of impact. Indicators were identified from documents produced for the REF, by Research Councils UK, in unpublished draft case studies undertaken at Kings College London or outlined in relevant publications (MICE Project n.d.). %PDF-1.3 The . The process of evaluation involves figuring out how well the goals have been accomplished. Impact can be temporary or long-lasting. Explain. This petition was signed by 17,570 academics (52,409 academics were returned to the 2008 Research Assessment Exercise), including Nobel laureates and Fellows of the Royal Society (University and College Union 2011). Impact is not static, it will develop and change over time, and this development may be an increase or decrease in the current degree of impact. What are the methodologies and frameworks that have been employed globally to evaluate research impact and how do these compare? The time lag between research and impact varies enormously. 0000002868 00000 n Impact is derived not only from targeted research but from serendipitous findings, good fortune, and complex networks interacting and translating knowledge and research. This is recognized as being particularly problematic within the social sciences where informing policy is a likely impact of research. To allow comparisons between institutions, identifying a comprehensive taxonomy of impact, and the evidence for it, that can be used universally is seen to be very valuable. 2008; CAHS 2009; Spaapen et al. While assessments are often equated with traditional testsespecially the standardized tests developed by testing companies and administered to large populations . trailer << /Size 97 /Info 56 0 R /Root 61 0 R /Prev 396309 /ID[<8e25eff8b2a14de14f726c982689692f><7a12c7ae849dc37acf9c7481d18bb8c5>] >> startxref 0 %%EOF 61 0 obj << /Type /Catalog /Pages 55 0 R /Metadata 57 0 R /AcroForm 62 0 R >> endobj 62 0 obj << /Fields [ ] /DR << /Font << /ZaDb 38 0 R /Helv 39 0 R >> /Encoding << /PDFDocEncoding 40 0 R >> >> /DA (/Helv 0 Tf 0 g ) >> endobj 95 0 obj << /S 414 /T 529 /V 585 /Filter /FlateDecode /Length 96 0 R >> stream However, the Achilles heel of any such attempt, as critics suggest, is the creation of a system that rewards what it can measure and codify, with the knock-on effect of directing research projects to deliver within the measures and categories that reward. It is a process that involves careful gathering and evaluating of data on the actions, features, and consequences of a program.