Despite many attempts to replace it, no alternative definition has . 0000007223 00000 n The fast-moving developments in the field of altmetrics (or alternative metrics) are providing a richer understanding of how research is being used, viewed, and moved. It is desirable that the assignation of administrative tasks to researchers is limited, and therefore, to assist the tracking and collating of impact data, systems are being developed involving numerous projects and developments internationally, including Star Metrics in the USA, the ERC (European Research Council) Research Information System, and Lattes in Brazil (Lane 2010; Mugabushaka and Papazoglou 2012). different meanings for different people in many different contexts. The first category includes approaches that promote invalid or incomplete findings (referred to as pseudoevaluations), while the other three include approaches that agree, more or less, with the definition (i.e., Questions and/or Methods- The most appropriate type of evaluation will vary according to the stakeholder whom we are wishing to inform. Assessment for learning is ongoing, and requires deep involvement on the part of the learner in clarifying outcomes, monitoring on-going learning, collecting evidence and presenting evidence of learning to others.. Figure 1, replicated from Hughes and Martin (2012), illustrates how the ease with which impact can be attributed decreases with time, whereas the impact, or effect of complementary assets, increases, highlighting the problem that it may take a considerable amount of time for the full impact of a piece of research to develop but because of this time and the increase in complexity of the networks involved in translating the research and interim impacts, it is more difficult to attribute and link back to a contributing piece of research. To allow comparisons between institutions, identifying a comprehensive taxonomy of impact, and the evidence for it, that can be used universally is seen to be very valuable. Cb)5. In the majority of cases, a number of types of evidence will be required to provide an overview of impact. In endeavouring to assess or evaluate impact, a number of difficulties emerge and these may be specific to certain types of impact. Understanding what impact looks like across the various strands of research and the variety of indicators and proxies used to evidence impact will be important to developing a meaningful assessment. Published by Oxford University Press. To enable research organizations including HEIs to monitor and manage their performance and understand and disseminate the contribution that they are making to local, national, and international communities. Narratives can be used to describe impact; the use of narratives enables a story to be told and the impact to be placed in context and can make good use of qualitative information. 2007). 2008), developed during the mid-1990s by Buxton and Hanney, working at Brunel University. Indicators were identified from documents produced for the REF, by Research Councils UK, in unpublished draft case studies undertaken at Kings College London or outlined in relevant publications (MICE Project n.d.). Impact is often the culmination of work within spanning research communities (Duryea et al. These traditional bibliometric techniques can be regarded as giving only a partial picture of full impact (Bornmann and Marx 2013) with no link to causality. Search for other works by this author on: A White Paper on Charity Impact Measurement, A Framework to Measure the Impact of Investments in Health Research, European Molecular Biology Organization (EMBO) Reports, Estimating the Economic Value to Societies of the Impact of Health Research: A Critical Review, Bulletin of the World Health Organization, Canadian Academy of Health Sciences Panel on Return on Investment in Health Research, Making an Impact. << /Length 5 0 R /Filter /FlateDecode >> Times Higher Education, Assessing the Impact of Social Science Research: Conceptual, Methodological and Practical Issues, A Profile of Federal-Grant Administrative Burden Among Federal Demonstration Partnership Faculty, Department for Business, Innovation and Skills, The Australian Research Quality Framework: A live experiment in capturing the social, economic, environmental and cultural returns of publicly funded research, Reforming the Evaluation of Research. Aspects of impact, such as value of Intellectual Property, are currently recorded by universities in the UK through their Higher Education Business and Community Interaction Survey return to Higher Education Statistics Agency; however, as with other public and charitable sector organizations, showcasing impact is an important part of attracting and retaining donors and support (Kelly and McNicoll 2011). Wigley (1988, p 21) defines it as "a data reduction process that involves the . Definition of Evaluation by Different Authors Tuckman: Evaluation is a process wherein the parts, processes, or outcomes of a programme are examined to see whether they are satisfactory, particularly with reference to the stated objectives of the programme our own expectations, or our own standards of excellence. In viewing impact evaluations it is important to consider not only who has evaluated the work but the purpose of the evaluation to determine the limits and relevance of an assessment exercise. It is time-intensive to both assimilate and review case studies and we therefore need to ensure that the resources required for this type of evaluation are justified by the knowledge gained. Downloadable! Although it can be envisaged that the range of impacts derived from research of different disciplines are likely to vary, one might question whether it makes sense to compare impacts within disciplines when the range of impact can vary enormously, for example, from business development to cultural changes or saving lives? A collation of several indicators of impact may be enough to convince that an impact has taken place. Over the past year, there have been a number of new posts created within universities, such as writing impact case studies, and a number of companies are now offering this as a contract service. (2007) surveyed researchers in the US top research institutions during 2005; the survey of more than 6000 researchers found that, on average, more than 40% of their time was spent doing administrative tasks. As part of this review, we aim to explore the following questions: What are the reasons behind trying to understand and evaluate research impact? It is therefore in an institutions interest to have a process by which all the necessary information is captured to enable a story to be developed in the absence of a researcher who may have left the employment of the institution. One way in which change of opinion and user perceptions can be evidenced is by gathering of stakeholder and user testimonies or undertaking surveys. SROI aims to provide a valuation of the broader social, environmental, and economic impacts, providing a metric that can be used for demonstration of worth. Incorporating assessment of the wider socio-economic impact began using metrics-based indicators such as Intellectual Property registered and commercial income generated (Australian Research Council 2008). The book also explores how different aspects of citizenship, such as attitudes towards diverse population groups and concerns for social issues, relate to classical definitions of norm-based citizenship from the political sciences. This atmosphere of excitement, arising from imaginative consideration transforms knowledge.. , , . By allowing impact to be placed in context, we answer the so what? question that can result from quantitative data analyses, but is there a risk that the full picture may not be presented to demonstrate impact in a positive light? The Goldsmith report (Cooke and Nadim 2011) recommended making indicators value free, enabling the value or quality to be established in an impact descriptor that could be assessed by expert panels. Providing advice and guidance within specific disciplines is undoubtedly helpful. To adequately capture interactions taking place between researchers, institutions, and stakeholders, the introduction of tools to enable this would be very valuable. It is perhaps assumed here that a positive or beneficial effect will be considered as an impact but what about changes that are perceived to be negative? 0000348082 00000 n This framework is intended to be used as a learning tool to develop a better understanding of how research interactions lead to social impact rather than as an assessment tool for judging, showcasing, or even linking impact to a specific piece of research. In this article, we draw on a broad range of examples with a focus on methods of evaluation for research impact within Higher Education Institutions (HEIs). HEFCE developed an initial methodology that was then tested through a pilot exercise. One of the advantages of this method is that less input is required compared with capturing the full route from research to impact. Evaluative research is a type of research used to evaluate a product or concept, and collect data to help improve your solution. A Preferred Framework and Indicators to Measure Returns on Investment in Health Research, Measuring Impact Under CERIF at Goldsmiths, Anti-Impact Campaigns Poster Boy Sticks up for the Ivory Tower. 2007) who concluded that the researchers and case studies could provide enough qualitative and quantitative evidence for reviewers to assess the impact arising from their research (Duryea et al. n.d.). In this case, a specific definition may be required, for example, in the Research Excellence Framework (REF), Assessment framework and guidance on submissions (REF2014 2011b), which defines impact as, an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia. Any person who has made a significant . 4. However, there has been recognition that this time window may be insufficient in some instances, with architecture being granted an additional 5-year period (REF2014 2012); why only architecture has been granted this dispensation is not clear, when similar cases could be made for medicine, physics, or even English literature. The development of tools and systems for assisting with impact evaluation would be very valuable. Using the above definition of evaluation, program evaluation approaches were classified into four categories. Even where we can evidence changes and benefits linked to our research, understanding the causal relationship may be difficult. This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. The Value of Public Sector R&D, Assessing impacts of higher education systems, National Co-ordinating Centre for Public Engagement, Through a Glass, Darkly: Measuring the Social Value of Universities, Describing the Impact of Health Research: A Research Impact Framework, LSE Public Policy Group. The Payback Framework enables health and medical research and impact to be linked and the process by which impact occurs to be traced. Replicated from (Hughes and Martin 2012). Evaluate means to assess the value of something. Systems need to be able to capture links between and evidence of the full pathway from research to impact, including knowledge exchange, outputs, outcomes, and interim impacts, to allow the route to impact to be traced. New Directions for Evaluation, Impact is a Strong Weapon for Making an Evidence-Based Case Study for Enhanced Research Support but a State-of-the-Art Approach to Measurement is Needed, The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance, Evaluation in National Research Funding Agencies: Approaches, Experiences and Case Studies, Methodologies for Assessing and Evidencing Research Impact. This might describe support for and development of research with end users, public engagement and evidence of knowledge exchange, or a demonstration of change in public opinion as a result of research. While aspects of impact can be adequately interpreted using metrics, narratives, and other evidence, the mixed-method case study approach is an excellent means of pulling all available information, data, and evidence together, allowing a comprehensive summary of the impact within context.