2010). Dennis Atsu Dake. The . Why should this be the case? There are areas of basic research where the impacts are so far removed from the research or are impractical to demonstrate; in these cases, it might be prudent to accept the limitations of impact assessment, and provide the potential for exclusion in appropriate circumstances. It has been acknowledged that outstanding leaps forward in knowledge and understanding come from immersing in a background of intellectual thinking that one is able to see further by standing on the shoulders of giants. 0000006922 00000 n This framework is intended to be used as a learning tool to develop a better understanding of how research interactions lead to social impact rather than as an assessment tool for judging, showcasing, or even linking impact to a specific piece of research. The range and diversity of frameworks developed reflect the variation in purpose of evaluation including the stakeholders for whom the assessment takes place, along with the type of impact and evidence anticipated. Time, attribution, impact. To achieve compatible systems, a shared language is required. In the educational context, the . 0000008591 00000 n The introduction of impact assessments with the requirement to collate evidence retrospectively poses difficulties because evidence, measurements, and baselines have, in many cases, not been collected and may no longer be available. 2005). 6. From an international perspective, this represents a step change in the comprehensive nature to which impact will be assessed within universities and research institutes, incorporating impact from across all research disciplines. A Preferred Framework and Indicators to Measure Returns on Investment in Health Research, Measuring Impact Under CERIF at Goldsmiths, Anti-Impact Campaigns Poster Boy Sticks up for the Ivory Tower. (2007), Nason et al. Despite the concerns raised, the broader socio-economic impacts of research will be included and count for 20% of the overall research assessment, as part of the REF in 2014. Impact is assessed alongside research outputs and environment to provide an evaluation of research taking place within an institution. It is possible to incorporate both metrics and narratives within systems, for example, within the Research Outcomes System and Researchfish, currently used by several of the UK research councils to allow impacts to be recorded; although recording narratives has the advantage of allowing some context to be documented, it may make the evidence less flexible for use by different stakeholder groups (which include government, funding bodies, research assessment agencies, research providers, and user communities) for whom the purpose of analysis may vary (Davies et al. In the UK, evaluation of academic and broader socio-economic impact takes place separately. n.d.). The first attempt globally to comprehensively capture the socio-economic impact of research across all disciplines was undertaken for the Australian Research Quality Framework (RQF), using a case study approach. Research findings will be taken up in other branches of research and developed further before socio-economic impact occurs, by which point, attribution becomes a huge challenge. "Evaluation is a process of judging the value of something by certain appraisal." Characteristics of evaluation in Education Below are some of the characteristics of evaluation in education, Continuous Process Comprehensive Child-Centered Cooperative Process Common Practice Teaching Methods Multiple Aspects Continuous Process The Oxford English Dictionary defines impact as a 'Marked effect or influence', this is clearly a very broad definition. Impact is derived not only from targeted research but from serendipitous findings, good fortune, and complex networks interacting and translating knowledge and research. While assessments are often equated with traditional testsespecially the standardized tests developed by testing companies and administered to large populations . SROI aims to provide a valuation of the broader social, environmental, and economic impacts, providing a metric that can be used for demonstration of worth. Over the past year, there have been a number of new posts created within universities, such as writing impact case studies, and a number of companies are now offering this as a contract service. However, it must be remembered that in the case of the UK REF, impact is only considered that is based on research that has taken place within the institution submitting the case study. Prague, Czech Republic, Health ResearchMaking an Impact. It is acknowledged in the article by Mugabushaka and Papazoglou (2012) that it will take years to fully incorporate the impacts of ERC funding. Impact assessments raise concerns over the steer of research towards disciplines and topics in which impact is more easily evidenced and that provide economic impacts that could subsequently lead to a devaluation of blue skies research. 8. The ability to record and log these type of data is important for enabling the path from research to impact to be established and the development of systems that can capture this would be very valuable. Teresa Penfield, Matthew J. Baker, Rosa Scoble, Michael C. Wykes, Assessment, evaluations, and definitions of research impact: A review, Research Evaluation, Volume 23, Issue 1, January 2014, Pages 2132, https://doi.org/10.1093/reseval/rvt021. 2009; Russell Group 2009). Muffat says - "Evaluation is a continuous process and is concerned with than the formal academic achievement of pupils. Measurement assessment and evaluation helps the teachers to determine the learning progress of the students. The difficulty then is how to determine what the contribution has been in the absence of adequate evidence and how we ensure that research that results in impacts that cannot be evidenced is valued and supported. Although metrics can provide evidence of quantitative changes or impacts from our research, they are unable to adequately provide evidence of the qualitative impacts that take place and hence are not suitable for all of the impact we will encounter. 0000342958 00000 n The term "assessment" may be defined in multiple ways by different individuals or institutions, perhaps with different goals. (2005), Wooding et al. We suggest that developing systems that focus on recording impact information alone will not provide all that is required to link research to ensuing events and impacts, systems require the capacity to capture any interactions between researchers, the institution, and external stakeholders and link these with research findings and outputs or interim impacts to provide a network of data. Perhaps, SROI indicates the desire to be able to demonstrate the monetary value of investment and impact by some organizations. The time lag between research and impact varies enormously. Cooke and Nadim (2011) also noted that using a linear-style taxonomy did not reflect the complex networks of impacts that are generally found. The origin is from the Latin term 'valere' meaning "be strong, be well; be of value, or be worth". What indicators, evidence, and impacts need to be captured within developing systems? Key features of the adapted criteria . Here we address types of evidence that need to be captured to enable an overview of impact to be developed. Collecting this type of evidence is time-consuming, and again, it can be difficult to gather the required evidence retrospectively when, for example, the appropriate user group might have dispersed. As part of this review, we aim to explore the following questions: What are the reasons behind trying to understand and evaluate research impact? Oxford University Press is a department of the University of Oxford. Hb```f``e`c`Tgf@ aV(G Ldw0p)}c4Amff0`U.q$*6mS,T",?*+DutQZ&vO T4]2rBWrL.7bs/lcx&-SbiDEQ&. The most appropriate type of evaluation will vary according to the stakeholder whom we are wishing to inform. In the UK, there have been several Jisc-funded projects in recent years to develop systems capable of storing research information, for example, MICE (Measuring Impacts Under CERIF), UK Research Information Shared Service, and Integrated Research Input and Output System, all based on the CERIF standard. In the UK, the Russell Group Universities responded to the REF consultation by recommending that no time lag be put on the delivery of impact from a piece of research citing examples such as the development of cardiovascular disease treatments, which take between 10 and 25 years from research to impact (Russell Group 2009). It is therefore in an institutions interest to have a process by which all the necessary information is captured to enable a story to be developed in the absence of a researcher who may have left the employment of the institution. New Directions for Evaluation, Impact is a Strong Weapon for Making an Evidence-Based Case Study for Enhanced Research Support but a State-of-the-Art Approach to Measurement is Needed, The Limits of Nonprofit Impact: A Contingency Framework for Measuring Social Performance, Evaluation in National Research Funding Agencies: Approaches, Experiences and Case Studies, Methodologies for Assessing and Evidencing Research Impact. It incorporates both academic outputs and wider societal benefits (Donovan and Hanney 2011) to assess outcomes of health sciences research. A Review of International Practice, HM Treasury, Department for Education and Skills, Department of Trade and Industry, Yes, Research can Inform Health Policy; But can we Bridge the Do-Knowing its been Done Gap?, Council for Industry and Higher Education, UK Innovation Research Centre. This is a metric that has been used within the charitable sector (Berg and Mnsson 2011) and also features as evidence in the REF guidance for panel D (REF2014 2012). Evaluation is the systematic collection and inter- pretation of evidence leading as a part of process to a judgement of value with a view to action., Evaluation is the application of a standard and a decision-making system to assessment data to produce judgments about the amount and adequacy of the learning that has taken place., 1. For full access to this pdf, sign in to an existing account, or purchase an annual subscription. The authors propose a new definition for measurement process based on the identification of the type of measurand and other metrological elements at each measurement process identified. x[s)TyjwI BBU*5,}~O#{4>[n?_?]ouO{~oW_~fvZ}sCy"n?wmiY{]9LXn!v^CkWIRp&TJL9o6CjjvWqAQ6:hU.Q-%R_O:k_v3^=79k{8s7?=`|S^BM-_fa@Q`nD_(]/]Y>@+no/>$}oMI2IdMqH,'f'mxlfBM?.WIn4_Jc:K31vl\wLs];k(vo_Teq9w2^&Ca*t;[.ybfYYvcn A very different approach known as Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions (SIAMPI) was developed from the Dutch project Evaluating Research in Context and has a central theme of capturing productive interactions between researchers and stakeholders by analysing the networks that evolve during research programmes (Spaapen and Drooge, 2011; Spaapen et al. 0000002318 00000 n More details on SROI can be found in A guide to Social Return on Investment produced by The SROI Network (2012). 0000004731 00000 n 0000009507 00000 n The case study does present evidence from a particular perspective and may need to be adapted for use with different stakeholders. RAND Europe, Capturing Research Impacts. A taxonomy of impact categories was then produced onto which impact could be mapped. The ability to write a persuasive well-evidenced case study may influence the assessment of impact. (2011) Maximising the Impacts of Your Research: A Handbook for Social Scientists (Pubd online) <, Lets Make Science Metrics More Scientific, Measuring Impact Under CERIF (MICE) Project Blog, Information systems of research funding agencies in the era of the Big Data. 0000342980 00000 n If this research is to be assessed alongside more applied research, it is important that we are able to at least determine the contribution of basic research. An alternative approach was suggested for the RQF in Australia, where it was proposed that types of impact be compared rather than impact from specific disciplines. What are the reasons behind trying to understand and evaluate research impact? We will focus attention towards generating results that enable boxes to be ticked rather than delivering real value for money and innovative research. Where narratives are used in conjunction with metrics, a complete picture of impact can be developed, again from a particular perspective but with the evidence available to corroborate the claims made. It is worth considering the degree to which indicators are defined and provide broader definitions with greater flexibility. Assessment refers to a related series of measures used to determine a complex attribute of an individual or group of individuals. 2006; Nason et al. Classroom Assessment -- (sometime referred to as Course-based Assessment) - is a process of gathering data on student learning during the educational experience, designed to help the instructor determine which concepts or skills the students are not learning well, so that steps may be taken to improve the students' learning while the course is 0000346296 00000 n To evaluate impact, case studies were interrogated and verifiable indicators assessed to determine whether research had led to reciprocal engagement, adoption of research findings, or public value. The Payback Framework systematically links research with the associated benefits (Scoble et al. By evaluating the contribution that research makes to society and the economy, future funding can be allocated where it is perceived to bring about the desired impact. The inherent technical disparities between the two different software packages and the adjustment . The main risks associated with the use of standardized metrics are that, The full impact will not be realized, as we focus on easily quantifiable indicators. 0000007307 00000 n Johnston (Johnston 1995) notes that by developing relationships between researchers and industry, new research strategies can be developed. Merit refers to the intrinsic value of a program, for example, how effective it is in meeting the needs those it is intended help. This is being done for collation of academic impact and outputs, for example, Research Portfolio Online Reporting Tools, which uses PubMed and text mining to cluster research projects, and STAR Metrics in the US, which uses administrative records and research outputs and is also being implemented by the ERC using data in the public domain (Mugabushaka and Papazoglou 2012).