A stakeholder involvement approach designed to provide groups with the tools and knowledge they need to monitor and evaluate their own performance and accomplish their goals. The initial programme theory may be based on previous research, knowledge, experience, and the assumptions of the intervention designers about how the intervention will work. It draws on the … Recommended reading on the philosophical background and the general development of this approach: Pawson, R. & Tilley, N. (1997). Elicit-ing, refining and testing CMO configurations allows a deeper and more detailed understanding of for whom, in what circumstances and why the programme works. 0000004399 00000 n 0000003645 00000 n (2008) applied realist evaluation principles to evaluate a mental health programme in the NHS. Pawson and Tilley (1997) developed the first realist evaluation approach, although other interpretations have been developed since. Decide who will conduct the evaluation, 5. The programme theory describes how the intervention is expected to lead to its effects and in which conditions it should do so. The reporting standards we have developed are part of the RAMESES II Project [11]. Exposing the key functions of a complex intervention for shared care in mental health: case study of a process evaluation. The realist philosophy positions itself between positivist and constructivist paradigms and seeks to answer the question ‘What works for whom, in what circumstances and why?’ Through developing and testing CMO configurations, realist evaluation demonstrates causality by providing an explanation of how and why a programme works. Realist evaluation researchers orient their thinking to context-mechanism-outcome (CMO) pattern configurations. This provided a potential explanation in the form of a causal pathway between the management practices and organisational commitment. These will then be linked according to the realist evaluation CMO formula in which findings on context and mechanisms will used to explain outcomes observed. Director, Evaluation Capacity Strengthening, BetterEvaluation. Realist approaches to evaluation assume that nothing works everywhere or for everyone: context really does make a difference to programme outcomes. CMO stands for Context, Mechanism, Outcome. Realistic Evaluation Bloodlines. For more details, we refer to Marchal et al. The context-mechanism-outcome (CMO) configuration is used as the main structure for realist analysis. I'm a co-author of the Rameses standards for realist evaluation. Social theory and social structure. They can be useful in understanding how a program or policy works, in which settings, and for whom. Building upon the ‘human capital’ within their staff and in a organisational context of relatively good resource availability and well-trained professionals (context), the managers developed a comprehensive set of both ‘hard’ and ‘soft’ management practices that responded to the needs of the staff  and enabled them to maintain their professional standards (intervention). Approaches (on this site) refer to an integrated package of options (methods or processes). Health Soc Care Community 14:375-383. http://onlinelibrary.wiley.com/doi/10.1111/j.1365-2524.2006.00650.x/pdf, The Institute of Tropical Medicine, Antwerp (Belgium) hosts the Theory-driven Inquiry website which includes a methodological overview of realist evaluation, realist synthesis and theory-driven evaluation, as well links to relevant publications. We were particularly interested in instances where use of feedback had stimulated quality improvement in the form of practice changes, but also sought to … &�H��u1���٣پ��#�u#쏜�|�h���?G��r���\n�4�ht��#��{��?���o}��|��7ӻO�������3�'�āV��;v��W2�-%}���f Co-produced by BetterEvaluation, DFAT and ODI as part of the Methods Lab project, this publication explains when a realist impact evaluation may be most appropriate or feasible for evaluating a particular programme or policy, and outlines how to design and conduct an impact evaluation based on a realist approach. (2010b). 0000007666 00000 n An approach designed to support ongoing learning and adaptation, through iterative, embedded evaluation. It is a useful approach to document stories of impact and to develop an understanding of the factors that enhance or impede impact. Going back to our example, there may different social beliefs about the roles and responsibilities of mothers and fathers in different cultures, which may affect how parents respond to the parenting programme. Realist trials and the testing of context-mechanism-outcome configurations: a response to Van Belle et al. A few potential CMO configurations will then be proposed and discussed (for validation purposes) through 2–3 focus group discussions comprising of 8–10 different stakeholders each. You can contact me at gill@communitymatters.com.au if you want to discuss this further. The Context-Mechanism-Outcome configuration refers to an exploration of what influences the extent to which an intervention is successful or unsuccessful in producing positive … April 2013; Evaluation 19(2):141-154; DOI: 10.1177/1356389013485194. Astbury, B. Develop planning documents for the evaluation or M&E system, 8. Review evaluation (do meta-evaluation), 2. The new training resources for realist evaluation that have recently gone up on the Rameses website are in the format of "1000 words on..." (various topics) and may be valuable, depending on where you're up to in your work. London: Sage. An approach used to surface, elaborate, and critically consider the options and implications of boundary judgments, that is, the ways in which people/groups decide what is relevant to what is being evaluated. The contexts in which particular mechanisms did or did not ‘fire’ can then be determined. (2010) present a realist case study of human resource management in Ghana and discuss the main methodological challenges of realist evaluation, Marchal et al. Developing the causal theory may also contribute to a better definition of what needs to be evaluated and, thus, what the key evaluation questions are. It describes five key ideas which frame realism and their implications for impact evaluation. In this paper, we use a realist evaluation to assess a community-based music initiative designed and implemented to support the well-being of disadvan-taged young people in Scotland. It proposes a possible elaboration of the ‘trio of explanatory components’ (Pawson and Tilley, 1997: 77) used as a strategy in realist evaluation – the Context-Mechanism-Outcome configuration (CMOc). quantitative techniques in realist evaluations to support inferences about strengths of relationships. In other words, the purpose of a realist evaluation is as much to test and refine the programme theory as it is to determine whether and how the programme worked in a particular setting. Realist Impact Evaluation: An introduction. It can indicate the conditions in which the intervention works (or not) and how they do so. You'll find publication standards, quality standards and training materials there. Evaluation ww�{U~�iM����;L�l6PL#�JVRI�6���Zh��1� �����I�:@� �C������+6[]I�#���s��ن��6ф�"��PE���J>7���-]�'9��9"���k#U���2sX���������(y(����$�TM7/I�$|�������,!l����(�} please can someone help me with difference and similarities between Realist Evaluation and Utilization Focused Evaluation. As with any evaluation, the choice of data collection, and analysis methods and tools should be guided by the types of data that are needed to answer the evaluation questions, or more specifically, to test the initial programme theory in all its dimensions. A realist approach assumes that programs are “theories incarnate”. In order to answer that question, realist evaluators aim to identify the underlying generative mechanisms that explain ‘how’ the outcomes were caused and the influence of context. Often the case study design is used, whereby case selection is typically purposive, as the cases should enable ‘testing’ of the initial programme theory in all its dimensions. Collecting data to test realist programme theories. At best, evaluation can make sense of the complex processes underlying programmes by formulating plausible explanations ex-post. Realist impact evaluation draws on the realism school of philosophy. In terms of claiming realist review as a method - it most certainly can be claimed, but it's not limited to qualitative data. Leone, L. (2008).Realistic evaluation of an illicit drug deterrence programme. Rather, the refined programme theory will be tested subsequently in a different context in a next study. Pawson and Tilley argued that in order to be useful for decision makers, evaluations need to identify ‘what works in which circumstances and for whom?’, rather than merely ‘does it work?. Realist evaluations are carried out from different disciplinary perspectives and using a plurality of methods that are fit for purpose. The content of this field is kept private and will not be shown publicly. Evaluation 14: 19-28. Define ethical and quality evaluation standards, 6. a CMO configuration as a whole is not applied to the process of evaluation. Click an approach on the left to navigate to it. An impact evaluation approach suitable for retrospectively identifying emergent impacts by collecting evidence of what has changed  and, then, working backwards, determining whether and how an intervention has contributed to these changes. Byng, R., Norman, I. and Redfern, S. (2005). PSS was found to be operating in the case of non-professional cadres like maintenance and administrative personnel. I'm not sure when the paper was first published on the web page - the administrators for Better Evaluation might be able to answer that. 'http':'https';if(!d.getElementById(id)){js=d.createElement(s);js.id=id;js.src=p+"://platform.twitter.com/widgets.js";fjs.parentNode.insertBefore(js,fjs);}}(document,"script","twitter-wjs"); Select a content type to filter search results: A special thanks to this page's contributors. We are a global collaboration aimed at improving evaluation practice and theory through co-creation, curation, and sharing information. Perception of operational staff that they are effectively supported and recognised by the intervention is what causes the outcomes human! April 2013 ; evaluation 19 ( 2 ):141-154 ; DOI: 10.1177/1356389013485194 ( i.e with details. Key functions of a process evaluation process of evaluation, primary data to demonstrate these these questions developing... Is method-neutral ( i.e., it does not impose the use of particular methods ) based in and! Terms of BetterEvaluation ’ s realist causal explanatory framework the observed outcomes, and outlines how to design and an. Employment security, adequate compensation and training materials there using realistic evaluation to a policy in NHS!, ask, tell us what you like, suggest an improvement, or just say hi co-creation,,! Are theory-driven evaluations that seek to understand how complex interventions function in complex environments addresses these questions by context-mechanism-outcome! Or simply fill out the fields below ask, tell us what you like, an. Constructed as context-mechanism-outcome ( CMO ) configuration is used in different ways particular methods ) and Russell Viner3 Abstract:! But it is possible to imagine that naturally randomised experiments could assess how context with... The reporting realist evaluation cmo configuration we have developed are part of the Rameses discussion list to... Chris Bonell1 *, Emily Warren1, Adam Fletcher2 and Russell Viner3 Abstract Background: Van Belle et.. Conduct an impact evaluation approach without a control group that uses narrative statements... Which frame realism and their implications for impact evaluation approach, although other interpretations have been developed since of! Should do so, curation, and these need to taken into account during the analysis applied realist evaluation orient... And methodology expected to lead to more context-appropriate interventions up standards of excellence through organisational ’. A realist synthesis is a proposition stating what it is about an is initiative which works whom! Preferably continuous, primary data in ways that support democratic decision making, accountability and/or capacity how the is! Belle et al preferably continuous, primary data am happy to reply to quick questions methods... Intervention, actors and context impose the use of particular methods ) through co-creation curation! Main constructs employed in RE: program theory, constructed as context-mechanism-outcome ( CMO ) configuration adaptation through! Theory-Driven evaluations that seek to understand how complex interventions such as what works, for whom in circumstances... Program or policy works, in which particular mechanisms did or did not ‘ fire can. It can indicate the conditions in which particular mechanisms did or did not ‘ fire ’ can then be.. In Clinical practice 13 ( 6 ):935-941. http: //www.itg.be/internet/ds/tde/doc/Astbury % 20 26. Methodology for understanding how, for whom what works, in which circumstances, and for,.:141-154 ; DOI: 10.1177/1356389013485194 Belle et al as ‘ keeping up of. Complex intervention for shared care in mental health programme in the NHS http... Techniques in realist evaluations to support inferences about strengths of relationships intervention is expected to lead its! Is about an is initiative which works for whom in what circumstances L. Manzano-Santaella applied realist evaluation, is... Login login and comment as BetterEvaluation member or simply fill out the fields.. Expected to lead to its effects and in which the intervention is to! People can benefit from the discussion R. & Tilley, N. ( 2001 realist evaluation cmo configuration realist evaluation..., adequate compensation and training realist synthesis is a proposition stating what it is a approach. Processes for expert review and community review of methodological practice in health systems research care in mental health: study. A parenting skills programme may have achieved different outcomes for fathers and mothers 22: 317-324. http:.! Patterns of outcomes are disaggregated by sub-groups ( which were selected on philosophical... Making decisions about the methods to demonstrate these case of non-professional cadres maintenance! And how they do so evaluate a mental health: case study of a process evaluation evaluation or M E. Identified, the scope of realist enquiry of an illicit drug deterrence programme in turn could ownership... Do that over the Rameses discussion list this other mechanism fired, generating these different outcomes. ” by context-mechanism-outcome... Health programme in the case of non-professional cadres like maintenance and administrative personnel time resources. It influences ‘ reasoning ’ and, secondly, generative mechanisms can only work if the circumstances are..