How do you solve a problem like the black box? Process evaluation and its application alongside complex health interventions

By Stephen McIntyre

Complex evaluations for complex interventions

Many interventions in health research can be considered complex, involving numerous interacting components, target behaviours, organisational levels, and outcomes. To ensure that we evaluate interventions appropriately, evaluations must reflect this complexity. Though randomised controlled trials (RCTs) are considered the ‘gold standard’ of evaluating health interventions, increasingly the need to understand the processes and mechanisms through which interventions operate has been recognised. Without this information, it may not be possible to interpret trial outcomes and potentially effective interventions may be discarded prematurely.

Process evaluations address these concerns by investigating how, why, and for whom interventions are effective or ineffective. Recently, process evaluation research has thrived with the aim of opening the ‘black box’ of intervention research (i.e. understanding how interventions produce their effects). To assist researchers and policy makers in this task, the Medical Research Council (MRC) recently released guidance on the conduct and reporting of process evaluations of complex interventions.

2b

Key process evaluation functions according to MRC guidance

Process evaluations are used to assess a variety of factors including:

Ultimately, a comprehensive understanding of these factors may provide evidence to inform if an intervention can be sustained, scaled-up, or transferred to different populations or contexts successfully.

Pick and mixed-methods

Given the range of potential aims associated with process evaluations, it’s not surprising that they are associated with a variety of methods. Reflecting their exploratory nature, process evaluations often incorporate qualitative methods to investigate how interventions are understood and how implementation and contextual processes are perceived. Quantitative methods may focus on quantifying and gaining insights on these issues in larger samples, measuring key process variables, or testing hypothesised causal pathways. Commonly used qualitative methods include interviews, focus groups, and researcher-conducted observations; while quantitative methods may include questionnaires and secondary analysis of routine data.

Setting the twittersphere alight with #mixedmethodsmemes

Although multiple and mixed methods are often used in process evaluations, they are not always integrated or are only combined in a limited manner. Tonkin-Crine et al. (2016) provide an example of a well-integrated mixed-methods process evaluation that was conducted alongside a trial to promote prudent antibiotic prescribing in general practice. They develop a comprehensive analysis by triangulating data from semi-structured interviews and questionnaires conducted with both patients and healthcare professionals. By comparing key findings across analyses, they examine areas of agreement and disagreement across datasets, as well as findings that are present in one but absent in another dataset.

Process evaluation frameworks and approaches

As well as the MRC framework on the conduct and reporting of process evaluations, a number of reporting and evaluation frameworks exist that offer alternative or complementary approaches to conceiving and conducting evaluation efforts. A recent framework devised by Grant et al. (2013) provides specific guidance on designing and reporting process evaluations conducted alongside cluster-RCTs. It highlights considerations such as: cluster and target population processes (e.g. intervention delivery, responses to the intervention), maintenance of intervention processes, associations between trial processes and effectiveness, unintended consequences, theoretical explanations for change, and context.

Theory-driven evaluation is another approach to process evaluation that explicitly examines the programme theory of an intervention (i.e. the explanation of how the intervention is expected to function and a description of what activities need to occur before others). For example, realist evaluation suggests that complex interactions between mechanisms (i.e. the resources offered by an intervention and how they influence participants’ reasoning) and contexts (e.g. social, political, historical) produce intervention outcomes.

Meanwhile, theory of change is a methodology that emphasises stakeholder involvement throughout the evaluation process. In this approach, a highly specified diagram is typically constructed that specifies the individual steps that need to occur for an intervention to achieve its intended goals as well as the assumptions underlying these changes.

3

My team’s work

My team are involved in the AFFINITIE programme – a project aiming to develop and evaluate two theory and evidence informed feedback interventions to promote evidence-based blood transfusion practice. The interventions are being evaluated in two pragmatic, cluster-RCTs and parallel process evaluations. The AFFINITIE process evaluations will assess intervention fidelity multidimensionally according to five fidelity domains proposed by the Behaviour Change Consortium: study design, training, delivery, receipt, and enactment. We’re collecting a range of data including semi-structured interviews, questionnaires, and web analytics data (e.g. downloads and usage of online intervention content). Planned mediation analyses will also examine associations between observed fidelity and trial outcomes.

Process evaluation resources

Process evaluation holds promise to open the ‘black box’ of intervention research but it remains an emerging field with plenty of opportunities for advancement. For those interested in learning more about process evaluations or my team’s work, here are some resources you may want to check out:

 

 

stephen

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s