Developing a systems-based approach to research use in education
Abstract
Models of research use in education tend to focus on specific elements of education systems or underplay the complexity of system change. Within other public policy areas, notably health, more work has been undertaken to integrate systems thinking when considering knowledge mobilisation and research use. In this paper, we survey public policy system change literature to develop a set of system dimensions. We use these to examine models relating to research use that are widely referenced in education. We then apply these dimensions to the work of the Education Endowment Foundation (EEF), the UK's What Works Centre for Education, which aims to support evidence-informed practice at all levels of the education system. We focus on its work to embed research-informed practices in regional school systems, through a case analysis of two ‘scale-up campaigns’ to mobilise evidence relating to the effective deployment of teaching assistants (educational support paraprofessionals). The findings highlight the value of using the system dimensions framework as a diagnostic tool to understand how to effect system change, highlighting the key role of brokerage and system leadership at different system levels; school-level capacity to implement change; and system relationships.
Context and implications
Rationale for this study
Current approaches to supporting research use in education underplay the complexity of system change. Failure to acknowledge the complexity of research-use systems is likely to result in less-than-optimal approaches and interventions to improve research use.
Why the new findings matter
By applying a systems perspective, we explore how research use can be more effectively supported.
Implications for educational researchers and policymakers
- Research use emerges as a myriad of interconnected ‘moving parts’ that need to function optimally and be aligned. Weakness in any area of the system, or interactions between system actors and activities across system levels, can potentially impede research use.
- A systems perspective, using the dimensions table presented in the paper, can be used to examine the functioning of existing systems and make informed decisions on where best to intervene to support practitioners’ use of research.
- There is value in exploring multi-stranded mobilisation strategies that work together at different levels of the system e.g. school, regional policy, national.
INTRODUCTION
Models of research use in education tend to focus on specific elements of education systems or, as we go on to argue in this article, underplay the complexity of system change. Within other public policy areas, notably health, more work has been undertaken to integrate systems approaches when considering knowledge mobilisation and research use. A focus on considering the system is important because jurisdictions across the world are increasingly developing national and/or federal infrastructures for research use in education (Malin et al., 2020). This requires understanding the approaches taken at the system level, in addition to the level of school (e.g., Godfrey's, 2016 ecological approach) or individual practitioner (such as Cain's, 2015 work on knowledge practices of teachers).
We focus here on the English system, which has a wide range of structures in place to support knowledge mobilisation in education. The Education Endowment Foundation (EEF) is a core element of this system, funding over 150 randomised trials, producing evidence-based guidance reports on aspects of education practices, and supporting a network of intermediary Research Schools that act as knowledge brokers (as we discuss further in The Education Endowment Foundation and the Scale-Up Campaign). The EEF sits alongside grassroots teacher networks, a new professional body for teachers, the Chartered College of Teaching, and a range of third sector groups including the Teacher Development Trust, Ambition Institute, the National Institute of Teaching, Centre for the Utilisation of Research Evidence in Education (CUREE) and the National Foundation for Educational Research (Coldwell, 2022).
There is some evidence of increasing awareness of research evidence in some areas. We discuss this in the body of the paper in relation to effective deployment of teaching assistants (TAs), but there are also more recent signs of potential increases in interest and demand for research-informed work. For example, a recent study of schools' responses to the Covid-19 pandemic in England (Achtaridou et al., 2022) found EEF resources to be the most useful source of guidance during this period, with 90% of primary schools finding them useful. However, teachers' and leaders' applied use of research evidence in England is quite low, and (at least until 2019) this had not changed even after several years of significant work. The results of an evidence review (Nelson & O'Beirne, 2014) and two related inter-linked surveys (Nelson et al., 2017; Walker et al., 2019) found that direct experience in the classroom and the views of other teachers are more important sources for teachers than other evidence, especially external academic research evidence. Walker et al. (Walker et al., 2019, p. 4) note teachers were much more likely to draw ideas and support from their own experiences (60% of respondents identified ‘ideas generated by me or my school’), or the experiences of other teachers/schools (42% of respondents identified ‘ideas from other schools’), when deciding on approaches to support pupil progress.
In other public policy areas, similar issues have been found, and one explanation that informs the approach in this article relates to the argument of Best and Holmes (2010) in the health sector. They suggest that previous ‘generations’ of approaches to research use have only partially understood how research evidence can inform practice. They identify that the first generation—linear models—suggest a one-way process from evidence to practice. Such linear approaches imply a model of practice that is centred on the uptake of finished products, often research-based programmes, which naturally prioritise the use of top-down incentives, communications and policies as a means to widespread research use. Best and Holmes suggest such approaches, by themselves, have limited use, except in a small range of conducive organisational settings. Second-generation approaches—relationship models—can be more effective in local settings with strong interpersonal relationships, but do not recognise that multiple, overlapping forms may be needed for more widespread change. Thus, they suggest that third-generation system-based approaches (which we discuss in more depth in Systems approaches in social policy research: developing a set of systems dimensions) are of more use in a range of more complex settings. These ideas have parallels in educational thinking; for example, Opfer and Pedder's (2011) systemic model of teacher professional learning—akin to Best and Holmes' third-generation models—developed from a recognition of the shortcomings of previous ‘product-process’ models, akin to Best and Holmes' first-generation models.
In this paper, we develop this approach in relation to research use in education. First, we survey public policy system change literature to develop a set of system dimensions to help frame our understanding of research use models. We use these to examine four models relating to research use that are widely referenced in education, outlining to what extent these models adequately reflect system approaches.
We then apply these dimensions to the work of the Education Endowment Foundation (EEF), the UK's What Works Centre for Education, which aims to support research evidence-informed practice at all levels of the education system. Although other papers (Edovald & Nevill, 2021; Gough et al., 2018) discuss EEF's role in the system, this article is the first to focus specifically on its work to embed research-informed practices in regional school systems. We do so by re-examining findings of two ‘scale-up campaigns’ focused on mobilising research relating to the effective deployment of teaching assistants (educational support paraprofessionals) using the systems dimensions set. This provides one of very few detailed examinations of systems approaches to attempt to increase research use in education in the research literature currently. By using the systems dimensions set as an explanatory framework, the article aims to advance the field in understanding system change processes for research use in education.
In the final section, we consider the insights emerging from this analysis for the research-use system in education, and implications are discussed for the application of system-based approaches to promoting and studying research use in education and more broadly.
Before we move into the next section, we define some key terminology. In the field, a variety of terms are used for a set of closely related concepts: terms such as knowledge mobilisation, evidence use and research-informed practice all cluster around the relationships between research evidence and professional practice. In this paper we use the term research use to mean engaging with research evidence including both considering and acting to create changes in practice in education. This includes all of Weiss's (1979) widely used categories of research use: instrumental (direct application of research-informed practices, interventions and resources), conceptual (using research indirectly, in conjunction with other types of evidence to rethink and discuss approaches) and strategic (using research indirectly, to legitimise an approach or persuade others of its value and/or affirm an existing practice). We only use other terms for research use when directly citing other authors' work. Turning to the term ‘research evidence’, there are numerous definitions, from Stenhouse's (1975, p. 142) celebrated ‘systematic inquiry made public’, to a version used in a previous study of evidence-informed teaching (Coldwell et al., 2017, p. 10): ‘quantitative and qualitative research findings generated by external researchers; evidence reviews such as those produced by the Sutton Trust, EEF and John Hattie; external evaluations; and/or research produced by teachers/schools that is underpinned by rigorous and systematic enquiry’. The first of these perhaps lacks specificity; the second perhaps has too much. For the current paper, we draw on the second to define research evidence as findings and synthesis of findings from rigorous and systematic inquiry, hoping to capture that it can involve both direct and indirect use of findings, with an inclusive definition of research.
SYSTEMS APPROACHES IN SOCIAL POLICY RESEARCH: DEVELOPING A SET OF SYSTEMS DIMENSIONS
Systems approaches are widely used in social policy evaluation, development and analysis, often within a pragmatic frame aimed at drawing out insights into, for example, (geographically) local system change. This use of systems approaches has led to wide application across a range of policy areas, for example, as a response to repeated policy and practice implementation failures that are premised implicitly or explicitly on simple linear cause and effect models. These models do not take account of the complexity of change processes in and between systems that we go on to outline below.
- System components are structured interdependently and hierarchically. This interdependence can involve tight or loose coupling.
- Organisations in and linked to the system are also interdependent, drawing on permeable boundaries between and inside organisations.
- Systems work to produce outcomes, which can create feedback mechanisms, which can change the system.
- These processes operate holistically, and lead to emergent outcomes (related to the working of the system as a whole, irreducible to the working of the components within it).
- This may lead to stability in the system; or instability and change.
A third system approach is of particular importance for this article: understanding system change. The focus here is not on how systems behave or behaviours in systems, but on understanding how to create system change. Thompson et al. (2016) associate this with the closely aligned approach of complexity theory, drawing on Cilliers' (2013, p. 38) definition, that ‘complexity is a characteristic of a system’. Phelan (1999, p. 238) identifies that ‘Several terms carry virtually the same definition in both theories, including system, emergence, dynamic, nonlinear, adaptive, and hierarchy. Both theories [systems theory and complexity theory] also share a belief that there are universal principles underlying the behavior [sic] of all systems.’ Thompson et al. (2016, p. 13) draw on this perspective to argue that ‘systems theory is focused on identifying and optimizing relationship characteristics whereas complexity is focused on understanding what influences interactions so that conditions may be created to support further interactions. In essence, complexity is more exploratory whereas systems theory is more confirmatory.’
In this paper, we draw on all of these perspectives to analyse the approaches and models used in relation to research use in education. To do so, we develop a set of what we term ‘systems dimensions’, which we lay out in the remainder of this section. This framework has been developed by drawing from two key sources. First, we utilise prior published work; in particular, previous categorisations in the ‘systems approaches’ fields outlined above, as well as work of relevance to systems conceptualisations in education (especially school leadership and school improvement/change), social evaluation and research use. Secondly, we have drawn on our own experiences as researchers, evaluators and system actors (we have each taken a range of roles in intervening in educational systems to influence change in relation to research use) to develop the categorisation in the context of research in education.
The first dimension to consider is the definition of, and perspective on, the system(s) in question. Systems may be defined in relation to geography (local, regional, national setting); focus (in our case, education; but often aspects of health); organisation or setting (some focus on schools, for example); groups involved (children; specific groups of people); or some combination. Further, taking a system approach does not simply focus on a single system: the focus is on sets of interconnected systems (Haynes et al., 2020). An example in education is the novel approach to understanding professional learning of Opfer and Pedder (2011), who discuss the interactions between change processes in relation to different activity systems, although their focus is centred on the teacher rather than the wider system as we look at it here. In this model, the teacher activity system (focused on teacher professional learning) occurs within three nested subsystems: the teacher, the school and the learning activity system.
A related important issue is the perspective from which one views the system. This is apparent in particular when examining educational systems and their representation in various system models, as we go on to consider in the next section. In education research-use systems, perspectives could focus on the research producer, the policy maker, the research user (teacher, school leader etc.) or the system researcher.
A second dimension relates to system agents or actors and the activities they engage in. Clearly, the work of knowledge producers (and production), and users (and use) are central. We also draw attention to two other roles and related activities here. The first is leadership: key actors in relation to educational systems are leaders, given the primacy of the school as organisation and the associated importance of senior leaders within schools. Boylan (2016) discusses related system leadership activities as ‘a form of leadership practice or orientation’ (Boylan, 2016, p. 61), which Boylan (2018) further develops as a form of adaptive leadership that requires understanding of the features and complexity of particularly local education systems, a style with an orientation towards and understanding of complexity in the local system.
The other role and activity identified as important within both systems and evidence-informed practice literatures is brokerage. There are numerous definitions (Rycroft-Smith, 2022); one we find helpful is Farley-Ripple et al.'s (2017) description of research brokerage as ‘a dynamic and complex set of actors, activities and motivations, within which research is exchanged, transformed, and otherwise communicated’. In a comprehensive review of what she refers to as knowledge brokering, Rycroft-Smith (2022, p. 41) argues that ‘linear and instrumental models are of little use when it comes to the kind of complex, large-scale and wicked problems we see in educational contexts. For those, we require collaborative entanglements—in other words, systemic approaches.’ In this context, in earlier work (Maxwell, Willis, Culliney, Coldwell, Demack, et al., 2019; Maxwell, Willis, Culliney, Coldwell, & Reaney, 2019; Maxwell, Coldwell, et al., 2019) we discuss the role of system brokerage, outlining a set of attributes of system brokers including being ‘professionally credible and skilled adult learning facilitators’ and being able to ‘provide challenge as well as support’ and ‘have strong communication, interpersonal and organisational skills’. As identified below, given that systems are nested, so brokerage in system contexts may itself be nested, at the overall macro system level (brokerage to change the system overall), at the meso level (for example, within smaller geographical areas) and at the micro level (typically at the organisational, for example school, level).
A third dimension is a set of characteristics of systems, which are outlined in the literature. First, systems exhibit features of complexity. Working within a policy evaluation frame, Walton's (2016) approach provides a set of core features of complexity to be aware of in open systems as follows: non-linearity; emergence; adaptation; and uncertainty. These features all apply to complex systems using Rogers' (2008, p. 32) distinction between the simple (single linear paths from inputs to outcomes), complicated (with several causal strands) and complex (recursive and emergent). System theory and complexity intersect in the classic form of the Complex Adaptive System; the theory of which draws together elements of complexity and system theory to understand how self-organising systems develop (Buckley, 2017). In education, there are echoes of this approach in the series of think pieces that underpinned the development of the ‘school-led self-improving system’, influenced by the work of Hargreaves (2014) for the then National College for Teaching and Leadership.
A further system characteristic is nesting or layering. The example above from Opfer and Pedder of a set of nested sub-systems in considering teacher professional change is a good example here. Linking to this discussion Thompson et al. (2016) drawing on complexity theory, provide a useful set of attributes they used to consider research that uses a complex adaptive systems approach, which can be useful to consider (Table 1).
| Parent attribute | Referent attributes |
|---|---|
| Connections | Connections, Relationships, Interconnections |
| Communication | Communication, Conversation, Information Flow, Information Exchange, Interactions |
| Learning | Learning, Sense Making, Learning Culture |
| Adaptation | Adaptation, System Adaptation, Innovation |
| Diversity | Diversity, Cognitive Diversity, Diversity of Information, Diversity of Perspective, Diversity of Views |
| Equilibrium | Equilibrium, Disequilibrium |
| Agents | Agents, Agents in a System, Input from Agents |
| Unpredictability | Unpredictability, Uncertainty, Levels of Certainty |
The final dimension we highlight is system change. Modelling of change processes in the system has been undertaken in a number of fields. Of particular relevance is organisational change literature in education, which has increasingly focused on integrating system approaches in recent years. For example, Fullan's work has focused on this area for some time, identifying a set of drivers for change in education systems as: capacity building, group quality, pedagogy and explicitly systemic approaches (as opposed to drivers of accountability, the individual, technology and fragmented approaches) (Fullan, 2011). In his most recent work, Fullan (2021) updates this list to become: Wellbeing and Learning, Social Intelligence, Equality Investments and Systemness (sic). Within some of this literature (for example, the work of Hargreaves, 2014 outlined above) researchers may assume a ‘developmental’ or ‘maturity model’ (assuming there is a move from less to more mature systems) or there may be no such assumption, and change is simply seen as movement without assuming a move from less to more mature.
Linked to this work on system change, at the cutting edge of systems research is the development of understandings of system change causal processes to help understand change within and between complex systems. Much work has been done here in relation to realist evaluation in particular. Realist perspectives consider that the social world should be understood as external to individuals and subject to complex change processes, which can only be partly uncovered via empirical research, therefore requiring theorisation of such processes to underpin empirical enquiry (see, for example, Pawson & Tilley, 1997). In this context, for example, Shearn et al. (2017) take a three-level model of system change. First, they draw on Archer's (1995) morphogenetic approach at the macro level: a cyclical model suggestion that social structures and cultures constrain and influence agency (and so actions), which in turn cement or modify structures and cultures. Secondly, they draw on normalisation process theory (May et al., 2009) at the meso level, a theory that ‘describes how organizations change to adopt new practices’ (Shearn et al., 2017, p. 6) via a set of ‘normalisation’ constructs (coherence, cognitive participation, collective action and reflective monitoring). Finally, the Michie et al. (2011) COM-B model is used at the micro level to explain behaviour change (B) as linked to Capability (C), Opportunity (O) and Motivation (M). Other work in this area incudes Langer et al.'s (2016) six mechanisms linked to evidence-informed decision-making.
These system dimensions are summarised in Table 2, and the key questions they raise for researchers in thinking about the systems they are studying. It is important to note that these dimensions are neither exhaustive nor definitive: other researchers drawing on other literature, empirical work and system practices may well organise these differently, and potentially add new categories. Furthermore, although we present them separately for analytical purposes, they are clearly inter-related and partially overlapping, as we make clear in The Education Endowment Foundation's Scale-up of Research Evidence on the Effective Deployment of Teaching Assistants. So, this is a starting point which we would hope would be developed as other researchers conduct further studies in this field. With the caveats in mind, in the next part of the paper, we consider these dimensions in relation to evidence-informed teaching in the English system. Before we do so we examine a range of models and examine to what extent they align with systems models.
| Dimensions of the system | Categories | Key questions to consider |
|---|---|---|
| 1. Definition and perspective | ||
| (a) Defining the system | Focus; issues; groups; organisations; place |
What are the key systems and how are they defined in the case under study? How do they inter-relate? |
| (b) System perspective | Research producer, user, policy-maker, system researcher (etc.) |
What perspective is the system viewed from? |
| 2. System actors and activities | ||
| (a) Actors | Leaders; brokers; system knowledge producers; users |
What are the actors and the roles they fulfil in the system under study? What are the key relationships and forms of relationships between system actors? |
| (b) Activities | Leadership, brokerage; production, use at macro, meso and micro levels |
What are the activities being undertaken at the overall system level; at a regional level within the system; at the organisational level? By whom and how? What are the interrelationships, resulting synergies and tensions between activities at different system levels? |
| 3. System characteristics | ||
| (a) Complexity | Simple; complicated; complex | Is the system simple (involving single linear paths), complicated (involving multiple causal strands, organisations and mechanisms) or complex (recursive and emergent) |
| (b) Nested, layered | System change occurs at different levels and layers which may be nested | How is layering and interactions between layers relevant in the system under study? |
| (c) Features of CAS (Thompson et al., 2016) | Connections/relationships; Communication; Learning; Adaptation; Diversity; Equilibrium; Unpredictability | To what extent are these features relevant, and how might they be operationalised, in the system under study? |
| 4. System change | ||
| (a) Drivers | Drivers to enable system change (e.g., capacity building, group quality, pedagogy, well-being, social intelligence, equality, systems thinking) |
What drivers are being used to create system change? How do they inter-relate, and are they mutually supportive? |
| (b) Maturity orientation | Maturity orientated model; no maturation assumptions in model | Is the system assumed to move from less to more mature or is there no such assumption? |
| (c) Causal change processes and mechanisms | Causal change processes within and across system layers | What causal mechanisms are considered to underpin change processes in the system under study? |
APPLYING A SYSTEMS LENS TO RESEARCH USE
Developing system-based approaches to research use
Here we further interrogate the literature on research use, building on our introduction in section 1 of Best and Holmes' (2010) ‘three generations’ of system approaches. Researchers have recognised a series of limitations of traditional, linear research-use approaches, centred on evidence that ‘passive’ dissemination of research alone is unlikely to impact significantly on practitioners' behaviours (Best & Holmes, 2010; Levin, 2013; Nutley et al., 2007). In terms of dissemination, communicating research is more likely to impact on decision-making if it is also combined with strategies that generate opportunities, motivation and skills for engaging with that research (Langer et al., 2016). In terms of applying research, implementation is increasingly recognised as a social process, with interactions and social influence being key factors in determining how evidence gets used and applied in practical settings (Domitrovich et al., 2008; Dyssegaard et al., 2017).
In this context, it is unsurprising that interactive approaches, which support direct engagement and dialogues around research evidence have been of particular interest over the last 10 years (Best et al., 2008). In these relationship models, knowledge is drawn from multiple sources (e.g., practice, theory), not just from the researcher, with an emphasis on collaboration when both generating and using knowledge. Until recently, there has been relatively little research on the nature and impact of effective research-practice partnerships (Langer et al., 2016), although insights from research are beginning to emerge (Gu et al., 2020; Henrick et al., 2017; Metz et al., 2020).
Turning next to the importance of relationships (Best and Holmes, 2010's second-generation approach), Best's influential work on ‘knowledge to action’ systems captures this principle well, in highlighting that activities and processes to encourage research use do not work in isolation, but sit within complex systems outside of research, with multiple actors and influences, each with their own priorities, motivations, processes, time-scales and world views (Best & Holmes, 2010). One of the features of such a systems model is that the degree of research engagement is a function of the effective integration with external organisations and the systems in which they operate.
Although the end objective may be the instrumental use of evidence at scale, Meagher and Lyall (2013) argue that an effective research strategy also needs to value activities with ‘non-instrumental’ impacts. These are impacts which may not immediately results in changes in behaviour, but build latent potential for changes to occur in the future (aligned with Weiss's (1979) ‘conceptual’ research use as noted in Introduction). Non-instrumental impacts include: building capacity and networks that are reused in future activities to encourage research use, supporting changes in culture and attitudes, as well as changes in people's thinking that do not immediately manifest on their behaviours (conceptual impacts). A systems-based approach, Best and Holmes' (2010) third-generation approach, to research use encourages a broad range of these impacts. It does so by recognising that building capacity in the wider education system can create readiness for future research engagement and use.
Examining research-use models using a systems perspective
A range of different visual models have been used as explanatory frameworks for research-use systems in education. In this section we apply the dimensions of systems, outlined in Table 2, to review some of these models, exemplify the different system dimensions and examine how they can apply to research use in education. Further, this analysis allows us to consider the extent to which these commonly used models were reflected in the example on TA deployment presented later in the paper.
Four models (described as models 1–4 and presented in Figures 1-4, respectively) have been selected that have been previously used as frameworks to explain research use in education policy and practice (see Figure 1). The models are widely cited in policy reports and have been used to inform strategic decisions of organisations that are seeking to further the development of evidence-informed policy and practice, including national Knowledge Brokerage Organisations (KBO) such as the UK's Education Endowment Foundation (Cabinet Office, 2018; Waddell, 2021). The models have also been selected in order to provide different perspectives on research use in education and illustrate a range of ways in which research-use systems can be represented. The different perspectives and emphases of the models enables us to show how the dimensions of systems, outlined above (Table 2), apply to research-use systems in education. The four models are briefly outlined below, before we discuss them in more depth in Discussion:
Model 1. What Works Network evidence ecosystem model (Cabinet Office, 2018), adapted from the MAGIC Digital and Trustworthy Evidence Ecosystem (MAGIC, 2016)
Model 2. Campbell and Levin model for knowledge mobilisation in education (Campbell & Levin, 2012)
Model 3. EPPI Centre's Research-use Ecosystem (Gough et al., 2021)
Model 4. Quality Use of Research Evidence (QURE) framework (Rickinson et al., 2020)
A conceptual framework, developed as part of the Q Project in Australia (Rickinson et al., 2020), to ‘define and elaborate what “quality use of research evidence” might mean in education’. The model is intended as a resource to improve the use of research evidence within and across schools and school systems. It is being used to design professional learning programmes that build capacity for evidence-informed practice in schools across Australia.
Discussion
Each of the four identified models exemplifies some features of system approaches in that they represent interdependent relationships between different elements and activities with the research-use system. Yet, when the ‘dimensions of systems’ are applied to examine different research-use models we see that each model often focuses on overlapping, but different issues, groups and organisations. For example, in models 1–3, research production is emphasised explicitly, either as one activity (e.g., models 2 and 3) or broken down into fine-grained constituent activities such as primary research and secondary synthesis (e.g., model 1). In these models, the research producer(s) is/are represented as a discrete organisation and actor in the overall research-use system. This highlights a further characteristic of these three models, in that the use of research is framed within the wider context of research production and mobilisation. The QURE model (Figure 4) implies the production of evidence through the central core component of ‘appropriate research evidence’, although the way in which evidence is generated is not considered or represented explicitly, meaning the emphasis of the model is more on the use of research and the characteristics of the research-user.
The selection of elements in the system models is influenced by perspective—where one is positioned in relation to the overall system. For example, in models 2–4, the research-use activities are represented in the context of the wider systems outside of evidence use (e.g., policy, accountability, funding) and so can be described as nested systems. The tight focus in model 1 on the research ‘production to use’ process is perhaps understandable given its purpose in describing activities of the UK government's What Works Centres (including the Education Endowment Foundation); however, the omission of the wider context and representation of systems outside of research suggests that evidence use operates in isolation. This omission is significant, as What Works Centres are not always explicit about how their strategic decisions are informed by an analysis of the existing system(s) in which they are intervening (Gough et al., 2021).
The model that is selected to represent research-use systems, and the elements it contains, also has an impact on which, and how, different actors within the system are represented. For example, in model 2, intermediaries that mediate between research production and use are represented explicitly as actors in the research-use system, whereas in other models, research mediation and brokers are implied. Research intermediaries are third party brokers that bridge between the creation of research knowledge and its use. Building on the description in Systems approaches in social policy research: developing a set of systems dimensions, this can involve, for example, communicating evidence, translating research into more accessible forms, supporting implementation, building capacity, and embedding research use activity into existing systems. Research brokers are varied, widely placed and loosely characterised, including policy-facing bodies such as the media, think tanks and lobby groups, through to practice-facing brokers such as professional organisations, practitioner networks, private companies and local government. The representation of brokers as discrete actors in system models is significant, as it represents research use as an active, mediated and context-dependent process.
Although model 4 does not represent brokers and intermediaries, it does identify individual teachers and leaders as specific actors in the research-use system, whereas the other models treat the ‘user’ as a single entity. This fine-grained focus on the research user represents a wider characteristic of model 4, in that it unpacks research use in schools, and the interconnected components that contribute to research use, with greater specificity than in the other models. This further illustrates the way in which perspective influences what is privileged within system models.
As discussed in Systems approaches in social policy research: developing a set of systems dimensions, one characteristic of systems is that they feature different degrees of complexity, including simple (involving single linear paths), complicated (involving multiple causal strands) and complex (recursive and emergent) systems. Model 1, describing the activities of the What Works Centres, is the simplest of the models in that it describes evidence ‘production to use’ as a single circular linear path. The three other models represent evidence use with greater complexity, suggesting there are interdependent and context-specific relationships between multiple causal factors. For example, in models 2 and 3, two-way arrows are used to illustrate connections between activities and actors in the research-use system, which are mediated through relationships.
Nevertheless, although these non-linear models capture complexity more effectively, they still struggle to capture the dynamic, evolving nature of interactions between system elements, which is a feature of complex adaptive systems. For example, model 4 effectively represents factors and actors at different levels of a research-use system—individual, organisational, wider system—but it does not fully capture the dynamic relationships and change processes between those levels. Indeed, the representation of causal processes and mechanisms between system elements is notably absent across all four research-use models. As such, the adaptive, unpredictable and emergent nature of complex systems is generally implied in these models rather than made explicit.
Table 3 provides a summary of how each of the four models relates to the system dimensions in Systems approaches in social policy research: developing a set of systems dimensions. For clarity, the key relevant dimensions for research use systems are addressed rather than every system dimension.
| Model | Definition of the system | System perspective | System actors | System activities | Complexity | Nested, layered | Connections/ relationships | Maturity orientation | Causal change processes and mechanisms |
|---|---|---|---|---|---|---|---|---|---|
| 1. MAGIC Evidence Ecosystem (Cabinet Office, 2018) | Research production to use across different sectors | Research intermediary | Actors not shown explicitly but implied in the activities | Research production to use activities shown | Simple | None | Simple one way connections | Maturity is implied through the cyclical representation of activities | Causal mechanisms are assumed in the arrows but not described |
| 2. Campbell and Levin (Campbell & Levin, 2012) | Research use system in education | System researcher | Actors shown explicitly. Wider system actors alluded to | Activities shown for research production, mediation and use | Complicated | Yes—research use is situated within social and institutional context | Dynamic, two-way relationships suggested in the arrows. Nature of the relationships not shown e.g., trusted | Scope for maturity but it is not necessarily implied | Causal mechanisms are assumed in the arrows but not described |
| 3. Gough et al. (Gough et al., 2021) | Evidence ecosystem in the context of wider education systems | System researcher | Actors not shown explicitly but implied in the activities. Wider system actors alluded to | Activities shown across production, engagement and use | Complicated with some features of complexity | Yes—research use is situated within wider systems and contexts | Dynamic, two-way relationships suggested in the arrows. Nature of the relationships not shown e.g., trusted | Scope for maturity but it is not necessarily implied | Causal mechanisms are assumed in the arrows but not described |
| 4. Quality Research Use in Education (QURE) (Rickinson et al., 2020) | Research use system in education | A user-based perspective | Specific actors not shown although level of actors is shown e.g., individual, organisational | Specific activities not shown, although overall model is on research use | Complicated with some features of complexity | Yes—system layers shown relating to micro (individual), meso (organisational), macro (system-level) | ‘Relationships’ shown as a specific element of the system. Connections and interactions between levels in the system is implied | Scope for maturity but it is not necessarily implied | Causal mechanisms are described in the model |
In summary, all four models suggest that research use is influenced by multiple interacting factors and activities. Yet, although all models focus on some dimensions of systems, none of them look at all, and what is privileged in each system model is influenced by the perspective of those developing and using the models. If visual models are being used to inform decision making within organisations trying to support research use (e.g., What Works Centres), the choice of model could influence how the system is understood and so where effort is focused. For example, if research brokers are not represented explicitly in the model, users could feasibly overlook brokerage activities when considering options on how and where to support research use activities. Care should therefore be taken to recognise the strengths and limitations of any particular model and those dimensions of the system that are represented and missing.
In the next section, we introduce the development of the Education Endowment Foundation's research-use strategy, to set the context for the case analysis that follows.
THE EDUCATION ENDOWMENT FOUNDATION AND THE SCALE-UP CAMPAIGN
The Education Endowment Foundation's developing research-use strategy
The Education Endowment Foundation (EEF) was established in 2011 with a substantial endowment from the UK government (£125 million), as an independent charity ‘dedicated to breaking the link between family income and educational achievement’ (Education Endowment Foundation, 2022a). In 2013, the EEF was designated as one of nine UK What Works Centres, a network of intermediary organisations across different areas of policy and practice (e.g., healthcare, crime reduction, local economic growth), expected to ‘create, share and use high quality evidence for decision-making’ (Cabinet Office, 2018).
EEF's initial approaches to research use centred on two outputs: the Teaching and Learning Toolkit Education Endowment Foundation, 2022b) and the publication of reports from EEF-funded programme evaluations (Education Endowment Foundation, 2022c). The most likely routes through which schools would engage with these resources has been through press and media coverage, social media and newsletters, accessing the EEF website, and presentations by EEF staff at conferences and events. As such, EEF's early approach tended towards traditional, linear dissemination activities, aligned with what Best and Holmes (2010) refer to as first generation strategies.
Recognising the limitations of these strategies (see Developing system-based approaches to research use), the development of partnerships between researchers, practitioners and intermediaries became a key objective in developing EEF's research-use strategy. A further consideration has been how these relationships sit and work within external structures and systems, with the EEF sitting within a range of wider systems. These include policy, school improvement, funding, accountability systems and so on, that operate at national (macro), regional (meso) and organisational (micro) levels. Models 2 and 3 represent research-use activities and relationships sitting within wider systems and contexts.
- multi-stranded strategies working at the macro, meso and micro levels of change (e.g., practitioner, school, regional and national policy levels)
- co-creation of knowledge and resources between researchers and practitioners
- actionable guidance, with support for implementation
- ‘active’ forms of KM, involving interactions and face-to-face engagement
- working with practitioners as partners in mobilising knowledge and integrating evidence into local contexts
- building capacity, relationships and networks to share and use evidence
- the importance of distributed, collaborative leadership and accountability throughout the system.
An important consideration, as a national organisation, has been determining how to provide intensive support for engagement and implementation while maintaining broad reach and impact. Traditionally, more interactive approaches to increasing research use in education have been applied at a much smaller scale, typically working with relatively small clusters of schools (Sharples & Sheard, 2015). As such, the aim has been to include both light-touch and more intensive elements, which interact across different system ‘levels’, for example, national, regional and school level.
An example of an EEF research-use strategy is the organisation's mobilisation ‘campaigns’: multi-stranded initiatives to scale up evidence-based guidance on specific school improvement issues. These campaigns included Improving Literacy at Key Stage One and the focus of the next section: Making Best Use of Teaching Assistants (Education Endowment Foundation, 2022d). Underpinning each campaign is an EEF Guidance Report, with additional resources made available to help schools apply the guidance in their context (Education Endowment Foundation, 2022e). These guidance reports and resources are mobilised through a range of channels and strategies, tailored for each campaign. These strategies include direct communications to schools; press/media engagement; sector-led training and coaching; influencing policy; activating stakeholder networks; and scale-up of evidence-based programmes. Campaigns have both national and regional activities, designed to interact and reinforce each other. For example, the Making Best Use of Teaching Assistants guidance report and resources have been referenced in national communications from Unison (the main trade union for teaching assistants), and from OfSTED, the national school inspection body (OfSTED, 2016). In the next section, we focus on the regional element of this campaign.
THE EDUCATION ENDOWMENT FOUNDATION'S SCALE-UP OF RESEARCH EVIDENCE ON THE EFFECTIVE DEPLOYMENT OF TEACHING ASSISTANTS
Context and methodology
Two regional pilots were initiated and steered by the EEF as one element of the broader national campaign to promote the use of the recommendations set out in the EEF guidance document on TA deployment: Making the Best Use of Teaching Assistants (Sharples et al., 2016) in primary and secondary schools across England (see Figure 5). Both pilots shared a common focus on enabling schools to embed research-informed evidence on TA deployment by establishing regional support to implement the seven recommendations set out in the EEF guidance document (see Figure 5). Although this was the core aim of the South and West Yorkshire pilot, in Lincolnshire the use of research evidence on TA deployment was perceived as an initial step, which could support, in the longer term, the broader intention, of creating a sustainable network of ‘evidence-ready’ schools in the region. The South and West Yorkshire pilot spanned the 2016–17 academic year and engaged 432 schools. The Lincolnshire pilot engaged 283 schools across the 2017–2018 academic year.
In this paper we re-examine the evaluation findings of both pilots through a systems lens to illuminate the nature and complexity of scaling up research use in schools, and draw out areas requiring greater attention if the scale-up of research use is to be successfully achieved. It is not our intention to methodically examine the finding of the pilots in relation to each system dimension in Table 2, since, as indicated in Systems approaches in social policy research: developing a set of systems dimensions, these dimensions are highly interrelated and, in some instances, overlapping. Instead, we use the system dimensions, as appropriate, as conceptual tools to trace a strengthening application of systems thinking over time to the design and facilitation of the pilots and to develop explanations for the successes and limitations in differing approaches to scaling-up research use.
Concurrent mixed methods evaluations were undertaken for both pilots, together with an impact evaluation to assess impact on pupil attainment in the South and West Yorkshire pilot. Full details of the evaluation methodologies can be found in Maxwell, Willis, Culliney, Coldwell, Demack, et al. (2019), Maxwell, Willis, Culliney, Coldwell, and Reaney (2019), Maxwell, Coldwell, et al. (2019) and Sibieta and Sianesi (2019). Our interpretations for this paper are drawn from an examination of the qualitative findings of both evaluations. Below we provide a brief summary of the qualitative data collection and analysis methods that generated these findings. To provide context for the qualitative findings, we also summarise the findings of the impact evaluation in South and West Yorkshire and change-over time analyses undertaken on data from pre- and post-pilot surveys in both evaluations.
The qualitative findings were constructed by applying an adaptive theory analytical approach (Layder, 1998) to data generated through: longitudinal semi-structured stakeholder interviews (South and West Yorkshire—15; Lincolnshire—17 plus a focus group with 13 participants); school case studies comprising semi-structured interviews or focus groups with senior leaders, teachers and TAs and review of school documents (South and West Yorkshire—14 cases, including data generation from 20 senior leaders, 43 teachers and 52 TAs; Lincolnshire—2 cases, 14 participants in total); and 15 semi-structured telephone interviews in Lincolnshire with the member of school staff responsible for leading the implementation of the project. ‘Adaptive’ conveys that ‘the theory both adapts to, or is shaped by, incoming evidence while the data itself is simultaneously filtered through, and is thus adapted by, the prior theoretical materials’ (ibid., p. 5), allowing deductive and inductive approaches to be combined. The qualitative findings provide rich, in-depth insights into the dynamic nature of the research-use system, as well as the nature and interactions of multiple factors related to research production, mediation and brokerage, use within schools, the wider national education system and regional school improvement systems. When viewed through a systems lens, the qualitative findings offer plausible explanations for the overall modest success and internal variability of the pilots in achieving the intended longer-term outcome of raising pupil attainment and the medium-term outcome of changing TA practices in schools. The key findings in relation to these outcomes are briefly summarised before returning to examine the qualitative findings. The assessment of the impact of the South and West Yorkshire pilot (Sibieta & Sianesi, 2019) on pupil attainment found that Key Stage 2 (KS2) English test scores of the 43,000 pupils across all schools in the region had improved by 0.03 standard deviations compared to a synthetic control group. No difference was found in relation to the KS2 mathematics tests and there was no empirical evidence to suggest that attainment in English or mathematics was higher in South and West Yorkshire schools who had participated in the pilot than in non-participating schools in the region. Caution is needed in interpreting these impact findings due to: substantial changes in KS2 tests and curriculum that may have biased the findings in ways which are unpredictable; the ‘active control’ of a high-profile national TA scale-up campaign; as well as the relatively short time that had elapsed between pilot completion and impact measurement.
There was modest evidence from survey and qualitative data in both regional pilots that practices in participating schools became more closely aligned with the EEF recommendations on TA deployment over the duration of the pilot. A pre- and post-comparison survey undertaken in Lincolnshire, using a propensity-matched sample, indicated alignment to some, but not the majority of, the EEF recommendations increased to a greater extent in participating Lincolnshire schools than in comparison schools. In South and West Yorkshire, where a comparison survey was only administered post-campaign, very limited survey evidence was found of reported practices in participating schools being more aligned with the EEF recommendations than in comparison schools. Interpretation of the survey and qualitative findings should take account of the methodological limitations of the ‘active control’, and particularly for South and West Yorkshire, the absence of the pre-pilot comparison survey. Of note is that in South and West Yorkshire there was some, albeit fairly limited, variation in the extent of the alignment to the EEF recommendations between the local authority areas that comprise the region, where, as we go onto explain, differing approaches were taken to scaling up research use.
Applying the systems dimensions to research-use pilot findings
Overview
Applying the system dimensions in Table 2 led us to bound the systems of interest (dimension 1) to the set of complex interactions and relationships (dimension 3c), drivers (dimension 4a) and consequent causal changes (dimension 4c) that were intentionally facilitated and/or emerged in pursuit of embedding the research object—that is, the recommendations in the EEF's guidance report, Making the Best Use of Teaching Assistants (see Figure 5) and associated tools and resources—in the two pilot regions.
These interactions, relationships and causal processes were found to occur within and between nested systems (dimension 3b) that operated at the macro (national system), meso (regional) and micro (school) levels. EEF project leaders were the key actors (dimension 2a) who primarily engaged within and across the national and regional level systems. Regional stakeholders primarily engaged within the regional system and some also engaged across the regional and school level systems boundary. School leaders and the designated school implementation leads also engaged across the regional- and school-level systems boundary as well as within the school-level system. Teachers' and TAs' engagement was usually confined within the school-level system.
To provide a deeper insight into the operations of the various system dimensions we apply the conceptual lens of brokerage to examine the TA project as a change intervention within a nested system. Applying Farley-Ripple's definition of brokerage (set out in Systems approaches in social policy research: developing a set of systems dimensions) enables exploration of the complex relationships, interactions, activities, drivers and ensuing causal processes within and across the nested system, regional and school levels.
Before turning to brokerage, it is important to consider the role of a key non-human actor in facilitating research use, the research object. The findings from analyses of the regional brokers and school staff interviews and focus groups indicate that the following qualities of the EEF guidance and associated resources supported the mediation of research use: namely that they were provided by a trusted brand (EEF) and were perceived to be credible, convincing, accessible, user-friendly and based on robust research. However, in line with other research (Langer et al., 2016), findings at baseline in Lincolnshire, over a year after the guidance had been in the public domain, indicated that the production and general promotion of the research object was insufficient, on its own, to trigger widespread changes of practice in schools. In both pilots, interactions and relationships and brokerage of research use within and between the system, regional and school level systems were found to be necessary to drive implementation of the TA guidance at scale.
A further consideration in interpreting the findings below is the perspective from which the system was viewed (dimension 2). Inevitably as the pilots were initiated and steered by EEF, the EEF project leaders were highly influential in initially shaping the perspective from which they and other actors viewed the system as a whole. As the findings below outline, EEF's perspective evolved over time. The findings reveal the interplay of, often implicit, perspectives on the system from the regional stakeholders and school staff within their interactions with each other and EEF project leaders.
System level brokerage
The evaluations identified EEF's role in facilitating research use as one of ‘system broker’—working both within the national education system and directly at a regional level to stimulate and support schools' use of the EEF guidance. The concept of system brokerage has not, to our knowledge, previously been explored in depth. EEF stakeholders used the metaphor of ‘orchestration’ to describe the nature of their system brokerage activity during much of the period that the pilots were operating. As we will go on to discuss, in the later stages of the Lincolnshire pilot this internal description evolved to an analogy of a ‘jazz band’.
Turning first to EEF's system brokerage within the national education system, a substantive national campaign was undertaken to promote widespread engagement with the EEF TA guidance. This including mailing hard copies of the guidance to all schools in England, emails promoting the guidance to schools, local authorities and a wide range of national education organisations, over fifty presentations, and direct engagement education policy-makers and key national stakeholders, such as Teaching Unions. As evidenced by the pilot evaluations and other research, the national campaign was highly influential in raising awareness of the EEF TA guidance and encouraging school to implement the guidance recommendations (Maxwell, Willis, Culliney, Coldwell, Demack, et al., 2019; Maxwell, Willis, Culliney, Coldwell, & Reaney, 2019; Maxwell, Coldwell, Willis & Culliney, 2019). There is some, albeit limited evidence, that EEF's relationships and interactions with national stakeholders helped frame an external environment that was more conducive to schools engaging with the EEF guidance report. For example, the guidance was referenced in national communications to schools from Unison (the main trade union for teaching assistants), and from OfSTED, the national school inspection body. However, some school leaders in both pilots still reported that they felt constrained in implementing research-informed practices due to the accountability regime imposed by government, OfSTED requirements and conflicting national education priorities, indicating an inherent tension between the national and school level systems that was not resolved during the pilots.
A further way in which EEF's system brokerage at a national level influenced conditions and activity at a regional level was through their ability to facilitate linkages between other system-level actors and the regional activity. For example, they provided training inputs by an Ofsted inspector, who discussed the EEF's TA guidance in the context of the national accountability system, and researchers involved in generating the evidence on effective TA deployment. As we outline later, EEF were also key actors in the ‘translation’ of the research evidence to resources for practice.
We move now to EEF's system brokerage within the pilot regions. As a system broker initiating the pilots, it is perhaps unsurprising that EEF's perspective on research use was highly influential in shaping the design of each pilot. Early interviews with EEF stakeholders, as the South and West Yorkshire pilot was being developed, indicated that systems approaches informed some aspects of the project, for example coordination of activity within and across the national and regional systems and co-construction of approaches and resources. However, a mainly linear conception of research use, broadly aligned to model 1, underpinned the approach to regional scale-up, indicating a system perspective driven by EEF's role as a research provider seeking to ensure that the evidence is taken up by schools. This led EEF to implement a ‘commissioned model’ of scaling up research use, whereby EEF contracted practice-based intermediaries, designated as advocacy providers, in each local authority area within South and West Yorkshire to design and deliver programmes of training and support to schools they recruited. EEF steered and supported advocacy providers in their approaches, with a particular focus on ensuring fidelity to the TA guidance. EEF system brokerage was primarily channelled through the relationship with the advocacy providers, with little evidence of EEF aiming to understand, and interact with, the existing school improvement systems operating within the region's local authority areas.
Early interviews with EEF stakeholders in the Lincolnshire project indicated that systems approaches were more central to the design of the pilot, with an emphasis on operationalising a more complex conceptualisation of research use, which included engaging with the regional school improvement system and influencing key stakeholders within the system. This led to the development of an ‘embedded model’ of research use. Implementation of this model began with EEF developing an understanding of how the existing regional school improvement system operated, and identifying influential regional leaders who could motivate schools and embed research use in regional systems (e.g., school improvement). The EEF then supported the establishment of a regional strategic steering group to oversee the research use scale-up development and delivery. The EEF provided steering, support and challenge to regional stakeholders. In addition, they offered guidance and resources to the Teaching School 1 commissioned by the strategic partnership to deliver training and support to clusters of schools throughout Lincolnshire. The evaluation found that this sustained system brokerage, prior to the establishment of training and support for schools, was a key enabler in building the motivation of regional leaders to underpin school improvement with research evidence, and facilitated the development of a regional infrastructure to support both the pilot and longer-term research use. Although it was outside the scope of both pilots to assess longer-term sustainability, there were clear intentions at the end of the Lincolnshire pilot to continue to embed research use throughout the regional school improvement system. In contrast, in the South and West Yorkshire pilot, where EEF had little engagement with the regional school improvement system, there were no indications that it had led to a stronger strategic focus on research use at a regional or local authority level.
EEF, as system broker, also interacted directly with school leaders in the micro-system in both pilots, through presentations at launch and training events, and used this engagement to emphasise the need to maintain fidelity to the evidence.
Findings from both evaluations indicate that the stimulus and influence that EEF was able to bring to bear, nationally and regionally, emanated from other actors perceiving EEF to be a trusted brand with high levels of expertise in relation to the research, with the energy and commitment to drive research use forward.
Regional level brokerage
Regional brokers in both pilots played a pivotal role in mediating research use at the school system level. The variation in schools' engagement and outcomes across the different advocacy provision in South and West Yorkshire, and the relatively modest outcomes across both pilots, highlights several moderating factors associated with regional brokers that impinged on schools' implementation of the EEF guidance. These findings indicated that brokering research use across the regional and school system boundary requires regional brokers that have: a deep knowledge and understanding of schools and the research-evidence; organisational and professional credibility; as well as programmes that are tailored to, and directly support the implementation of, the EEF guidance in participating schools. Co-construction of specific approaches to support in-school implementation by regional brokers and the designated school lead was reported by both groups to enhance the mediation of research use.
Regional brokers in Lincolnshire, who occupied leadership roles within the regional school improvement system, were central in establishing a county-wide research use infrastructure. Analysis of interviews indicates that their success was mediated through a collaborative approach to change in the regional school improvement system, their capability, commitment and drive to effect change, and credibility as trusted and respected school and/or regional system leaders.
It is important to note that the relationship between EEF as system brokers and the regional brokers was not solely unidirectional. Co-construction of approaches and resources was evident in both pilots and influenced on-going system brokerage, as well as adding to the research object resource bank.
School level brokerage and context
Actors within schools taking part in the pilots comprised head teacher; the staff member designated to lead the implementation of the TA guidance, if this was not undertaken by the head teacher; teachers; and TAs. The attributes of the school broker—that is, the staff member leading the implementation or the TA guidance and their positioning within the school hierarchy—was reported in both pilots to moderate the impact of the regional brokerage at the school level. The training and support provided by regional brokers was found to be most effective in facilitating schools to implement the TA guidance when the school broker was committed, enthusiastic and had the capabilities to effect change. In addition, school brokers had to have the authority within the school to drive whole-school change, either through occupying a senior leadership position or being supported, and enabled, to do so by the school's senior leadership team.
More generally, the school context and characteristics were found to be a crucial limiting factor in the implementation of practices aligned with the TA guidance. There was consistency in the finding across the pilots that implementation was more effective when senior leaders were committed and understood that implementing the guidance required whole-school change as well as more targeted activity such as TA training. In addition, senior leaders had to be prepared to allocate the staff time and resources necessary to effect sustainable change and ensure that there was a clear plan for implementation. School culture was also found to be pivotal in either facilitating or impeding implementation of the TA guidance. Implementation of the TA guidance was more successful in schools where all staff felt enabled to effect change and there was a ‘no fear of failure’ ethos, which resulted in teachers and TAs being open and responsive to change. A culture of trust, particularly between teachers and TAs, and effective teacher/TA communication were also associated with effective implementation. Implementation was more successful in schools where staff absence rates were low and there was stability over time in the leadership team. Different types of school faced different challenges. For example, small schools often struggled to engage with training activities provided by regional brokers as they were unable to release staff to attend events, whereas large schools sometimes found it difficult to implement the changes consistently across large staff groups.
The importance of the context of implementation, in this case the school, has been underplayed in much of the research-use literature, which initially tended to focus on the research object and its mediation, later moving on to take greater account of brokerage. However, the findings of the TA pilots that implementation of research-evidence into practice is dependent upon a conducive culture and requisite skills, mindsets, relationships and structures within schools, resonates strongly with implementation science literature (Vanderkruik & McPherson, 2017) and school improvement literature (Desimone, 2009; Fullan, 2011, 2021), as well as the recent evaluation of EEF's Research Schools Network (Gu et al., 2020). This in turn indicates that both system and regional brokerage can only be effective if they both pay attention to, and actively seek to support schools to make the changes necessary to create conditions favourable to the implementation of research-informed practices. As Gu et al. (2020) also found, scaling-up of promising practice had to be based on individual and bespoke support to develop staff capacity, build an ethos of evidence-based improvement, and help leadership to support new approaches.
The implications of the findings relating to schools' capacity for research-informed implementation are discussed in Discussion and conclusion.
Learning from the application of a systems lens
The re-examination of the TA pilot evaluations, through a systems lens, has highlighted the dynamic nature and complexity of the web of activities, interactions and interrelationships involving the research object (the EEF's guidance on TA deployment and associated resources), the key actors (system, regional and school brokers and other school staff), and other actors in the national, regional and research evidence production systems. Put simply, this represents a myriad of interconnected ‘moving parts’ that need to function optimally and be aligned. For example, conditions in the national education system and national directives need to support, and not impede, research use in schools. Understanding research use from this systems perspective may provide an explanation for the variable and modest impacts found in the pilots, since the evaluations indicated that impediments to embedding the TA guidance in schools' practices could be triggered at any point and level within the system, whether national (macro), regional (meso) or school (micro). As the evaluations show, impediments could be associated with weakness in approaches, activities and/or relationships. For regional and school brokers, impediments also emanated from their positioning within their regional education system, or their school and others' perceptions of their characteristics and prior experiences.
It could be argued that the South and West Yorkshire pilot, and particularly the omission of any deliberate engagement with the regional school improvement system, is illustrative of the early stages of maturity (dimension 6b) of a research-use system. It is debatable, definitionally, as to which side of the complicated/complex system (dimension 5a) borderline the pilot occupied. The presumption underlying EEF's shaping of the South and West Yorkshire pilot, that careful ‘orchestration’ of activity and actors would lead to the TA guidance being embedded in school practices across the region, aligns most closely with the concept of a complicated system. It excludes consideration of key features of a complex adaptive system, such as unpredictability, emergence and tipping points.
The Lincolnshire pilot, which included the broader intention of creating a research-use infrastructure embedded within the regional school improvement system, appears to be aligned more closely with the development of a complex adaptative system. The scope and time-limited nature of the evaluation did not allow exploration of the longer-term development of the Lincolnshire research-use system, when features such as adaptation, equilibrium and emergence may have become evident. However, the very early evidence on sustainability from the Lincolnshire pilot provides some indication of promise for an approach to scaling up researchuse predicated on initiating and supporting the development of a complex adaptive research-use system.
The re-examination of the TA pilot evaluations also illustrates how visual conceptual models, such as those presented in Applying a systems lens to research use, struggle to fully represent the dynamic nature of research-use systems. None of the models capture the evolving set of interactions between system activities and actors that featured in both TA pilots, nor the type of interactions that were acting as causal change mechanisms, for example trusted, coordinated, fluid relationships. Brokerage was taking place at different levels of the system (i.e., system, regional and school level) and in different ways, which again the models struggle to accommodate. Overall, the complex, adaptive and emergent nature of research-use systems, demonstrated in these pilots, is challenging to represent explicitly on a static model. This does not mean visual conceptual models aren't of value, rather that care should be taken, as noted at the end of Applying a systems lens to research use.
DISCUSSION AND CONCLUSION
Taking a systems approach to considering research use suggests that multiple activities and actors need to be aligned; and weakness in any area of the system, or interactions between system actors and activities across system levels, can potentially impede research use in schools. Applying a systems lens in this way creates opportunities to examine research-use systems and make research-informed decisions as to where best to intervene. Knowledge brokerage organisations, such as the Education Endowment Foundation, aim to facilitate the functioning of research-use systems and thus can be seen as interventions into pre-existing research-use systems (Gough et al., 2021). When considering where to intervene, a systems perspective can be used to assess the functioning of the existing research-use system and identify current impediments to research use, which can then inform the choice of intervention strategies (Gough et al., 2021). For example, if an identified limiting factor within the system is the capability of the user population to critique and interpret evidence, activities that build the capacity and skills of research users would be a worthwhile strategy to consider. Alternatively, it may be deemed that the relationships between different system actors is less than optimal—for example, research brokers and users—so putting effort into enhancing interactions between those actors could be beneficial.
The example of research-informed TA deployment, covered in this paper, illustrates a developing diagnostic application of a systems perspective/approach to the design and review of a research-use campaign by EEF. This development is most simply illustrated by comparing the set-up of the pilots. Little attention was paid to the operation of the regional system in South and West Yorkshire, whereas detailed early work in Lincolnshire focused on developing understanding of the regional system and establishing relationships with key regional stakeholders.
In turn, the qualitative findings from the two scale-up pilot evaluations have informed a diagnostic systems approach to a more recent regional scale-up pilot in Bristol. The findings of the pilots reported in this paper suggest that a common limiting factor to effective use of the research was the underlying capacity of the school to lead and manage change effectively. For example, implementation was sensitive to the school's culture, the way in which different stakeholders across the school were engaged (including leadership), and the ability to create shared understanding and buy-in for the changes in practice, amongst others (see Overview). As a result, attending to the implementation capacity of the research user, in this case the school, as a key actor in the system was deemed to be a viable route to increasing the scale up of research-informed practices.
In response to these insights, at the time of writing (mid-2022) the Bristol scale-up pilot has been set up by the EEF, which, in addition to providing training and support on TA deployment, is providing explicit training and support for schools on research-informed implementation. Qualitative findings from the previous scale-up pilots have been used to frame a set of common implementation ‘pinch points’ that schools can potentially encounter when applying the evidence on TA deployment, for example, ‘a lack of implementation leadership’. More attention is being placed on developing the necessary relationships and collegiate learning environments within, and between, schools, which is consistent with a systems-based approach to research use. For example, schools are being encouraged to use an implementation team for the project, with representation from staff across the school to create distributed leadership for implementation. Building on the promise of the Lincolnshire scale-up pilot, EEF have co-designed the project with regional actors including the local council and school leaders with the aim of creating a lasting research-use infrastructure that is embedded in the regional school improvement system.
Although this diagnostic approach to system intervention is considered here from the perspective of a system-level broker, like EEF, it could equally apply to other system actors such as research users. Obvious outputs from systems analyses are Theories of Change or logic models, which we discuss further at the end of the paper.
A further consideration that emerges from the EEF case study on TA deployment is the degree to which research-use systems self-organise, and the implications for system-level coordination and leadership. An observation from both pilots was that the research-use system operated through a complex set of interactions between different system actors at the macro, meso and micro levels that relied on trusted relationships and a mutual understanding of coordinated, but differentiated, roles. As such, the research-use systems did not self-organise (at least in the early stages) and required more active coordination and support to do so. This highlights the potential need for system leadership in research-use systems and raises questions as to what that entails and who is best placed to provide it.
Boylan (2018) suggests that system leadership requires understanding the features and complexity of the existing systems and a leadership style that is orientated towards understanding and embracing the complexity in the system. The examples discussed in this paper are consistent with that hypothesis, although they reveal questions on the precise nature of the systems leadership and where it sits in relation to the research-use system. As discussed, the metaphor of an orchestra conductor was initially used in the EEF pilot projects to exemplify the system leadership role, yet this characterisation implies there is a precise script that everyone adheres to, managed by a discrete, external system leader. In the EEF scale-up pilots on TA deployment, the role of system leadership was more fluid than this, moving from the system broker (EEF) in the initial stages of the pilots to the regional brokers as the research-use system matured. In this respect, system leadership was more embedded in the system than the orchestra conductor metaphor implies. The analogy of a jazz band feels more appropriate, as it better captures the dynamic and emergent nature of the music/system, the natural improvisation and fluid interactions at play, and the looser, integrated and more distributed nature of system leadership (i.e., the rhythm section in the band).
The systems dimensions framework developed and utilised in this paper has proved to be helpful in understanding the research-use system (and systems it interacts with) and their features, which has allowed us to derive the insights noted above; we found it a useful addition to the existing ‘systems approaches’ research literature in this regard. Further work is needed to both develop the systems dimensions framework, trialling it in other contexts and supplementing it with appropriate evaluation techniques. Further, complementary or alternative approaches to dealing with systems work could be developed to suit different examples (such as interventions of research schools in local systems). In earlier papers (Coldwell, 2019; Coldwell & Maxwell, 2018), we have discussed how evaluation frames such as logic models and theory of change (Connell & Kubisch, 1998; Dyson & Todd, 2010) struggle to deal with these features, although attempts have been made. For example, Rogers' (2008) approach to using programme theory to evaluate complicated and complex interventions advocates the use of annotated and circular visualisations, and we have built on Rogers' work to suggest multiple pathways ‘using a set of interlocking models at different system levels’ (Coldwell, 2019, p. 108). A further point relates the theorisation of causal change processes from a systems perspective. Working within a logic model frame, in the South and West Yorkshire pilot discussed above, we attempted to differentiate a set of sequenced processes relating to a campaign encouraging take up of the Guidance Report and the use of intermediary organisations to facilitate research use (Maxwell, Willis, Culliney, Coldwell, Demack, et al., 2019). As indicated in The Education Endowment Foundation's Scale-up of Research Evidence on the Effective Deployment of Teaching Assistants, this approach underplays the inter-related, emergent and adaptive features of system change. As we note in Systems approaches in social policy research: developing a set of systems dimensions, there are examples of more complex attempts to theorise change processes at different scales (e.g., Shearn et al., 2017); but there is much to do in developing this area. Building impact evaluations in the context of emergence and uncertainty within such frames is therefore a developing project.
In conclusion, the starting point for this paper was a recognition that although a systems approach is recognised as being important in both understanding research use and implementing change to support research use, there have previously been no systematic attempts to fully articulate what such an approach might involve, nor to apply it. Table 2's systems dimensions represent the first articulation of such an approach; and the application to existing models and the TA deployment projects provide a new analysis using this approach. In particular, the TA deployment case develops the conceptualisation of system brokerage in research-use systems. Dealing with complex, emergent, unpredictable, adaptive systems requires research and evaluation methodologies to account for this, which to date have not been available; this article's aim is to provide the basis for the development of such methodological work, and provide a framework for developers of research-use initiatives that is grounded in systems approaches.
FUNDING INFORMATION
The evaluations were funded by the Education Endowment Foundation.
CONFLICT OF INTEREST
There are no conflicts of interest to declare.
ETHICS STATEMENT
The evaluations of the pilot projects to scale-up the use of research evidence on the effective deployment of TAs were given ethical approval by Sheffield Hallam University prior to commencement of the study.
Endnote
Open Research
DATA AVAILABILITY STATEMENT
Data is not publicly available as this would enable the identification of research participants.
