close

36: Creating a framework that enables deep learning

Learning can occur at many levels but creating a learning framework that allows for deeper insights to emerge is of crucial importance for navigating towards sustainability. This work card describes what to think about to create a framework that enables this type of deep learning.

How to do this workcard

Working with the coalition and additional implementation team members, in a workshop setting, develop a learning based framework to support implementation. Separate this thinking from the traditional monitoring and evaluation process which is dealt with in Work card 37. 

Suggested approach: Workshop with coalition and implementation team.

Time required: 3hrs

Facilitation skill level: High – maintain focus on learning not monitoring and evaluation

Resources: Prioritized list of key assumptions

Learning versus monitoring and evaluation

There has been a large body of knowledge developed in recent years on Monitoring, Evaluation, and Learning (MEL), especially in the international development sector. Typically, these MEL schemes have focused on accountability, which means demonstrating to donors and others that resources have been spent appropriately. While this is obviously an important activity, it does not necessarily lead to deeper learning and will not really help you to navigate towards more sustainable futures.

When working with complex systems, deep learning approaches are needed to question and challenge your underlying assumptions. Within this type of framework any specific kind of monitoring an evaluation can be developed. Photo: iStock.

Rather than starting to develop specific indicators for success, the most critical task is actually to establish a learning framework that enables deep learning. Within this framework any specific monitoring scheme can be developed later but starting with a learning framework means you are likely to ask the right questions at the right level for the right reasons, rather than just developing sets of indicators and measures that may not drive real learning and change. See the attached case from AWARD in South Africa, an organization that uses a deep learning approach in their projects to tackle issues of sustainability, inequity and poverty.

Different levels of learning

Learning can occur at many levels. Basic learning, or “single-loop learning”, can come from simply asking questions about the expected outcomes from an action. For example, in a project to restore fish populations in a river, after a series of interventions you may ask by how much the numbers of fish increased. This level of learning is useful for assessing the efficiency and/or effectiveness of an action towards achieving an outcome in a context of certainty rather than uncertainty, but it is unlikely to drive deeper change in how people think about the river system. It is, for example, very unlikely that you capture any unintended consequences of the fish restoration activities.

When working with complex systems, a “double-loop” or “triple-loop” approach to learning is more appropriate. These types of approaches push you to deeper levels of enquiry; questioning and challenging your underlying assumptions. So, in the fish example above, following on from the initial evaluation questions about the relative success of achieving the increase in fish numbers, you might also ask questions about the broader consequences of your actions and the objectives of the project, for example:

  • In addition to influencing fish stocks, what other effects did the intervention have?
  • Why do we want to increase fish numbers?
  • Who will benefit from increasing fish numbers?
  • Who may be disadvantaged?
  • What are the key assumptions we are making when we assume increasing fish numbers will improve food security for local people?
  • What are the important assumptions we are making about the data and information relating to fish populations and the ecosystem?

These types of questions can then drive deeper reflections about the system, about the process of setting the objectives, and about the assumptions that were made at the time. This may drive more fundamental system change, reorganizing structures and changing power dynamics, and in some cases, transformative change can emerge. These kinds of triple-loop questions include, for example:

  • Who set the objective of increasing fish populations?
  • Was the process fair and transparent?
  • Was it open to all stakeholders, including the less powerful or marginalized?
  • How well does the decision-making process match the scale and dynamics of the system and issues?
  • Is the long-term vision for the system a truly sustainable and fair one?

Making sure that you learn deeply

Setting up a learning framework that allows for this type of deeper enquiry, will force you to confront deeper issues about the underlying purpose, the structure and the function of your focal system. This type of framework will provide a good foundation for assessing if your action strategies continue to move you towards a sustainable, safe and just future in a highly uncertain and rapidly changing world (figure 36.1). Discuss with your Coalition and additional members of the Implementation team how you can develop such a framework to capture the outcomes of your implementation process.

Figure 36.1. When working with complex systems, a deep learning approach is best suitedas this will force you to confront deeper issues about the underlying purpose, the structure and the function of your focal system. Illustration: E.Wikander/Azote, adapted from Pahl-Wostl (2009).

Case

Reporting for social learning in the RESILIM-O project

The South African non-profit organization AWARD (Association for Water and Rural Development) works with participatory project implementation to tackle issues of sustainability, inequity and poverty. Their RESILIM-O project ( Resilience in the Limpopo Basin – Olifants River), initiated in 2012, focuses on reducing vulnerability to climate change through improved transboundary water and biodiversity governance of the Olifants River Basin, shared between South Africa and Mozambique. Through resilience-building activities, the project seeks to improve the health of key aquatic and terrestrial ecosystems as the basis for sustainable livelihoods for people in the river basin.

Riverbed in the Olifants River basin. Their RESILIM-O project focuses on reducing vulnerability to climate change through improved transboundary water and biodiversity governance of the Olifants River Basin, emphasizing the importance of a social learning approach. Photo: iStock.

The AWARD approach clearly shows the usefulness of embedding monitoring and evaluation in a deeper learning framework. The RESILIM-O project is firmly grounded in a social learning approach that views learning as an ongoing collaborative process, in which all stakeholders and the project team are engaged in a shared learning process. Recognizing that complex problems such as sustainability issues typically require collective action, AWARD seeks to engage people in collective learning and sense-making.

The three-layered reporting framework developed by AWARD, which includes monitoring, case-based evaluation and meta-evaluation. Illustration by E. Wikander/Azote, adapted from illustration on http://award.org.za/project/resilience-in-the-limpopo-basin 

This perspective permeates AWARD’s project design and ways of working, the organization actively seeks to create processes that foster a “safe” learning space for new, collective and transformative understandings from all parties. One of AWARD’s many innovations is their use of monitoring and reporting as a tool to better foster and capture learning. In many development programs, monitoring and evaluation is largely focused on quantitative indicators, and used primarily for performance management and accountability. When used for these purposes, monitoring and evaluation usually offers limited opportunities for learning. AWARD therefore decided to expand the purpose of reporting to also being a tool for reflection and learning, while still serving the purpose of accountability.

To do so, AWARD uses three level of reporting. On the basic level this includes more traditional monitoring compatible with frequently used donor indicators, but at deeper levels the reporting also includes case-based evaluation and meta-evaluation, reflecting for instance new emerging lessons, the quality of the process, and whether or not the program is meeting its objectives. 

 

Read more:

http://award.org.za/project/resilience-in-the-limpopo-basin

References

Here are some suggested references if you want to read more about the issues discussed in this work card:

Pahl-Wostl, C. 2009. A conceptual framework for analysing adaptive capacity and multi-level learning processes in resource governance regimes. Global environmental change. Volume 19, Issue 3, August 2009, Pages 354-365

Open this card Close this card
close

37: Setting up monitoring and evaluation in a deep learning context

Once you have established a framework that enables deep learning, you are ready to develop a detailed scheme for monitoring and evaluation. This work card helps you identify relevant indicators for system change at different levels, in the short term and long term.

How to do this workcard

Once a deep learning framework has been developed, the same group of people can focus on developing the Monitoring and Evaluation approach.  In a workshop setting, focus on developing M&E indicators and processes that feeds into the learning approach, as well as meets the accountability and reporting obligations to donors and stakeholders.

Suggested approach: workshop

Time required: 3hrs

Facilitation skill level: High

Resources: Outputs from Work card 32 and 36

Developing relevant indicators

While it is difficult to give very specific recommendations about which variables to monitor and evaluate, since this will of course vary a lot with different systems and contexts, there are a number of generic recommendations that still can be made. First, it is important to try to assess both how critical system dynamics change and how the option space changes following your interventions. Indicators must also cover both ecological/environmental variables as well as variables related to human wellbeing and stretch across relevant system scales. Where possible, include the people that are involved in testing the actions and are influenced by them in determining what appropriate indicators are, and in measuring the effectiveness of the actions. Asking people what the earliest indications would be that your actions are creating the desired change can reveal important information about system dynamics.

A farmer using his laptop in the field, India. While relevant variables to monitor will vary with the context, you should think about including both short-term and long-term indicators, reflecting social and ecological outcomes and processes. Where possible, involve stakeholders in the monitoring and evaluation. Photo: iStock.

Short-term indicators

In the short term, try to identify indicators that:

  • Provide a rapid feedback about how well the implemented action strategies are creating the desired change, especially considering those that are
    • focused towards key knowledge gaps and important assumptions that you have made
    • focused on changes across system domains (use your systems models to guide your thinking about key social, ecological and economic variables here)
    • easily measured or observed (i.e. low cost, easy to collect)
  • Reduce the risk of doing harm in the system, especially those that are
    • focused on detecting unintended consequences
    • focused on vulnerable groups of people
  • Are located in key learning ‘nodes’ in the system, look for changes in
    • language, metaphors, and conceptual models (these are all indicators that thinking is changing)
    • practice and routines for how people manage and relate to their landscapes
    • networks and flows of information

Long-term indicators

In the longer term, try to identify measures that identify:

  • Changes in feedbacks
    • for example, where traps start to destabilize, or where feedbacks shift from reinforcing to balancing
  • Changes in structures and contexts of the system
    • changes in key system variables, for example, the appearance of a new important factor in the system
    • changes in network structures and resource flows
    • changes in representation of people on decision making bodies, including empowerment of previously disempowered groups
  • Change in the system option space
    • Changes in important option space dimensions. Are they increasing or decreasing? Have existing trends changed pace?
  • Change in the trajectory of the system;
    • Changes in goals, purpose and vision for the system

References

Here are some suggested references if you want to read more about the issues discussed in this work card:

Greig, L. A., D. R. Marmorek, C. Murray, and D. C. E. Robinson. 2013. Insight into enabling adaptive management. Ecology and Society 18(3): 24. http://dx.doi.org/10.5751/ES-05686-180324

Roux, D.J. & Foxcro , L.C., 2011, ‘The development
and application of strategic adaptive management within South African National Parks’, Koedoe 53(2), Art. #1049, 5 pages. doi:10.4102/koedoe. v53i2.1049

Open this card Close this card