LEAP framework

 

Video Transcripts

picture of Alasdair McKinlay

Alasdair McKinlay, Communities Scotland - on community engagement


Community engagement is also very important in terms of the policy context in Scotland just now.

The Scottish Executive Ministers are committed to local people being involved in the decisions that affect their lives. This is particularly important in regeneration. There’s a lot of evidence that shows that if you do involve people in those decisions you end up with more sustainable solutions. That can mean better public services but it can also mean improvements in health or employment, and in self esteem and self confidence. And that is reflected now in a lot of legislation. There are duties placed on people to involve people in decision making processes: in housing, in health, and perhaps most importantly just now in community planning, in the Local Government (Scotland) Act 2003

picture of Alan Barr

Alan Barr, Scottish Community Development Centre - on need-led planning


“I suppose when you start to think about LEAP, you think about it as something which is focused on, planned change. And if you’re planning to change something, you have to ask yourself, “What is it we are trying to change?” So you need to start with some understanding of what the issues are - what the needs are - that you're trying to address.

Now most people would say, “That’s perfectly obvious, where else would you start?” But, actually, many people - many organisations particularly - tend to start somewhere else. They start with the question, “What have we got available, and how are we going to use it?” Then they think how they are going to use it, instead of thinking about what they actually need to do…what they wanted to do in the first place. Does that make sense?

So, one of the things you’ve therefore got to get people thinking about is how they understand what need actually is. So, you’ve got to first of all ask, “Well, whose needs are we talking about here?” LEAP’s about community development. Community development is about working with people to achieve the things that they want to achieve. So, you’re talking about how the community perceives need.

And that’s fine as a starting point, we want to know what people feel matters to them - whether it’s about health or housing or whatever it is - and we want to make sure they can explain that, and then operate on the basis of those needs.

But the difficulty is that community development is also about fairness and social justice and so on. So, you don’t just respond to whatever it is people want to do. You have to place what they want to do in the context of a more objective view of needs. So, you’ve got to think about how do needs here compare with needs there. How do the needs in this context relate to what we would expect to be a reasonable standard of provision…or whatever. So, we get into this language of felt need, expressed need, comparative need, normative need - I don’t want to go into all of that - but we have different ways of looking at it.

So, we need to think then, what sort of need is it, whose needs are we starting with. What we’ve then got to do - I think - is to consider the issue of what starting with needs might do to people’s perception of the problems. Some people say, if you start with needs, you start with something that is very negative. You start by saying, what’s wrong, instead of what is right and what we can do about it. And that is a contrast between what is called a need-led approach and asset-based approach. I think we’ve made a false distinction between those two things. When we talk about being need-led, we say, “Yes we know what it is we want to change; but in order to change that, we need at the same time to look at the abilities and skills that people have to bring to that change”. So, need-led, is not sinking people into all the negatives, it’s actually finding a starting point to use the positives to build for purposeful change.”

picture of Jane Dailly

Jane Dailly, Scottish Community Development Centre - on planning in partnership


“Partnership working is increasingly a central feature of the way we are required - or should be working - to address quality of life issues in communities. It’s intended to ensure that agencies can work together in a more coherent or joined-up way in order to provide more effective services, but also so that communities are included as equal partners in that decision making process. In order, again, that services with that community influence can be more responsive, more effective, and more sustainable.

I think the important thing about LEAP - or the LEAP framework - in relation to partnership working is that it takes as its starting point an understanding that yes, we should be working in partnership because the issues that we’re trying to deal with for communities are quite complex or multi dimensional. So, they are not necessarily the domain of any one sector or one agency. So, we need to be working in that joined-up way. But it also takes as its starting point the idea that there’s a really important process that needs to underpin that way of working. And if it’s not there, then the impact of partnership working can be really limited. I think that the LEAP framework supports us to engage with that process in a couple of ways.

Firstly, the approach that the LEAP framework describes, asks us to consider planning as a deliberative process that involves all stakeholders in change - or whatever we are trying to do - in that debate and dialogue. I think really that that’s the essence of partnership working is that willingness to approach planning as a really important process in itself. It’s a process that we go through that cements partnership working, that tells us why we are working in partnership, and that builds the relationships that need to be there for partnership working to be effective.

Secondly, what LEAP does is that it takes us through - in a step by step way – it takes us through the process that that shared dialogue needs to go through in order to build effective and sustainable partnerships. It takes us through the process of identifying and agreeing the need that we are trying to address, it takes us through the process of visioning the change - the shared outcomes that partners want to see. And it takes us through the process of building accountability to each other and to communities. I think that is an incredibly important part of the planning process and of working in partnership. We need - if we want to take forward a shared agenda - we need to do that on the basis of a shared understanding, a shared vision, and a shared idea of our responsibilities.”

picture of Karl Monsen-Elvik

Karl Monsen-Elvik, North Lanarkshire Volunteer Centre and Development - on using LEAP framework at a national level

The Volunteer Centres across Scotland, they work in such diverse communities. The ranges of needs are so different depending upon the circumstances they face. And yet we try as a network - although we are 32 organisations separate and independent - to work in concerted way to deliver on the Scottish Executive’s Volunteering Strategy.

I think what LEAP did for us… it allowed us to be able to deliver on the strategy but to design the particular work in each area according to the local need. So we spent a bit of time investigating in each area what we felt volunteering could do to improve the lives of people who live there. How it would affect, for example, young people who were not involved in other activities, or people who were unemployed or looking to develop their skills in some kind of way. As well as just straight-forward people who want to do something to help their communities.

Therefore, we were able to design outcomes particular to the area where that volunteer centre operated from. And then - in line with the Volunteering Strategy - target the resources we got from the Scottish Executive towards meeting those needs that we’d identified. So, the Executive were happy because they got the opportunity to get a reporting framework which is consistent throughout Scotland. But the individual volunteer centres didn’t have to fit into a one-size-fits-all reporting framework. They had the chance to set the agenda according to the local need that they had identified by their engagement with the different parties in the area.

The way we used to report our work was: we would list what we would do during the year - and I suppose it would be things like - number of visits to schools to talk about volunteering; number of training session to organisations on good practice and volunteering; involving volunteers, recruiting volunteers…and it was all about counting beans and things like that. I suppose at the end of the year - if you had done those things - yes sure, you would have done something to develop volunteering in the area. But using the LEAP approach - where we focus on what really needs to happen - the bit about doing things, that came later on as part of the plans and the outputs that we would do.

We discovered that during the year we might decide to change the way we were doing things because it wasn’t taking us to what we eventually wanted to go to. And whereas in the past that would look like you weren’t meeting your targets. What we got was an opportunity to say, “Well no, let’s just choose some new targets that would take us to the outcome that we are trying to achieve”. The good thing about that was, of course, that those who fund us also recognise the value…that it was the outcomes that we are trying to achieve, the real differences in the communities that we are trying to achieve…and not just - you know - tick boxes as to the number of times we produced a leaflet, or the number of times we gave a talk.

The LEAP framework has allowed us a chance to sometimes stand back a bit, and look at what we think we should be doing. Rather than just thinking we know what should be done, what the needs are, what the priority should be. We took the time at the beginning of the process to begin to ask our staff teams, our volunteers, partners we work with in a variety of settings, what they felt volunteering could do to contribute to towards achieving activities they were trying to achieve success on.

So, we felt that when we came up with our objectives, with our outcomes, they were much more geared and relevant to the areas we were working in. And then when we came to write the plans actually, it wasn’t just - as had been the case in the past - that someone got given the job to write this work plan, and then that’s it done and we don’t have to worry about it anymore. It was a piece of work that everybody got involved in. We got reports back not only from the training we delivered on, and how it could roll out, but even unsolicited support and feedback. How invigorated staff teams felt when they felt they were having the opportunity to say, well actually, from my perspective, as someone who works for the groups - not necessarily as a manager working at a strategic level - this actually works better for us, and it would be good if we could do it this way instead of the old way.

I think that feedback gave us the confidence that because various people were involved in deciding what actually needed to happen - and not just one or two people who thought they knew what had to happen - it was much more relevant to the community. And also people bought into it more, and were looking out for the outcomes, and were happy to report on it throughout the year rather than at the end of the year think…”Oh no, we have now got to put our report in.”

picture of Stuart Hashagan

Stuart Hashagan, Scottish Community Development Centre - on learning-based planning and evaluation


Within the LEAP framework you'll realise that the first letter in LEAP stands for L for learning, and in a sense I think that's probably the most important part of LEAP. It's the main reason for doing planning and evaluation, in our view, is actually to learn about what you're doing, how you're doing it, how you might be able to do it better, and whether you're actually having an impact.

So, learning is really the first reason. I think one particular reason for doing it in the community development field is because we're working with very complex issues, complex relationships. It's quite hard to track down the relationship between cause and effect. So that complexity, the range of people who are involved, makes it more important that we actually try to work out what the impact that each partner has on the change that may take place. So it's really, as I say, about learning and it takes us beyond what evaluation traditionally was about which is usually about accountability and sometimes gets translated into targets, outcomes that people need to hit, and I think we need to go a bit further than that and actually understand how it is that we're achieving change, whether we can do it better. So, that's really the first thing to say about LEAP as a learning programme.

So, why is that learning important? I think first of all, we do need, if we've set outcomes - as we do - and if we've agreed how we're going to try to work towards achieving those outcomes, we actually need to know whether we are achieving those outcomes or getting towards those outcomes in they way that we want it to. So we need to understand whether things are changing. We need to be able to look at how that change is taking place. That's partly because we want to know whether we are achieving change. But it's also so that we can be accountable to each other, not accountable to a target. So that communities can hold public bodies to account, and indeed the other way around. Public bodies can work with communities and say actually say, "You said you would support networking or whatever and it doesn't seem to be taking place". So, because we're working in a partnership environment, it's very important that all the partners are aware of what everybody's contribution is supposed to be and whether they're sticking to that.

Finally, it is important that we give some value to the participative way of working and I think that's in contrast to some of the traditional ways that evaluation has taken place. So, really if we are to say "Working with communities is important", "Working in partnership is important", we need to be able to provide some value, some information, and evidence that does actually show that that is a valuable approach. And that's in contrast to other approaches which are simply looking for outputs in the field of health: whether heart disease, for example, has changed. So, we really want to know about the processes and the impact on peoples lives.

So, that's why we think it's important. The learning, the learning approach, what do we actually need to learn about? First of all I think we need to learn about our theories. We have theories that if we do something it will lead to such and such a consequence, and we need to actually test that out, check it out to make sure that that theory is actually working. And that's why it's important to be clear about the links between outputs and outcomes within the LEAP model. The second thing we need to learn about and understand is our methods. Whether the way we actually go about supporting people, training people, building networks or whatever it is. Whether that actually is effective, whether it leads to the change that we want. And indeed whether it's efficient if you like, so that it does it at a reasonable cost. Not just of money, but also resources, of time and energy and so on. So, we need to know about efficiency.

Finally and importantly, we need to learn about whether we're actually being inclusive. Of course, the whole basis of working in communities, is about engaging with people who may not otherwise have the opportunity to be involved so we need to make sure that we're being equitable in the way we operate.

So, the final things I wanted to say was really about who should be involved in this learning from experience, and really the short answer is everybody. Communities do need to learn about how change takes place; what their contribution is; how they can do more using the resources and skills they've got in order to achieve change. People who are called practitioners - whether it's in community work, or health, or social work, or regeneration - they similarly need to know whether the things they're doing – bringing from their agencies - are contributing to change. And, very importantly, I think policy makers, and managers - particularly at partnership level - also need to know that they have a role, and a clear role, within that. So, those three things are all important, and I think there is a tendency for the community level to be the one that's the focus of the greatest effort in evaluation. And I think it's important that we don't let policy makers and managers away with not being evaluated. They've got a clear role in trying to create the conditions, and the environment, and the capacity within which change can take place. And that needs to be evaluated just as much as the change taking place on the ground. So, it's really important to I think say that the LEAP model isn't just about what happens locally in communities; it's also about what happens at, what you might call, programme level and indeed at policy level. All these things are important.

picture of Deryck Irving

Deryck Irving, Greenspace Scotland - on outcome-focused planning and evaluation


“Greenspace Scotland, our partners and ourselves, have been trying very hard recently to point out that green space in urban areas isn’t just about environment: it’s about health, it’s about economy, it’s about quality of life for people who live in towns and cities. But when we come to look for an evidence base looking at projects, it’s sometimes very difficult to show that detail. The reason for that, we’ve decided, is because most projects - because of the way they are funded and because of the way they are planned - are focused on outputs or physical actions and missed out on those wider, longer term impacts. So we were very keen to find a way to get around that, because we found that people were actually missing out the good quality work that’s being done and actually losing out on some of the stuff we should be showing to people: which is where outcome- focused planning and evaluation comes in.

We got very interested in the approach because it’s based on partnership and participation which is very much how we see Greenspace. But it also allows you, at the beginning, because you are starting with a vision for change, to get full agreement about not only what you are trying to do but how you will be able to find out whether you did it or not. And this really does move things forward. It’s a really exciting tool for planning in a way that is more effective. It’s about evaluation in a way that proves what you have done and helps you to learn what you have done and how to do better in the future. A partnership tool, and ultimately it’s a really important tool for us ensuring how we actually impact on other people’s agendas and therefore why they should be playing the game with us.”

picture of Seamus Connolly

Seamus Connolly, Kelvin Clyde Greenspace - on community involvement


“I’m employed by Kelvin Clyde Greenspace which is a small project contained within Glasgow City Council. We work on environmental improvement projects in Glasgow. The whole ethos is involving communities in these projects from…we either facilitate the projects or we work with the communities to develop their, either, community green space or land within their areas.

We view the fact that the communities have been involved - from the early stages of the project right through to development and sustainability of any green space environmental projects - as really, really important to ensuring that it is what the community want, and it is deliverable, and that they’ve got a real feel, and a real ownership of the actual projects themselves.

One of the projects that we have just recently completed is up in the East End of Glasgow. It was a new community park, the first new urban park in Glasgow for 20 years. It was an area of derelict land and we never really came up with an idea what we wanted to do with the land, we just took it to the community for them to explore possibilities that this land could be used for. What we then discovered by working with the communities is that there was a lack of quality green space in that area. So we developed a brief in partnership with the community for it to be turned into a quality green space. The community wanted play areas for all age groups, plus an area of garden space that they could actually use to relax in, plus an area where the kids could maybe just go and play football.

So, from the early inception of starting that from the development stage we took the community all the way right through the development stage and implementation of the project. And now that the project has now been completed, they are on board, they have formed their own 'Friends of Beardmore Park' group and they are looking at sustainability of the project. Making sure that it is used positively and there is events on for all age groups and sections of the local community.

The 'Friends of Beardmore Park' - there are five local authority services that sit round the table with the 'Friends of Beardmore'. And they are there (as opposed to delivering council services) they are there to offer advice to the 'Friends of...' group for various events that they want to put on. They are not there to say "We are here to deliver". They are there to assist the 'Friends of...' deliver events for the local community. It could be a teddy bears picnic, if the local community want to put on a teddy bears picnic, who do they approach for doing risk assessments? Events? Manuals? all manner of things. They are there to offer advice, as opposed to delivering a service.

So, what ‘s really happened is the council officers from various departments have been quite enthused by the reaction that they get within local communities; to see that the local communities do want to have voice, and they do want to have ownership of areas within their local community.”

picture of Margaret Layden

Margaret Layden, Project Co-ordinator, Playbusters - on planning in partnership


“I think from our own point of view, we have worked very well with agencies. We have got a good reputation of working with all agencies and everyone having their part to play. We have got to realise, that people have maybe different targets they have to reach, but it’s how your targets also fit in with them and everything is in relation to each other. We have got to be clear on what the goal is there, and the goal is to have a stronger community. So if you keep that in focus then you are going to benefit from that. That is really where we need support, in everything, to members of the community because first of all they have be aware their views are being listened to, they are being acted on. But they also are a part of that bigger picture. Everyone benefits from that, they appreciate then that it’s just not their local area but there is a strategic part they have to look at as well.”

picture of Celia O Lone

Celia O Lone, Co-ordinator PEEPS, Parents of East End Primary Schools - on community involvement


“I think it’s important that the people are involved at the planning stage of a project because it certainly moves things along a lot quicker, and you get much more community buy-in on it. People believe that this is going to happen. Back in the beginning we had a meeting with some potential funders and got told that it was a great idea…ten years it would take us to get it. The centre opened this week, three years later. And it was really because the community were driving it forward.

I think we addressed the issue of people’s needs quite successfully because they were involved from the beginning. Every user group that came through the door was asked quite clearly to sit down and talk about what they needed in a centre, and what would be their requirements. We also consulted from three year olds upwards. We had days when they came in and they got their chance to say what they would like to see….we used pictures. Right through to the pensioners in the community. And they were aware of it at every stage. We haven’t had an official opening yet but everybody has been through and seen it and they have been standing watching it getting built and waiting on everything happening.”

picture of Celia O Lone

Celia O Lone, Co-ordinator PEEPS, Parents of East End Primary Schools - on understanding local need


“I became involved with a project that was called the Bridgeton Community Learning Campus. Basically it was a group of local people and we were on a voluntary management committee. The community centre we were involved with was really run down, hadn’t had a lot of investment into it. It really didn’t suit the needs of the community anymore. We got together and we done some consultation with the community looking to see what different groups were looking for and came up with a plan that we thought that the community would actually buy into with us. From there we contacted the Social Inclusion Partnership Board to see if they could give us any help and basically we were looking for funding to see how we could progress.”

picture of Carole McLuckie

Carole McLuckie, Chairperson, Whiterose Management Conmittee - on the experience of being involved in community-based planning

“About a year and a half ago we started planning a park. It’s called the Beardmore park. It’s finished but it’s not finished, we are having an open day and things like that. The reason I went on the committee to plan the park was so that - I have got two young kids - so that they have got something that they would be interested in in this park where it’s easy access for them. The park that we’ve got just now is too far for my kids to go to. So this one is just round the corner, so that my kids would have something that I knew they would enjoy.”

picture of Jackie Ryan

Jackie Ryan, Co-ordinator, East End Child Safety Project - on the experience of being involved in community-based planning END AUDIO CLIPPED

“My experience of being in the planning stages of a project was the East End Child Safety Project. We were delivering child safety equipment to about 30 to 40 families a year and that was fine. But we decided - the community, the volunteers that were involved with it - we decided that we could go further out if we received more funding and we could reach further into the East End because the whole of the East End did have a need for child safety equipment due to the high level of low income families.

So, we went into five different areas in the East End. We were in Dalmarnock Community Centre, Bridgeton Community Centre, we were in Tollcross Community Flat, we were in East End Health Action and we were in Calton Child Care Project. Which meant we were then able to deliver a service to 300 to 400 families a year. So we went tenfold.”

picture of Yvonne Kucuk

Yvonne Kucuk, Chairperson, Cultural Connexions - on the experience of being involved in community-based planning

“I would have to go back to when the BCLC originated, that’s our group I’m involved with in Bridgeton. What we started was…it was like a council flat years ago we got for the community to meet up in. Everyone got together and we decided about the facilities in the area for the kids and the centre we had at that time just wasn’t sufficient, it didn’t meet our needs. So we got together. We got on the committee and then we planned to apply for money to get a brand new centre. Just two weeks ago we were given keys to it. So we were involved right at each stage. We were involved in the planning of it and what we wanted in it, where it was going to be, what sort of services were provided in it as well.”

picture of Margaret Layden

Margaret Layden, Project Co-ordinator, Playbusters - on why community involvement in planning is important

“I think really the community know better than anyone else, the issues within their own areas. And also - in a lot of ways - the solutions to the issues. And I think if the community aren’t involved at the initial stage, then nothing is going to be achieved there. So, I think it’s really important that we listen to their views and they can actually feel as if they are influencing what is happening. And I think that is what has been important about this project. It was originally set up so that parents and local people would influence and shape - and also looking at the local plans and strategic plans as well - and have their own stamp on what was happening.”

picture ofYvonne Kucuk

Yvonne Kucuk, Chairperson, Cultural Connexions - on why community involvement in planning is important.

“I would say that it’s essential for the local people and the community to be involved right from the start because it’s their ideas, it’s what they want. It’s what these people need. So, we take on board everyone’s ideas, we sit down and see if it’s feasible and we work towards it. There is no point in getting people in to build centres and say “There you go, that’s yours”, and it doesn’t suit us. So, it’s their wants, it’s their wishes. These are the people that are going to be using it so we have to take into consideration what they want.”

picture of Jackie Ryan

Jackie Ryan, Co-ordinator, East End Child Safety Project - on why community involvement in planning is important

“Well we thought it was important because it was volunteers that had taken this project through for quite a while. And it was volunteers and people that live in the community that knows what is better. The process of involving people was to have open days, to have information days, involving the usual suspects like your health visitors, your social work, your health board and different things. But it was making sure that the community knew about it and inviting local people along to say, “Well, this is what we think as a community, this is us. Do you want to get involved? Do you want to come on board?”.

We had a big open day, we had several open days, we had about three or four. And we thought it was important to get them on board and any sort of adjustments, fine tuning happened from there. And there was some. And it was things that even me as a local community member, I never thought of, other volunteers in the project never thought of but many hands make light work, and obviously different people with different views. And that is why we cover so many different aspects of child safety.”

picture of Celia O Lone

Celia O Lone, Co-ordinator PEEPS, Parents of East End Primary Schools - on participatory evaluation


“I have been involved in the evaluation of the project. I am now the Co-ordinator of the project. And we use the LEAP system and it’s been used to very much get the targets you can’t get on paper. We talk to the people, we sit and we discuss things and it’s very much about setting their own targets. We do the short-term, medium-term and long-term. And really, we meet everybody every six weeks. Everybody should have a small interview every six weeks. And it’s seeing if you are getting there on your targets but it’s also seeing if you have reached your target - that’s fantastic - and what’s your new target? It’s also seeing - if you haven’t reached your target - what’s been the barriers? What stopped you getting there? And how can we help you overcome those? And it can be very simple things that’s actually just held people back at that stage. And maybe just re-looking at their target and resetting it for a later date.

I think the project is very successful because we have been working that way. We are listening to the needs of the actual learners as opposed to being told, “This is how it should be done.” All our courses come from the needs of the community and we do have a 98% retention rate which is fantastic. I think it’s mainly because it’s came from their ideas and it’s what they have asked for.”

picture of Margaret Layden

Margaret Layden, Project Co-ordinator, Playbusters - on the learning that results from working in partnership

“Well, I think it’s a learning process for everyone involved, because you know there is a lot of partners round the table and that’s whether it’s communities or other agencies. People really need to feel that they are being listened to…that what they are saying, that they are part of that. I think, the learning process we’ve had is that…everything has to be transparent. There are no hidden agendas there, but that people can see the benefits of real partnership working, and that’s with the community involved throughout the whole process.

Another learning curve - I think - was that people need to be supported. We are talking here about…some of the volunteers have maybe been around for a long time, worked very hard within their communities, and have made changes there. But at the same time there are new volunteers and also we have different guidelines involved there as well. So it’s support to them as individuals, to make sure they receive that. I think that is a big part of what we learned, is that we really need to work together for that.

Also, I would say that another part of it would be the time-scales. You can’t hurry people along. You have to be aware that people need that time and there is enough consultation for example. That people are consulted along at every edge of the way and that information is distributed to them, and then we work on that.”

picture of Celia O Lone

Celia O Lone, Co-ordinator PEEPS, Parents of East End Primary Schools - on learning and community involvement

“I think when we started the project we were always told that we wouldn’t get there…and it came. You actually take it that your community doesn’t believe that you are going to get there. And we discovered it actually doesn’t come from your community, it’s from people outside your community that are telling you that you can’t do things. I think that we learned as a community you can be really strong. And working together you actually get there. And when you actually see the physical things starting to happen, actually it spurs the community on to look and see what’s next. As soon as the framework for the building went up people were like, “Alright we’re getting that, what’s next?” And it has really moved everybody on together.”

picture of Jackie Ryan

Jackie Ryan, Co-ordinator, East End Child Safety Project - on learning and community involvement


I learned - by going through this planning process - I learned that, there are people out there who want to make a difference. There are people out there that - give them half a chance - they will be transformed from a mother or a father that’s saying they are only a mother or a father - or they are only a grandparent - to say “No, I can make a difference, I have done this.” And, “I can help you, you can help me.” And it was all about building us into something really, really special in the community, for the community, by the community.”

picture of Sanjeev Sridharan

Sanjeev Sridharan, University of Edinburgh Research Unit in Health, Behaviour and Change) on evaluation

Let’s take the first point, what is this thing we call the programme? Most evaluation tends to focus on, does the programme work? I think we often don’t stop and ask the question, “What is this thing we call the programme”? Let’s take a programme like ‘Prevention 2010’ or ‘Have a Heart Paisley’. Take 'Prevention 2010', which is being implemented in many, many sites in Scotland. If 'Prevention 2010' means something quite different in a site in Glasgow, as compared to what is being implemented in Dundee, then answering the question "What is 'Prevention 2010'?" might mean something different in Glasgow as compared to Dundee. So, our first task as evaluators is trying to define - trying to explicate as I like to say - what is the programme? That’s one, but part of one too is also trying to understand if different stakeholders - say within ‘Prevention 2010’ within Dundee - have very different views of what constitutes ‘Prevention 2010’. So, explicating the different points of views of stakeholders is also important as a part of understanding, what is the programme?

The second question is, “How does the programme work?” It ‘s important to remember most of our community programmes are very complex initiatives. It’s not the same as taking an aspirin, where you take the aspirin and you can measure if your headache has come down or not. The "how" of most community initiatives - just because it is very complex - is extremely difficult to define. The second task of evaluators is to speak to stakeholders, help explicate why do we expect the programme to work, how would the programme work? Another way of saying this is, what are the linkages between the activities and the outputs that constitute the programme, and the outcomes you would expect to see. Keep this in mind, very different stakeholders will have very different views about how the programme is supposed to work. So, as an example, somebody working within the clinical side of things, might perhaps have a much more narrower view about the how of it, as compared to somebody from the community side of things. So, defining the how is critical.

The third question is - that evaluators need to address is - when would we expect to see changes as a result of the programme? Keep in mind that most evaluations run only for a short time. Maybe a year or two, or three if you are lucky. Most complex initiatives – most complex community programmes - often take many, many years to impact outcomes. So, we have a real problem if an evaluation runs only for two years, but the programme is likely to bring about change in three, five, seven years…then how would we know if the programme is working? There are far too many evaluation reports written where, at the end of the evaluation - at the end of the three years, the end of two years - the evaluator says “Not enough time has gone by before we can tell if the programme is working”. My view is, early in the life of a programme, early in the planning stage itself, the evaluator needs to speak to the different stakeholders - the funders of the programme, the folks who implement the programme, and community - a variety of stakeholders to help understand, when would we expect to see changes as a result of the programme.

Finally, one of the key questions the evaluation needs to address is: how do we know if a programme is working? This is often the focus of a lot of evaluations and, once again, most complex initiatives - most community initiatives - are not black boxes. Different stakeholders have very different views, about what the purpose of the programme is. I think an evaluator can help address the question - how do we know if a programme is working? - by speaking early in the life of a programme to the stakeholders to help define three things. Number one, do different stakeholders have different success criteria for answering the question, how do we know the programme is working? As it turns out - especially in complex initiatives - different stakeholders have very different yardsticks to define success. Understanding that early in the life of a programme is critical…that’s one.

Number two, as you address the first question - success criteria - what kinds of data do you need to help address that question. So an evaluator - and evaluations - need to engage in dialogue with stakeholders to address the question: how do we collect this data? Who is going to collect this data? What role would the evaluator play in collecting this data? What role would the programme play in collecting the data? This is critical in addressing the overall question, how does the programme, how do we know if the programme is working?

Finally, in terms of addressing how do we know the programme is working, the evaluator needs to talk to the folks on the programme about possible evaluation designs. What do I mean by that? Let’s take Have a Heart Paisley which focuses on heart disease. The focus of Have a Heart Paisley is to help reduce the CHD rates in Paisley. As it turns out, if we find that already the trend in Scotland is that heart disease rates are already dropping, how can we be sure that the reductions in heart risk are a result of Have a Heart Paisley as compared to larger trends like broader sector trends that we are seeing within Scotland. One way to do that is to think of a comparison community. That’s only one example but more broadly evaluators need to spend time thinking about possible designs that can help address the question, how do we know if a programme is working? Most critically the design is not simply an academic problem, it requires active engagement between the evaluator and the programme to address this question, to help clarify what design is required.

So, to summarise, coming back to your original question, what role can evaluation play at the early stages of planning? My view is the answer, the evaluation needs to address four questions. What is the programme? As I have argued, it’s not an easy question to address, even though it is a fairly obvious question. Two, how do we know, or how is the programme expected to work? This, once again, as I have said, different stakeholders within the programme will have very different views on it. Three, is when do we expect changes to happen as a result of the programme? Once again the evaluator has a role to engage with the stakeholders to help define that question. Fourth is, how do we know if a programme is working? This is the critical evaluation question, once again I have argued, this again requires an active engagement of the evaluation with programme planners to help address that question.

Back to top