FINDINGS FROM SNAPSHOT SURVEY 4:

OUTCOMES, IMPACT AND LEARNING

August 2017

Introduction
This bulletin reports on the fourth and final snapshot survey of the evaluation of the Local
Sustainability Fund (LSF). It asked all grant holders about the emergent impact and
outcomes of their projects, as well as some of the factors they thought could help to explain
these changes. It was sent to all 270 grant holders in July 2017 and we received 92
responses of which five were excluded due to being incomplete, giving 87 valid responses
(representing a response rate of approximately 33%). This is slightly lower than the previous
three surveys1. We would expect a decline in response rates throughout the evaluation
period and this response rate is nonetheless sufficient to draw conclusions, although our
ability to undertake analysis within sub-categories is limited due to the smaller number of
responses.

The survey contained nine questions, which were a mix of closed and open questions.
Because the number of respondents is less than 100, both the actual number of
respondents and the percentage figures will be quoted in this bulletin. All findings are self-
reported by grant holders and do not represent external verification of impact, outcomes
and changes.

We are very grateful to all of the grant holders who responded to this survey, and the
previous three surveys for providing invaluable data on their experiences throughout the
evaluation.

Changes in sustainability
Respondents were asked whether they thought they had seen any change in their
organisations strength, sustainability and resilience during the 12-month funded period.
Respondents were overwhelmingly positive, with 82 respondents stating that it had
improved (94%), of which 53 (61%) said it had improved greatly.

1
Snapshot survey 1 had a response rate of 67%, survey 2 of 49% and survey 3 of 41%.
LSF EVALUATION BULLETIN: SNAPSHOT SURVEY 4

Change in strength, sustainability and resilience Percentage Respondents

(n = 87)

Improved greatly 61% 53

Improved a bit 33% 29
Stayed the same 2% 2
Worsened a bit 2% 2

Worsened a lot 1% 1

Respondents were also asked the extent to which they thought these changes could be
attributed to LSF. Again, respondents reported positively, with the vast majority (83
respondents, 96%) feeling that it had been partially or predominantly due to LSF.

Extent to which changes are the result of LSF Percentage Respondents

(n = 87)

Predominantly due to LSF 45% 39

Partially due to LSF 51% 44
Nothing to do with LSF 3% 3
Dont know 1% 1

What helped or hindered this change

Respondents were asked a series of questions that sought to explore what grant holders felt
could help to explain these changes, examining factors within their organisation, LSF, and
the wider environment.

Factors that help to explain increases in sustainability

Respondents were asked to choose from a list of six factors that could help to explain any
increase in their organisations sustainability they had seen; which were a mixture of factors
associated with how LSF functioned and how the grant holders organisation worked, as can
be seen in figure 1.

The most popular factors were the staff of the organisation (81 respondents felt that this
was important or very important; 93%), the advisor (69 respondents; 81%) and the board
of trustees (67 respondents; 78%). Only 32 respondents (37%) considered the business
advisor to have been important.

2
LSF EVALUATION BULLETIN: SNAPSHOT SURVEY 4

Figure 1: If you have seen an improvement in your organisation's sustainability, how important do
you think the following factors were in helping to explain this? (base 87)

Not important
Your business partner
Moderately/slightly important
Very important/important
Your volunteers (not including the
business partner or your Board)

Your service users

Your Board of Trustees

Your advisor

Your staff

0 20 40 60 80 100

Elements of the Local Sustainability Fund programme

Respondents were also asked about the different elements of the LSF programme that had
helped their organisation improve its sustainability, choosing from five pre-determined
factors. In line with findings reported in the recent bulletin on outcome and learning, based
on the end-of-project reports, the most popular were freeing up staff time (76 respondents
or 88%) and bringing in external people to help and advise (67 respondents or 79%).
Figure 2: Which aspects of the way LSF worked have helped your organisation improve its
sustainability? (base 87)

Not at all 21
Sharing learning and insight with other grant holders 67
A little 13
A lot
9
Needing to report on progress and changes 65
26

7
Completing the ODT 48
45

1
Bringing in external people to help and advise 20
79

Freeing up staff time from delivery to enable focus on 4

8
organisational development 88

0 10 20 30 40 50 60 70 80 90 100

3
LSF EVALUATION BULLETIN: SNAPSHOT SURVEY 4

The high level of agreement with bringing in external people both backs up the earlier
finding of the perceived importance of having an advisor (see figure 1) but at the same time
contradicts the finding that the business partner was not valued so highly. This illustrates
that the advisor and business partner, while both a form of external input and support, have
often been seen and valued in quite different ways, something that will be examined in
more detail in the final evaluation report.

The part played by wider factors

Any changes in an organisations sustainability journey will also be affected by wider

developments in the environment within which the organisations exists. Respondents were
therefore asked if a range of wider factors could have acted to limit the impact that LSF had
on their organisations sustainability; they could choose from four factors as well as specify
an other category although no respondents chose to do this.

With the exception of a decline in volunteer numbers (72 respondents, or 84%, felt it had
not impacted at all)2, the majority of respondents tended to feel that these factors had
limited the impact of LSF to some degree (see graph 3), emphasising the importance of the
wider environment, something that will be examined in more detail in the final evaluation
report. The most notable was a rise in demand for your services, with 58 organisations
saying that it had impacted a lot or a little (67%).
Figure 3: Have any of the following wider factors limited the impact that LSF was able to have on
your organisations sustainability? (base 87)

Not at all 84
A little Decline in volunteer numbers 13
3
A lot

33
Rise in demand for your services 47
20

49
Decrease in income to your organisation 30
21

43
Staff changes within your organisation 36
21

0 20 40 60 80 100

2
This could also be because the respondents had not observed a decline in volunteer numbers during the
funded period. We will be examining the effect of changes in volunteer numbers in our forthcoming analysis of
the ODT scores for grant holders.

4
LSF EVALUATION BULLETIN: SNAPSHOT SURVEY 4

Outcomes
Respondents were asked about the extent which they agreed or disagreed with a series of
statements that described different changes and outcomes. These outcomes were based on
those identified within the Theory of Change which has been developed by NCVO and
Resources for Change as part of the evaluation of LSF, and the programmes original
objectives. For a more detailed exploration of outcomes and learning from projects, see the
evaluation bulletin examining the information provided in the grant holders end-of-project
forms.

Respondents reported very positively on all of the outcomes listed, with every one seeing
more than half of respondents saying they had observed these changes over the course of
the 12-month funded period. The most commonly cited, however, were thinking
differently (78 respondents; 91%), working more effectively (76; 89%), gathering better
evidence about our impact (70; 81%) and being better able to adapt to change (70; 80%).

Figure 4: Thinking about your position as an organisation now compared to before you started
your LSF project, to what extent do you agree or disagree with the following statements? (base 87)

We are thinking differently

We are working more effectively

We are gathering better evidence about our impact

We are better able to adapt to change

We have stronger financial and internal processes

We are collaborating more

We are more outward-looking

We have unlocked new resources

We have stronger leadership

We have better governance

We have better internal communication

We are better able to solve problems

We are involving our service users more

We are taking more risks

0 10 20 30 40 50 60 70 80 90 100

Too early to say Disagree Neither agree nor disagree Agree

In an open question, respondents were also asked whether they had experienced any
unexpected or unintended outcomes as a result of their LSF project, which they were able to

5
LSF EVALUATION BULLETIN: SNAPSHOT SURVEY 4

describe in their own words. Fifty-seven organisations provided a response to this question,
generally in limited detail, which can be summarised in three main areas:

(a) The planned impact was more pronounced than expected.

Most commonly this included bigger changes to the governance of the organisation than the
respondent had originally thought, but also included deeper impacts around developing a
better understanding of services users, developing new partners and networks, or the staff
team becoming closer than expected:

We were able to develop a new strategic partnership with a new partner which was
also unplanned before the project started and this has helped us secure our future.

Didn't expect to lever in such a tranche of new funding, which has secured our
position for the next 3 years.

I didn't expect to gain a network of colleagues in a similar position.

Unexpected increased beneficiary involvement and community spirit.

(b) Changes outside of the project (both internal and external to the organisation) had
unexpected impacts.

This included the unplanned and unexpected impact of staff restructures, staffing changes,
IT problems, events external to the organisation, and in one instance, the impact of an
unsuccessful merger, all of which created new challenges for the work of the organisation
and the project. In some instances, however, respondents noted that the receipt of LSF
funding had made them more able to cope with these challenges:

There were some things out of our control such as some key staff leaving, and a change
in staff with our key partner, both of which slowed the progress of our work, but the
impact would have been far worse had we not had the LSF funding.

(c) The involvement of the business partner had unexpected results and elements.

This included the business partner making less of an impact than the grant holder had
hoped for, cultural differences between the business partner and grant holder (including
differing timescales and working practices) which creating negative impacts, and the
business partner organisation collapsing in one instance:

We didn't gain from our Business Adviser's input in the way we had hoped. We
thought we would have a critical friend who could input from a business perspective
and support us with more efficient and effective ways of working.

What the Big Lottery Fund could have done differently

Respondents were asked what, if anything, The Fund could have done differently to allow
the project to have had a greater impact. Sixty-three organisations provided open responses
in their own words, 15 of which specifically said that The Fund could not have done anything

6
LSF EVALUATION BULLETIN: SNAPSHOT SURVEY 4

differently, suggesting a high degree of satisfaction with the programme. Of those who did
make suggestions, they can be grouped in to three main areas:

(a) Changes to the timing of the programme and its funding (30 respondents).

Respondents most frequently stated that a longer timer period, typically between 18 and 36
months, would have allowed them to have a greater impact (23 organisations mentioned
this):

'Getting this project completed in a year was a real struggle. It takes much more time
for change to happen and for the organisation to understand the impact of the
changes we have made as a result of the funding.'

'A longer-term period would have been more sustainable and ensured a longer-term
impact.'

Timing issues were also mentioned around having had better and more timely delivery of
funding at the start of the project, as well as having had more time to find an advisor:

'There was a problem releasing the money at the beginning which created a lot of
uncertainty and delayed things at a crucial time for us.'

(b) Changes to the funding (6 respondents).

Connected at some level to timing, some respondents described how having options for
follow-on funding would have allowed them to do more. Funding for core costs, for capital
investment, or simply more money was also suggested:

'We would have been better able to focus on our internal systems and income
generation if we had had some core costs covered; which would have made for an
even more impactful project.

'We would really have benefited from being able to seek a significant proportion of
capital money to support some of the changes suggested by our advisors and advice
received prior to the start of the programme alongside the development money.'

(c) Changes to how the advisor and business partner worked (5 respondents).

Some respondents felt that having had less emphasis and focus on business partners and
consultants would have allowed them to have had a greater impact, while in one instance,
greater clarity about how much the business partner should have been paid was
recommended.

Recommendations for future programmes

In the final open question, respondents were asked what single recommendation they
would have for the development of a similar programme if it was to be developed in the
future. Seventy-four organisations replied to this question, giving their ideas in their own
words, and their responses can be grouped in the following categories, in terms of how

7
LSF EVALUATION BULLETIN: SNAPSHOT SURVEY 4

commonly they were mentioned. The responses reflect both perceived weaknesses of LSF
(the things that grant holders wanted to change) as well as strengths (the elements they
wished to retain and felt worked well). As will be seen there is some overlap with previous
comments on what The Fund could have done differently, although in this question
respondents were encouraged to think more broadly than LSF.

(a) Timing

This was the most common recommendation, with 30 respondents making some form of
recommendation around programme timing, most commonly in terms of having a longer
funding period than 12 months:

'Change projects take time to initiate and to filter through an organisation and day to
day constraints and commitments make it hard to focus on strategic tasks quickly.'

Similarly, some recommendations were made around allowing for project extensions, or
more specific elements such as ensuring that any advisors are involved from the start.

(b) Partners and advisors

Fifteen organisations made recommendations around the role of external partners in similar
funding programmes. As has been seen in this bulletin and others, the role of business
partners and advisors in the LSF programme has been highly popular as well as more
controversial, with some respondents recommending that similar programmes should not
contain business advisors (as they had found them to be a more challenging and less
beneficial element of LSF) whilst others strongly recommended that it should be a part of
future programmes of this nature (seven respondents):

'We did get a business partner, but they really didn't add very much to the whole
process. So I would not make this mandatory if designing something in future.'

Other recommendations included making the process simpler or providing more support to
identify partners or throughout the relationship, as well as tailoring it to organisations:

'To think more carefully about the advisor and voluntary business partner roles and to
adapt these to different circumstances for each individual project.'

(c) Dialogue and communication

Seven organisations made some form of communication-related recommendation for the

development of future programmes. This included having more dialogue between the
funder and recipient, visiting grant holders more, and having a named link between the
funder and the recipient. Recommendations were also focused on having more peer-to-peer
learning opportunities, including building this in earlier in the programme, and more face-
to-face events, perhaps reflecting the fact that the peer visits came towards the end of the
funded period

(d) Flexibility

Respondents recommended having greater flexibility, to allow for wider, external changes
that grant recipients may experience:

8
LSF EVALUATION BULLETIN: SNAPSHOT SURVEY 4

'Ensure there is enough flexibility built into the grant to allow for unexpected changes to
the organisation.'

(e) Clarity

These recommendations were focused on having greater clarity around timescales and
decision-making processes and milestones.

(f) Funding

Some recommendations were made around providing core costs or funding to cover the
salaries of staff (reflecting similar comments made in the previous section), as well as
retaining the ability to backfill staff roles, which was a popular element of LSF:

'Allow for funding directly for salaries so that senior staff can dedicate time to the
strategic development.'

(g) Functioning of the funding programme

Respondents also made recommendations around how a similar programme could function,
with comments including having less monitoring, using an improved internal assessment
tool, and encouraging applicants to be more realistic in their planning, in terms of the
ambitions of what their projects could achieve (the latter point is also reflected in the
bulletin examining end-of-project monitoring forms in which grant holders frequently felt
they had been overly ambitious in their applications).

Next steps in the evaluation

This survey was the last of four that has been produced as part of the evaluation of LSF. The
information in the responses has provided valuable information which we will use to inform
the final evaluation report, which will be published before Christmas. In the meantime, if
you have any reflections on the findings presented in this bulleting please comment on the
knowledge hub where this bulletin has been uploaded, or email
nick.ockenden@ncvo.org.uk.