LSF EVALUATION BULLETIN

SNAPSHOT SURVEY 1
20th October 2016
This bulletin presents the findings from the first LSF snapshot survey which explored grant
holders experiences of project start up, the biggest challenges they have faced and their
reflections on the Online Diagnostic Tool (ODT). This information is just for grant holders and their
partners at the moment and shouldnt be published elsewhere.

Many thanks to all 67% of grant holders who responded to the survey your participation is
greatly appreciated and this gives us a high level of confidence that these findings are accurate.
This bulletin offers a brief summary of what we think are the key findings.

We hope that it is useful for you to compare your experience to the experience of grant holders
generally and we would also greatly value your input in interpreting the findings and drawing out
what you have found interesting or surprising. We are holding a webchat with grant holders for this
purpose on the 8th November between 12.30-1.30, which will focus on thinking through the most
important challenges that grant holders have identified in the survey. You can find out more about
joining the webchat on the learning network events page.

Satisfaction with LSF so far

Overall satisfaction with the LSF programme so far was high with 53% very satisfied and 43%
quite satisfied (NB: dont knows have been removed). Grant holders were especially satisfied with
the clarity of LSF aims and the LSF model. The only area with substantial dissatisfaction was the
receipt of funds with 27% either quite or very dissatisfied.

Chart 1: How satisfied are you with your organisations experience of the following
aspects of the LSF programme so far? (%)

The LSF programme overall 53 43 31

The LSF model 54 36 6 31

Communication from the Big

44 41 13 11
Lottery Fund
LSF administration (excluding
42 44 10 3
the receipt of funds)

The receipt of funds 21 37 16 19 8

The application process 45 46 8 11

The clarity of LSF aims 60 36 4

Very satisfied Quite satisfied Neither satisfied nor dissatisfied Quite dissatisfied Very dissatisfied
The most important challenges so far
In the open response questions, grant holders identified a number of key challenges that they
have faced in the first few months of their LSF project. The top four most important challenges
were:

Delay in the receipt of funds

This challenge included the difficulties caused to workload planning (such as LSF now running into
other projects), financial planning (such as not being able to pay advisors), recruitment difficulties
(such as having to delay recruitment until receipt of funds and potentially losing good candidates)
and sometimes undermining relationships with staff, volunteers, advisors and business partners
who were expecting the LSF work to begin.

For one grant holder delays in payment have meant that our start date clashed with the start
dates of two new projects and this has led to further delays. Every delay has significant impacts
and knock-on effects for other areas of work and capacity and for another delays in payment of
the funding, meant that we had a significant challenge in recruiting and employing [a Manager] to
take the LSF project forward. Whilst we had interviewed for the post and identified the most
appropriate candidate we were unable to give them a formal job offer as we had not received
notification that funds had been received.

Ensuring sufficient time is devoted to the LSF project

Grant holders faced a number of specific challenges around time including other big pieces of
work having to be delivered and the general demands on staff time to deliver rather than engage
in more reflexive activities. A number of responses suggested that some grant holders had not
built in enough staff time in their LSF budget and were therefore having to fund this themselves.

As one said none of us have the luxury of spare time and we have had to run this scheme
alongside the normal day to day demands of the operational aspects of the work. We recognise
how worthwhile it is and have given it priority, but the result has been many additional hours
worked above our normal week and another acknowledged that once we really got into the
project, it became apparent how much more work there is to do if we want to get it right and more
capacity needs building in than we originally thought which has been challenging.

Interestingly, despite the delays in the receipt of funds, the majority of grant holders (60%) started
their LSF project before they had received funding in June and most are sticking with the 12 month
project span.

The advisor role and relationship

The advisor role is a key element to the LSF model and will play a big part in underpinning the
success of individual projects (see assumption 2 in the LSF Theory of Change). There were a
number of challenges highlighted around this role including challenges in the initial recruitment
process and in establishing the relationship (including difficulties in knowing how much
organisational background the advisor requires and building a relationship in a short space of
time).
For one grant holder the most important challenge was helping the advisor understand the
organisation enough to make positive contributions and another found that the amount of time it
takes from core staff to ensure the advisers and business partners have a full understanding of the
organisation and its potential was the biggest difficulty they faced.

Others had faced challenges in establishing shared goals and setting clear objectives that will
benefit the organisation. For one, their most important challenge was that the advisor was simply
providing us with reports etc. rather than something more hands on that can really help us when
we have so few staff to take their recommendations forward. Another found that shifting the mind
set of consultants to move away from them giving us what they think we are asking for - and often
just giving us back some of the story that we would have provided them with at the initial meeting -
to them actually using their expertise and knowledge to 'teach' us something new, innovative,
experimental, novel.

Ensuring engagement across the organisation

A number of grant holders have faced challenges engaging trustees, management, staff,
volunteers, users and some partners in their LSF project. Partly this was seen as due to the
specific nature of the LSF programme.

One argued that because the LSF is not a traditional 'project', the main challenge has been to
galvanise the SMT around the opportunity and to take shared and collective ownership of the
project and another felt that the single most important challenge was the staff accepting the
introduction of the consultants to review, provide training and enable [us] to become contract
ready. Partly, these challenges were related to broader difficulties of enacting change within
organisations as one felt their most important challenge was convincing staff and trustees to work
in a new and different way.

The Online Diagnostic Tool

Grant holders were positive about the overall accuracy of the ODT results with 32% seeing them
as very accurate and 63% seeing them as quite accurate (NB: dont knows have been removed).
The ODT was seen as especially accurate in terms of identifying areas that require development
(45% felt this was very accurate).
 Chart 2: How accurate do you feel the following aspects of the ODT results were for
your organisation? (%)

The overall accuracy of the ODT results 32 63 41

Identifying areas that require development 45 49 52

The link between your answers and the results 29 62 8 1

The understanding of sustainability that was implied by

31 63 42
the questions

Very accurate Quite accurate Neither accurate nor inaccurate Quite inaccurate Very inaccurate

Fifty-one percent of grant holders found the overall ODT process and results very useful (NB:
dont knows have been removed). In particular, grant holders found the process of thinking
through the questions posed by the ODT (69%) and informing the development of your LSF
application (66%) very useful. Grant holders found the ODT as less useful at telling them things
that they didnt already know (27% found this element of the ODT not very useful or not at all
useful). These results generally supports assumption 1 in the LSF Theory of Change, which
assumes that the ODT provided a valid assessment of an organisation.

Chart 3: How useful were the following aspects of the ODT for your organisation? (%)

The overall usefulness of the ODT process and results 51 44 5

Telling you things that you didn't already know 14 59 23 4

Having an easy to understand numerical score around different

55 37 8 1
aspects of your organisations sustainability

Informing the development of your LSF application 66 29 5

The process of thinking through the questions posed by the ODT 69 29 2

Very useful Somewhat useful Not very useful Not at all useful

Overall, 90% of grant holders would recommend the ODT to other organisations to use (only 3%
wouldnt recommend it). As such, the vast majority of open responses around the ODT related to
its strengths. In particular, grant holders gave the following key strengths:
 o The clarity of quantification and prioritisation of different aspects of the organisation;
o As a measure of sustainability;
o As a reflective process;
o The ODT provided a challenge to the organisation / a different perspective on the
organisation.

Grant holders (including those who were positive overall) identified a number of key weaknesses
to the ODT including disagreement with specific findings or feeling that the questions didnt get to
the heart of sustainability, feeling quantification was too crude for some areas (although this may
be a symptom of a broader scepticism about the ability of any ODT to quantity different elements
of an organisations operations) and the seeing the ODT as being a useful moment in time
process but not functioning as a living document that informs organisational behaviour on an
ongoing basis.