Documenti di Didattica
Documenti di Professioni
Documenti di Cultura
OCTOBER 2017
1
Contents
1. Introduction and key findings ................................................................................................... 3
1.1. Grant holder profiles ...................................................................................................... 4
1.2 Potential routes to sustainability .................................................................................... 5
2. Change in ODT score ................................................................................................................. 8
2.1 Overall ODT score ............................................................................................................ 8
2.2 ODT sub-scores .............................................................................................................. 10
3. Thematic analysis .................................................................................................................... 13
3.1 Stage of sustainability.................................................................................................... 13
3.2 Size of organisation ....................................................................................................... 15
3.3 Change in volunteer numbers ...................................................................................... 15
3.4 Generating greater income ........................................................................................... 17
3.5 Improving systems ......................................................................................................... 17
3.6 Age of organisation........................................................................................................ 17
4. Sub-score analysis ................................................................................................................... 17
4.1 Market & opportunities ................................................................................................ 17
4.2 Track record & capabilities............................................................................................ 19
4.3 Quality & impact ............................................................................................................ 19
4.4 Sustainability .................................................................................................................. 20
4.5 Strategy & planning ....................................................................................................... 21
5. Conclusions .............................................................................................................................. 21
2
1. Introduction and key findings
Key findings
Across all grant holders the change in the overall ODT score ranged from a 33% increase
to a 7% decrease.
The average (mean) change in score across all grant holders was a 11% increase (average
score in the 'pre' project ODT score was 66% and 76% in the 'post' project score).
The vast majority (93%) saw an increase in their score over the course of their LSF project
and only 7% saw a decrease.
Of the five different sub-scores within the ODT, Track Record & Capability saw the highest
average increase of any, at 12%.
Organisations classified by the evaluation as 'Growers' (i.e. experiencing increasing
income prior to LSF) generally experienced lower levels of changes in ODT scores than
organisations classified as 'Squeezed middle' (little or no growth in years prior to LSF) or
'Survivors'(declining income in years prior to LSF).
Larger organisations (i.e. those with an annual income of over 500,000) were more likely
to have seen bigger positive changes in their sustainability score.
There is some evidence of increases in the number of volunteers over the course of the
LSF project and higher levels of overall ODT score growth, but only for larger
organisations with income over 250,000.
Our analysis
The purpose of this report is to analyse the Online Diagnostic Tool (ODT) scores for grant
holders, with specific focus on how scores have changed over the course of Local
Sustainability Fund (LSF) projects and what characteristics are associated with changing
scores. To do this we have compared the score that grant holders got when they completed
the ODT during the application stage in 2015, before their projects began (the pre score)
with the score they got after they had completed their projects (the post score). This
represented a period of at least 12 months, but up to 15 months in some cases. As well as an
overall score representing the holistic level of sustainability an organisation has, the ODT
also produces five sub-scores which focus on some distinct organisational areas, these
being: Sustainability, Market & Opportunities, Strategy & Planning, Track record & Capability
and Quality & Impact.
3
The analysis in this bulletin will focus on comparisons between pre- and post- scores in the
ODT for grant holders.1 Where relevant it will refer to other data from the rest of the
evaluation of LSF to help contextualise or explain findings from the ODT data.
The ODT was developed by Creative Co-op, on behalf of the Big Lottery Fund (The Fund). The
scores have been made available to NCVO to analyse. The analysis in this bulletin is based on
scores completed by 227 grant holders. Further detail on the ODT and our analysis of the
data is available in Appendix A. The appendix also describes a small bug within the tool itself
which inflated some of the scores slightly; we agreed with The Fund to continue our analysis
with this data as the inflation applied to both the pre- and post-scores so did not affect our
findings.
Before presenting the detailed results of the analysis, it is useful to highlight two key
observations we were able to make about grant holders after examining the ODT data.
These observations relate to the organisational profiles of grant holders at different levels of
sustainability and the potential routes grant holders can take towards building sustainability.
High Growth N= 51 High growth grant holders were most likely to:
1
Throughout this report any statistically significant findings will be flagged, all other reported relationships or
changes should be seen as directional or indicative.
4
Moderate Growth Moderate growth grant holders were most likely to:
N= 126
- Be a Squeezed middle organisation
(6% - 15% overall ODT - Have an income of over 250,000 (grant holders with an
score increase) income of 250,001 - 500,000 most likely to be seeing
6% - 10% growth, whereas grant holders with an income
of 500,001+ more likely to be seeing higher growth, of
11% - 15%)
- Have a Lower Mid or Upper Mid level of sustainability
from the pre ODT (grant holders seeing 6% - 10%
growth more likely to be Upper Mid, those seeing 11%
- 15% growth more likely to be Lower Mid)
- Have maintained the same number of volunteers over
its LSF project
- Be aiming to generate greater income through their
project, particularly through developing new funding
streams and diversifying income
- Those grant holders seeing the upper half of moderate
growth, 11% - 15%, are also more likely to have included
Impact assessment in their LSF project than not (52.4%)
Negative/low Growth Negative/low growth grant holders were most likely to:
N= 50
- Be a Squeezed middle organisation, although around a
(5% and below overall quarter of Negative/low growth grant holders are
ODT score change) Growers (24%)
- Have an income of less that 250,000
- Have an Upper Mid or High level of sustainability
from the pre ODT
- Have maintained or reduced its number of volunteers
over the LSF project
- Not be aiming to improve systems via the LSF project
5
Closed to open
Whether an organisation is Closed or Open in their outlook impacts upon sub-scores
differently and may indicated that grant holders go through a process of moving from a
Closed organisation which is linked to stronger improvement in the track record &
capabilities sub-score to becoming more Open which in turn is linked to improvements in
Market & opportunities score. This typology of how open or closed organisation are, in
terms of user involvement as well as their pursuit of partnerships or relationships, was
developed as a part of the Further analysis of ODT and application data bulletin and is
described further in Section 4.1
It could be inferred that a grant holder may begin their journey as a Closed organisation
and utilise this more inward-looking stage to focus on its own developing its internal
capabilities and the ability to demonstrate the effectiveness of their work. Having sufficiently
improved in these areas, focus then shifts on to more external issues such as partnerships
and surveying the market for opportunities, thus in becoming a more open organisation the
grant holder will begin to improve another element of sustainability.
The flow chart visually below demonstrates how this journey may work for some grant
holders.
6
Improving systems
Grant holders which aimed to improve systems through their LSF project tended to be more
likely to see higher levels of growth on two of the five sub-scores. The fact that improving
systems appears to help grow both Quality & impact and Track Record & Capabilities scores
implies that it could be the first step in the process of building sustainability for some grant
holders. Central to this is building the capacity of organisations in order for them to be able
to take new projects or develop new ways of working which ultimately help grow the
organisations long-term sustainability.
The flow chart below visually demonstrates how the relationship may exist and that if
properly implemented, improvements to key systems could make a real difference to the
sustainability of grant holders.
7
2. Change in ODT score
2.1 Overall ODT score
The overall ODT score serves as a holistic indicator of sustainability for each grant holder and
is derived from a range of questions asked as a part of the ODT. Looking at results for all
grant holders that completed the ODT at the start and end of their LSF project, the change in
overall score ranged from a -7% decrease to a +33% increase. Whilst bearing in mind this
variation, it is also useful to examine the average level of change in score in order to get an
indication of the overall levels of change being seen by organisations across the network.
The average change in score across all grant holders was a +11% increase, with average
score increasing from 66% in the first run of the tool to 76% in the post run.
The overwhelming majority of grant holders have seen some increase in their overall ODT
score over the course of their LSF project (94%) and only a small proportion have seen either
zero or negative growth (6%).
While it is not possible to directly attribute the increase in score for organisations to their
involvement in LSF, the scores do suggest that the majority of organisations have undergone
a positive journey in this period. Any changes in organisational sustainability are likely to be
relatively slow and will take time to be seen, so such changes observed here are
encouraging. Other data from the evaluation, including results from the snapshot surveys
and analysis of the end-of-project monitoring reports also suggest many organisations have
started to see notable and meaningful changes in their organisational sustainability. It is
worth noting, however, that while these changes in scores suggest a positive experience for
8
most grant holders, the scores themselves are subjective (based on self-reporting) and
somewhat approximate, which is a necessity of a tool of this nature. Grant holders we spoke
to in this evaluation tended to recognise that sustainability is a journey rather than an end
point, therefore it is not necessarily possible or desirable to achieve 100% sustainability
as these scores could suggest. Nonetheless, an increase in the percentage score can, for the
purposes of this bulletin, be equated with progress along a journey towards greater
organisational sustainability.
The table and chart below show grant holders grouped into five categories according to the
size of change in overall ODT score.
1% - 5% Growth 37 16.3
9
Change in overall ODT score (%)
-7% 0% Growth 1% - 5% Growth 6% - 10% Growth 11% - 15% Growth 16%+ Growth
For much of the analysis in this report, we will use the two combined sub-groups shown in
italics: organisations with Negative/low growth (a change in score of between -7% - +5%)
and Moderate/high growth (change in score of 6% and above). That provides sufficient
sample sizes in the two groups to be able to analyse differences between them, with the aim
of identifying key characteristics of those grant holders that have increased their score the
most. Just over three-quarters (78%) of grant holders fall into the Moderate/high group
with just under a quarter of grant holders seeing Negative/low growth (22%).
It is interesting to note that some (albeit a small minority) of grant holders experienced a
worsening situation with regard to their organisational sustainability during this period. As
with any positive growth, it is not possible to determine whether this decrease was the
result of LSF or was associated with wider environmental factors, although the latter is a
more likely explanation. Many organisations have, in other parts of the evaluation, noted
that the wider environment in which they operate remains challenging and has, in some
cases, acted to limit their progress towards achieving greater organisational sustainability. As
part of this analysis we also looked at the end-of-project forms from those organisations that
reported negative changes in score, although there was nothing notable to help explain this
change.
10
fed in to the overall sustainability score (section 2.1), which was automatically calculated by
the ODT.
The five sub-scores will now be reported on individually, to examine changes between the
pre and post ODT runs. The same groupings of Negative/low growth (change in score of
+5% and below) and Moderate/high growth (change in score of 6% and above) will be used
to help analyse each sub-score.
As the chart below shows that on average all sub-scores saw an increase and the average
level of change ranged from an +8 increase for Market & Opportunities to an +12% increase
for Track record & Capabilities.
11
Change in average ODT score from pre to post(%)
90
79 80
80 76 76
68 67 65 68 68 66
70
60 59
60
50
40
30
20
10
0
Market & Quality & Strategy & Sustainability Track record Overall ODT
opportunities impact planning & capability score
Pre Post
Sustainability
Change in sustainability score ranged from -10% decrease to +37% increase and the average
change in score across grant holders was +12%.
Over nine in ten grant holders (93%) saw some level of increase in sustainability score, with
7% seeing a decrease or zero growth. Just over three-quarters of grant holders (77%) saw
Moderate/high levels of growth (6% and above) with 23% seeing Negative/ low levels of
growth (5% and below).
This sub-score has the highest proportion of grant holders that have seen zero or negative
change in score at 14.5%, consequently around a third (34%) saw Negative/low growth.
Two-thirds (66%) of grant holders saw Moderate/high levels of growth.
A large proportion of grant holders (92%) saw an increase in this score, with less than one in
ten (8%) seeing zero or negative growth. Furthermore, 78% saw Moderate/high growth and
23% saw Negative/low growth.
12
Track record & capability
Track record & capability score saw the widest rage in score, ranging from -9% decrease to
+38% increase. The average increase was 12%, the highest average increase versus other
ODT sub-scores.
This sub-score also had the smallest proportion of grant holders that saw Negative/zero
growth (6%) or Negative/low growth (22%). In addition, 94% saw some growth in sub-score
and 79% saw Moderate/high growth.
This sub-score saw the largest proportion of Negative/zero growth at 11%, with a third of
grant holders (33%) seeing low/negative growth. In terms of growth, 89% saw some increase
in sub-score and 67% saw a Moderate/high increase.
3. Thematic analysis
In this section we will be analysing both overall ODT score and sub-scores according to a
range of themes that we have explored and developed in previous bulletins as part of this
evaluation. These range from the stage of financial sustainability grant holders were at the
start of the LSF programme to specific characteristics of their LSF project.
1. Growers those organisations who have seen consistent growth in income over the
period 2013 to 2015 (26 organisations, 13% of grant holders);
2. Squeezed middle those organisations who have seen little or no growth in income
over recent years (168 organisations, 74% of grant holders);
3. Fighting for survival or Survivors those organisations who have seen sharp
declines in income over recent years (29 organisations, 13% of grant holders).
Using this classification in combination with level of ODT score change allows us to identify
relationships between score growth and stage of sustainability at the start of the LSF
programme. As with earlier findings in this bulletin, this does not, however, allow us to
13
determine causation; we are not able to say the extent to which any observed changes in
score were the direct result of involvement in LSF.
When comparing the levels of growth between stages of sustainability groupings it becomes
evident that those grant holders who were Growers when first taking the ODT are less likely
to be seeing Moderate/high levels of growth than the Squeezed middle or Survivors. As
shown in the chart below, around four in five Squeezed middle grant holders (82 %) and
three-quarters of Survivors (77%) were seeing Moderate/high level growth versus only
59% of Growers. Instead Growers are more likely to have seen Negative/low growth than
other groups (41%). This could possibly be because Growers might be further along on their
journey towards sustainability and therefore have less far to travel in the future; this will be
examined in more detail in the final evaluation report.
80
58.6
60 76.9 82.1
40
20 41.4
23.1 17.9
0
Survivors Squeezed middle Growers
Negative/low growth Moderate/high growth
This pattern is also observed across all the ODT sub-scores, with Growers being least likely
to see Moderate/high increases in score compared with Survivors and the Squeezed
middle. It is particularly evident for the Strategy & planning score, where 55% of Growers
are seeing Moderate/high growth versus 77% of Survivors and 82% of the Squeezed
middle. The differences are least pronounced when looking at sustainability score:
Growers perform strongest on this sub-score versus other sub-scores with 62.1% achieving
Moderate/high score growth however they remain behind Survivors (77%) and the
Squeezed middle (79%).
It should be noted that the fact that Growers are seeing lower score growth is not
necessarily a sign that that they are underperforming as a group and instead could be an
indication that they are maintaining their already high(er) ODT scores, whilst Survivors and
the Squeezed middle are making progress in pushing up lower scores from the pre ODT
14
run. If we look at the description of Growers from the Needs Analysis and Thematic
Mapping report we can see that their LSF project was seen by many as a way of adapting to
growing pains that are associated with the sustained growth in income that these grant
holders had seen. Thus relatively low levels of growth during this period could be linked to
mitigating the rapid rises they have seen in different aspects of sustainability.
This also raises an interesting dilemma about which organisations to support around growth
in organisational sustainability, or more precisely, at which stage of their sustainability
journey would targeted support yield the greatest benefits or returns. The data suggest that
organisations at an earlier (or less sustainable) part of their journey have the most to gain
and therefore an investment from a programme such as LSF may demonstrate a bigger
impact for such organisations, whilst an organisation that is more stable or more sustainable
(i.e. the growers) may have less to gain. However, there is also the question of whether
programmes such as LSF would see it as more desirable to support organisations that are
more stable (but with less to gain) or organisations that are more vulnerable and unstable
(but with potentially more to gain). These issues will be examined in more detail in the final
evaluation report.
This pattern was also present to varying degrees across all ODT sub-scores other than for
Quality & impact. Interestingly for this sub-score grant holders with <250,000 were the
income band most likely to have seen Moderate/high score growth (72%) although the
differences in proportion were relatively small, with 250,001 - 500,000 being least likely to
see Moderate/high growth (63%). Whilst only being indicative, this finding may suggest
that smaller organisations are more concerned with improving the ways that they can
demonstrate the value and impact of their work than larger organisations.
15
volunteers. Furthermore, the number of volunteers a grant holder has could potentially be
an indication of organisation size, which has some impact on level score change itself.
8.8
10 5.9 6.7 6.7
4.4
0
Decrease No change Increase
-7% - 0% Growth 1% - 5% Growth 6% - 10% Growth 11% - 15% Growth 16%+ Growth
However, it should be noted that it is organisations with an income of over 250,000 that
appear to be driving this relationship. Although overall there is a moderately strong and
statistically significant correlation (R=0.25, p <.05,) between change in number of volunteers
and growth in overall ODT score, there is no such correlation for the smallest income grant
holders (<250,000). For the 250,001 - 500,000 group, on the other hand, the correlation
is more positive than the overall level (R=0.3, p <.05,). This indicates that it is size of
organisation in combination with change in volunteer numbers explaining this relationship,
rather than solely change in volunteer numbers.
This positive relationship exists across all ODT sub-scores but is most pronounced in terms of
the Market & opportunities score, where another moderately strong positive correlation
(R=0.27p <.05,) was found.
16
3.4 Generating greater income
By coding grant holder application forms, we were able to classify which organisations were
aiming to generate greater income through their LSF project; 134 grant holders in total.
Comparing those grant holders who were planning to generate income with those who were
not showed little difference in level of score growth between the two groups. The coding of
application forms also allows us to classify how grant holders were planning to generate
income, either through developing existing funding streams or by developing new funding
streams and diversifying income. Again, in terms of overall ODT score there were limited
differences observed between groups. However, for the Market & opportunities sub-score
there were some interesting findings, explored in section 4.
4. Sub-score analysis
This section further explores the data around some of the ODT sub-scores outlined earlier in
the bulletin and builds upon findings highlighted in Section 3 that are related to these. The
Market & opportunities, Track record & capabilities and Quality & impact sub-scores all
exhibit some interesting relationships with various characteristics of LSF project that do not
necessarily exist for overall ODT score. These will be examined below.
17
At an overall level, 69% of orgs that planned to generate greater income in any way
improved their market and opportunities score by a Moderate/high amount, compared to
61% of those that did not. This difference in Market & opportunities score was greater for
organisations that were ONLY diversifying and developing new funding streams, 73%
compared with 61% for those not including this in their project. Organisations that planned
to develop existing income were less likely to see Moderate/high growth in score (60%) than
those that did not include this activity in their application (68%). The chart below shows the
variation in Moderate/high score growth for Market & opportunities, with those
organisations who were developing existing funding streams having reduced scores (by 8%)
compared with increases of between 8% and 12% for those that did not.
15
11.8
10 8.3 8.6
5
Developing existing
funding streams
0
Generating greater Developing new funding ONLY developing new
income (total) streams & diversifying funding streams &
-5
income diversifying income
-10 -8.2
18
possibly involved in local networks, there were gaps in their links to others and they lacked
active involvement with partnerships or networks which would enhance the organisations
knowledge, skills or opportunities.
Around nine out of ten grant holders which were improving systems saw Moderate/high
score growth (88%) against 72% of those that were not. This finding indicates that improving
systems is a potential route for grant holders to build their capacity, something that
ultimately helps organisations to improve their sustainability and thus their overall ODT
score.
19
did not. At an overall level, 81% of organisations undertaking impact assessment saw
Moderate/high levels of growth compared with 56% of those that were not, demonstrating
a clear difference between the two groups.
Furthermore, the purpose of the impact assessment can also be ascertained from our coding
of application forms, falling into two main groupings: impact assessment for funding, and
impact assessment for service development. The chart below shows the levels of growth
across the types of impact assessment used and shows that impact assessment for service
development saw a slightly higher proportion of Moderate/high score growth than impact
assessment for funding.
However, almost a third (32%) of those grant holders employing impact assessment for
funding were likely to see the highest levels of score growth (16%+), more than twice the
proportion of those that were not (14%). A similar relationship existed for among those who
were using impact assessment for service development, with 29% achieving 16%+ growth
compared with 16% of those that were not.
80
40
20
18.7 18.5 12.2
0
Impact assessment (total) Impact assessment for Impact assessment for
funding service development
Negative/low growth Moderate/high growth
4.4 Sustainability
In terms of the Sustainability sub-score, the findings of note have been covered in Section 3
following the same patterns as overall ODT score.
20
4.5 Strategy & planning
In terms of the Strategy & planning sub-score, again the interesting findings have been
covered in Section 3 following the same patterns as overall ODT score.
5. Conclusions
This bulletin has analysed the different levels of change from pre to post in ODT score and
sub-scores, as well as exploring some of the factors that may lie behind the differing levels
observed. The potential linkages between sustainability and various key themes such as size
of organisation and the different aims of LSF projects were examined. The five sub-scores
were then also investigated to uncover any relationships that were specific to any specific
element of sustainability. The analysis undertaken allowed us to make some speculative
conclusions about both the potential routes grant holders have taken towards improving
sustainability and the broad profiles of grant holders which have achieved different levels of
score growth. These key findings were discussed in Section 1 of the bulletin.
21
Appendix: The Online Diagnostic Tool and our analysis
266 grant holders completed the ODT before their projects and 227 had completed it by the
end of July 2017, the point at which we analysed the data. No individual organisations were
named in this analysis but unique identifiers have been used to help us identify the change
between different categories of organisations.
The ODT contains a variety of different scores, including an overall measure of organisational
sustainability and multiple sub-scores which feed in to this over-arching score. This bulletin
reports on a variety of different measures when comparing the pre and post scores,
highlighting where there appears to be an interesting change and also where no change (or a
negative change) has been seen.
In July 2017 a bug was discovered in the program underlying the tool; in the score
calculations one question had been over-represented, resulting in the calculated scores
being higher than they should have been. The bug was fixed by Creative Co-op on 27th July
2017. The direct impact was on two indicators: Indicator 4, Financial Systems delivering,
where the score for that indicator was overstated by up to 18 percentage points and
Indicator 1, Cost base control, where the score for that indicator was overstated by up to four
percentage points. These indicators fed into three different scores, the Sustainability,
Strategy & Planning and Track Record & Capability scores, all of which were overstated by
three to four percentage points. The overall diagnostic score was overstated by up to two
percentage points. The error occurred consistently for the ODT scores for LSF grant holders
who used the tool before they received funding and after the end of their 12-month
projects, so the comparison between the before and after is still valid. Given this, and
given the fact that the overstatement on the overall diagnostic score was relatively small, we
agreed with the Fund to continue using the uncorrected scores for the analyses in this
bulletin, for the previous bulletin on the 'before' scores, and for the evaluation overall. To
ensure consistency a small number of organisations (five) were excluded from the
final analysis because their later scores had been calculated on the basis of the corrected
program after July 27th meaning the before and after scores would not have
been comparable.
22