COMMUNICATION STRATEGY/PLANNING - Monitoring and Evaluation - Employee and Organisational Development

Monitoring and
Evaluation
Evidencing Effective
Communications
within an Organisation

onitoring and Evaluation (M&E) plays an important role in developing

Marketing and Communications for your business.
M&E assures that we have statistical evidence to show what works and what
does not work and should be an ongoing exercise which features prominently
in your overarching communications strategy. M&E can also be used to engage
with employees and you can incorporate data collection for M&E purposes with
engagement, collaborating and working with employees and involving them by
using their opinions and perceptions to inform Marketing and Communications
development in your organisation.
From an external communications perspective, it is imperative that we know
how our stakeholders perceive the company. It is easy to make assumptions
about what people want to know without really investigating how they are
accessing channels and what information they are consuming. In order to gain
a clear picture, there should be an alignment with the strategy from a clear and
informed perspective, focussing on the needs of the business and the
requirement of clients and stakeholders rather than simply basing our decision
on trends or our own assumptions. One size does not fit all and each

organisation will require a different approach to gathering data.

NB This article is about evaluating client and stakeholder (internal and external)
communications. Take into consideration that M&E techniques for these purposes are
different to research about how customers/consumers perceive the service they have
received from the company. (Consumer research article to follow shortly).

What is already available

Part one in the Communications Strategy series already outlines some of the
techniques you can use to establish what M&E your company may already have
in place. Monitoring and Evaluation can be a lengthy process requiring
considerable amounts of time and resources, however there are ways in which
you can minimise cost and time by looking at what you currently have and using
existing data to build a foundation or baseline. From this point, you can look at
how you present the data and plan how you will compare data in the future.
Depending on the software and the expertise of your digital/web team, (if you
have one) it should be fairly easy to look at what people are reading on your
internal and external channels. For example, you may have an external blog in
place and want to know how many people have read it over a certain
period of time. Or you have a regular e-newsletter that you send to a certain
stakeholder group and you would like to assess whether it is being read in order
to consider changes to the channel or the content. You may want to show how
readership is rising on a particular channel and monitor whether there are any
particular trends and the potential reasons for this.
You may also want to conduct a review of your channels and present information
in a different way and M&E will help to inform this process. If a blog on your
external channel is not being read, is it because it hasnt been promoted
effectively? How are people reaching the blog? And are you optimising the
digital space by linking in from other areas (if appropriate). If the newsletter you
are disseminating is only being read by 25% of people on the database, is your
stakeholder database up to date? How many bounce backs are you receiving
and is the information still relevant for the target group? This kind of data paints
a picture and is invaluable in looking at how you plan communications and what
changes you will need to make to ensure that you are optimising your channels
and content.

Qualitative V Quantitative

There is no battle between these 2 types of research. Each is beneficial in its own
way and the type of research you choose depends on your M&E planning and
the objectives of your research. I like to use a combination of both, however this
may not be necessary and depends on how you choose to evaluate marketing
and communications in your organisation. Lets break down the 2 different types
of research and explore how you would use each type to support your Internal
and External communications planning. For the purpose of comparison, the
definitions given on the snap survey website give a comprehensive explanation
of both research types. NB In the case study example on the right, quantitative research

has been used as a baseline for further exploration using qualitative research - The definition
below suggests that this order is reversed, but (in my opinion) the order of research - if you are
using both- is interchangeable.

Qualitative Research is primarily exploratory

research. It is used to gain an understanding of
underlying reasons, opinions, and motivations.
It provides insights into the problem or helps
to develop ideas or hypotheses for potential
quantitative research. Qualitative Research is
also used to uncover trends in thought and
opinions, and dive deeper into the problem.
Qualitative data collection methods vary using
unstructured or semi-structured techniques.
Some common methods include focus groups
(group discussions), individual interviews, and
participation/observations. The sample size is
typically small, and respondents are selected
to fulfil a given quota.

Quantitative Research is used to quantify the

problem by way of generating numerical data
or data that can be transformed into usable
statistics. It is used to quantify attitudes,
opinions, behaviours, and other defined
variables and generalize results from a
larger sample population. Quantitative
Research uses measurable data to formulate
facts and uncover patterns in research.
Quantitative data collection methods are
much more structured than Qualitative data
collection methods. Quantitative data collection methods include various forms of surveys
online surveys, paper surveys, mobile surveys and kiosk surveys, face-to-face
interviews, telephone interviews,
longitudinal studies, website interceptors, online polls, and systematic observations.

So, when would you use Quantitative and Qualitative research to inform your
communications planning? A few examples are provided in the following
case studies.

Case Study 1
Research Type - Quantitative
Marketing and Communications role (Internal)
You want to run a quick survey to gain an initial impression about
perceptions of employee communications. You want to find out how well
people feel they are being informed and if they understand the objectives of
the business. You decide that you will create a 4 question survey with a view
to gathering responses and following up at a later date to observe whether
perceptions have improved after you have made some changes to internal
channels and content. Due to limited resources and time, you decide to use
outlook software to create a questionnaire and send to all staff. You pose
the following questions;
1. On a Scale of 1-10, rate the last internal newsletter content
(1 being poor and 10 being optimal)
2. Rate how well you feel informed about the organisations values;
Very Informed
Quite Informed
Could be better informed
Not informed at all.
3. How often do you access the intranet?
More than once a day
Once a day
Once every couple of days
Once a week
Never
4. On a scale of 1-10 how do you rate Internal communications?
(1 being poor and 10 being optimal )

COMMUNICATION STRATEGY/PLANNING - PERCEPTION INTEGRATION

Case Study 1 Continued

Research Type - Quantitative and Qualitative
Integratiion
You decide that you want to delve a little bit deeper and give people the option to
expand upon their answers to the questions. You add a box after each
question taking a qualitative approach and prompting further detail from
respondents. Your survey would then look something like this;

How often do you access the intranet?

More than once a day
Once a day
Once every couple of days
Once a week
Never

What do you access and why

1. On a Scale of 1-10, rate the last internal newsletter content

(1 being poor and 10 being optimal)

What article stood out to you and why?

On a scale of 1- 10 how do you rate Internal communications?

(1 being poor and 10 being optimal)

What do we do well and how can we improve?

2. Rate how well you feel informed about the organisations values;
Very Informed
Quite Informed
Could be better informed
Not informed at all.

If you answered could be better informed or not informed at all, what could we do
better to keep you informed

Looking at case study 1, we can see that the initial 4 question survey would give
us a snapshot of perceptions, opinions and preferences, but wouldnt tell us
much more. Allowing respondents to give reasons for their answers, we gain
further insight into motivational factors that encourage people to access certain
channels and a deeper understanding about the motivations behind the answers
to the quantitative questions. This gives your research another layer and added
depth.
Quick surveys can be beneficial if you have recently conducted a larger
evaluation project and want to follow up on areas of interest. They are also
useful as a starting point for deeper evaluation (especially if time is pressing).
The next case study observes different data gathering techniques using
quantitative and qualitative research methods in a new scenario.

Case Study 2
Research Type - Qualitative
Marketing and Communications Role - Social
Media Strategist
You have been asked to conduct a social media review in order to find out
how people are using social media channels in your company, what
comments they are making, where they are from and how the organisation
is responding. Your organisation wants to make sure that they are using all
the channels appropriately and optimising social media in order to engage
with stakeholders effectively. Your organisation would also like to know how
other organisations are using their social media channels and would like you
to write a document rationalising and comparing social media use in your
organisation. You have a digital team who you lead and manage and you
have decided to draw on their direct knowledge and expertise in order to
gather data for analysis to inform your paper.
You establish a time limited working project group inviting members of your
digital team to give their initial impressions of the current use of social
media in the organisation. In the first meeting, you ascertain what channels
are being used and note your teams opinions about what improvements the
business could make to enhance their social media presence. You ensure that
the meeting is structured and have a clear idea about what questions you
want to ask. (The qualitative data comes from the notes you make from your
project group). You also decide to follow up using quantitative
questions via a survey sent to your working group for further analysis.

Case Study 2 Continued

Research Type - Quantitative

You begin to make decisions about how to structure the data and start by clearly
showing what current social media channels your business uses, differentiating
between internal and external channels. You create a table showing the following
information.
Social Media Channels -External e.g.
Twitter
Facebook
Blogs (embedded into website)
Social Media Channels -Internal e.g.
Yammer
Forums
Blogs
You breakdown how often people are accessing the channel and ask your digital
team to provide you with the relevant data over a specific time frame. You also
request historical data in order to chart the progression of social media in your
business and look for trends in how it has been accessed over a period of time.
You also want to know what people are saying on these channels and ask your
team to compile data on all social media channels highlighting what people are
saying over a period of time.
Taking a thematic approach for ease of reference, you keep a note of all comments
using a spreadsheet. You decide to create a bar chart with the themes you have
developed in order to clearly show dominant topics and areas of interest. You also
make note of responses to comments given by the business you work for.
You find a comparable business (or ask you digital team to suggest one) and
observe their social media channels, looking at how they are being used in order to
draw a comparison using the same baseline criterion you have set for observing
your companies social media.
You review the notes from your digital working group about perceptions of social
media and look at the quantitative data you have gathered in order to write your
paper.

(All the above are quantitative data gathering techniques using existing and historical data
through systematic observation

COMMUNICATION STRATEGY/PLANNING - Monitoring and Evaluation - Employee and Organisational Development

COMMUNICATION STRATEGY/PLANNING - Monitoring and Evaluation - Employee and Organisational Development

As you can see from the case study examples, M&E can take many forms and be
used for different purposes.

What Software

There are many types of software you can use to develop a survey and again
depends on the size of the organisation, the amount of time you have to dedicate to M&E and your budget. If you want to conduct a quick survey, it is likely
that you have the tools in-house to do so.
Outlook
Dependent on your version of outlook,
you can create surveys using your email
application. This is beneficial if you use this
program as the database of respondents is
most likely embedded into the same
application. If you are pressed for time
and have limited resources. Outlook is
useful in developing a basic template.
Creating Polls in Outlook
Survey Monkey
Survey Monkey is simple to use and comes with a number of customisable options for survey and questionnaire creation. Dependent on your organisation,
you may find that a particular department already has the license to use this
software. You should perhaps do a little research and ask around before you
purchase a plan. There are tips on how to create a survey, as well as clear
instructions on how to use the software.
Survey Monkey

Share Point
Ive worked in organisations who are moving over to SharePoint. Im not sure
how prevalent this is, but it does come with a lot of integrated features which
allow you to create surveys.
Sharepoint
External agencies and consultants (software)
If you are working with external agencies or consultants to develop your M&E
plan, they are likely come with their own software for survey creation and
analysis. Ill outline how to work effectively with external agencies and
consultants further on in the article.
There are many more options and some of them come with a free trial, or have
certain features that you can use for free if you dont require everything the
software has to offer.

Gaining Feedback

Most of the software mentioned above gives you the option of developing your
own template and criterion and its important that you plan what questions you
want to ask in line with your objectives.
You should also take your audience into consideration when planning your
survey and developing your template. It is perhaps a lot easier to garner interest
in completing surveys with your internal audience. You should think about
approaching your external audience in a different way.
Questionnaires and surveys can be obtrusive and time consuming and whilst you
can gain the support of your internal audience by effective engagement techniques, think carefully about how you conduct research with your
external stakeholders. Again, dependent on what type of organisation you work
for, questionnaires can be embedded in different channels.

If you are evaluating communications, e.g. how effective are your channels,
marketing material, website etc, take into consideration that this area of
research is different to how customers/clients perceive the service they have
received from the company and you will need to use a different approach. This
is out of the scope of this article. (Consumer Research article to follow)

Anonymity Assurance

Dependent on the type of survey or questionnaire (anonymity in consumer

based surveys is not so relevant). It is important that you let people know that
people will be completely anonymous when responding to questions (unless
they specifically do not mind identifying themselves and you embed this option
into your survey). This is especially important in regards to employee surveys.
You should also be completely transparent and let people know that you will
not be using data to identify anyone during the process of collection.
If you are able to see respondents name and associated response through back
office survey software, you should limit the amount of people who can identify
responders and of course be cautious if you are printing data and viewing this
on your computer.
Some people actually do not mind their responses being associated with their
name, other people are more private and this should be taken into
consideration. When compiling your data, redact any information that may
show who responders are by their response. Ethically, this is important, but also
ensures that people (internal or external) from your organisation have complete
confidence in your data gathering methods which, in turn means that they are
likely to respond to further surveys and questionnaires. If at all possible, design your survey or questionnaire using a program which does not show who
responders are. Its far better to work with data that has no name associations.

Incentivising Why should people respond

I briefly mentioned in part 1 of the Communications Strategy series that

offering incentives for responding to questionnaires can be useful in garnering
support.
Incentives go beyond the whats in it for me concept of offering something
tangible like a gift in return for information (but this can also work and
consumer based questionnaires often offer incentives to fill out questionnaires
in the form of entering a prize draw). If you are conducting an employee or client
survey and you have a robust and effective communications function in your
organisation, you could frame an incentive as the opportunity to become involved and help the organisation make important decisions about communications. If you are inviting people from outside of a communications team to
attend a forum for the purposes of gathering opinions, remember that these
people are offering time out of their remit to support you. A lunch time meeting
(with lunch), or at the very least refreshments can only be an added bonus!

Working with External

Consultants
If you are employing external
consultants to manage and lead on your
Monitoring and Evaluation
criteria an/or analysis, here are some
tips and tricks to ensure that you get
the most from the agency you choose
to work with.

Write a brief.
Your brief should be clear and succinct, letting the agency know what you want
to find out through your research and how this will add value to communications
and development in your business. If your external agency is going to analyse
the data for you, make it clear how you want this to be presented, e.g. do you
want bar charts, graphs, inforgraphics, interactive elements, film and/or a
summary of findings to illustrate the data? Let them know if you intend to
embed the data into your communications planning and ensure that there is
scope to compare findings at a later stage.
Meetings
Try to conduct a face to face meeting with your external agency in the first
instance. Its been my experience that establishing a relationship in this way sets
a good foundation for collaboration and putting a name to a face is invaluable.
Make sure you discuss how you both intend to monitor progression and
development of the M&E plan and make sure you are happy with the
format of any questionnaires.

Sometimes this is a HR lead process with little or nocommunications input. In

other organisations I have worked for, there has been a collaborative effort with
HR leading on questionnaire development and the process of analysis and
communications ensuring that the survey and subsequent results were
publicised effectively through internal channels. If your employee survey is lead
by your HR department (or any other department for that matter), approach the
leading department and request that you are invited to meetings about planning
the survey. Depending on how responsive they are to your request, you may be
given the opportunity to contribute and have a say in the questions and format
of the survey.

Consultation and Engagement Collaborative

Approaches to Gathering Data
There are many ways in which you can develop your M&E plan. An important
part of these considerations will be how you can inspire people to respond to
surveys and questionnaires.

Contact
Ensure that the agency know the right people to contact at various stages of
development. You may be the initiator or lead for the project, but have trusted
members of a team that will carry the work forward whilst you focus on other
activities. Be clear about who is involved from the outset and ensure that a
process is put in place so you are kept updated at key stages throughout
development of the M&E plan.

If you are working in, or overseeing an internal communications team,

monitoring and evaluation can actually take on a secondary role in the form of
engagement if you plan carefully. Qualitative data is useful in gaining insight into
opinions and you dont have to use the form of a written questionnaire to get
feedback from the people in your organisation.

If you work in an organisation that conducts an annual employee survey, there is

some very useful data that internal and corporate engagement professionals can
draw on to help develop their strategy. My experience of who leads employee
surveys is varied.

Think about going beyond the scope of simply focussing on communications

specific based questions. Whilst you want to understand how people are
viewing and responding to your internal or employee engagement efforts, think
a bit deeper. Remember that effective communications aligns itself with the
business, so you will want to know how well people understand the business,
not just what they think about a certain channel.

Employee Surveys and the Internal Communications

Correlation

Networks, forums and working groups developed specifically for the purpose of
gaining insight into employee perceptions are a nice way to engage with people
in your organisation.

If people do not understand the objectives of the business, further investigation

may point to why this may be the case at this point you can start to look at what
channels people are accessing and whether this has an affect on perceptions and
opinions of the business. Communications has now moved beyond the concept
of we communicate, you listen and engagement is now very much what the
word implies.
Creating groups and networks to inform a strategy and for the purpose of M&E is
an effective and inspiring way to get people involved. Make it clear that you are
the communications expertise for the company and that the final decision rests
with you and then allow people to give you their opinion about how they view
the company. You may find that discussions become quite passionate and
people will enjoy letting you know
what they think.
Structure meetings in a way that will
enable you to formulate coherent
notes and ask specific questions. You
can make a request for people to join
your group by posting a message on
your internal channels, emailing
departmental leads and asking them
to cascade a request, or invite people that you know have an interest in
communications. Ensure that you have a broad range of people working across
the
organisation at different levels. These are the people you engage with on a daily
basis, who work in the business in different areas and are the best people to ask.
*Important Make sure that you stay on track some people may want to use
your forum to air their grievances and its important that you use your (and the
people you have invited) time productively. Ensure that the meeting does not
turn into a moaning session. For issues and concerns that go beyond the remit
of your role, have a contact name and number ready!

Data Comparisons and Analysis

You have reached the stage where you have developed your plan, created a
questionnaires and/or surveys template, gained feedback from responders and
now have the raw data ready to structure and present. What next?
Depending on whether you have conducted a deep dive or have decided to gain
an initial first impression by running a quick survey, if your organisation has not
really focussed on M&E for communications, or started it at some point and not
really kept up with the process, remember that you will want to set a baseline for
further research and exploration.
If you are continuing M&E efforts and already have a baseline for the purpose of
data comparison, then the task ahead is simply a matter of comparing historical
data with new data. If you are starting from scratch, its important that you build
a solid foundation.
Charts
Perhaps the simplest way to present quantitative data is by using charts and
graphs to show comparisons. A majority of survey software will actually
generate this for you depending on what you input. If you have integrated your
research techniques and have a combination of qualitative and quantitative
methods, you may want to look at taking a thematic approach to analysing your
qualitative data.
If you have asked people to respond with an opinion rather than scale, you can
look through the answers you have received and search for denominators.
People may have expressed a concern about knowing or understanding the
mission of the business or have requested new content in your employee
newsletter. The responses may be similar, but people will have expressed the
same opinion in different ways using different language.

COMMUNICATION STRATEGY/PLANNING - Monitoring and Evaluation - Employee and Organisational Development

COMMUNICATION STRATEGY/PLANNING - Monitoring and Evaluation - Employee and Organisational Development

You can then highlight all data which mentions
perceptions of the company and create a sheet
highlighting opinions. Again there is software
that can do this for you. You can also use qualitative opinion based data to support your
communications strategy and planning by pulling
direct quotes from the data if there is something
in particular you want to highlight. Remember
that you should always look for themes and be
cautious about bias. If one person in a group of
20 responses has expressed an interest in developing a new channel, how relevant
is this? If 15 people have expressed the same interest, then it may be time to actually consider highlighting this point and making some changes.

If you can make your presentation engaging and specific, you will find that the
data you have collected and subsequent analysis comes to life. If you can then
connect this with your plans for communications and highlight why the data will
inform changes you intend to make based on the targets and responses, you will
find that a picture begins to take shape.

Embedding Data into Strategies

M&E should inform your communications planning and depending on your

findings, you may want to include this in your strategy document,
highlighting key areas that will support communications development. The
point is that a strong rationale for making changes comes from evidence and
you can make it clear that your decisions are based on something more than
your own opinion.

Compilation Presenting your findings.

If you have planned your research well, you should find that there is a seamless
integration between your qualitative and quantitative data. How you choose to
present this depends on what you want to show, but remember to make
connections that support your strategy. Make sure that you create clear headings,
state the purpose of you M&E plan and highlight the standards, targets and
objectives you are aiming to achieve. Remember to set a specific time frame for
further comparisons to evidence progress.
You may want to give a presentation showing your findings and highlighting the
most important or relevant section of the analysis. Statistics can be interesting
and it all depends on how you present them. If you are reading from a sheet and
simply regurgitating dry statistics with no context, whats the point?

Jo Samuel is a Strategic Marketing and Communications Consultant with 10 years experience

advising, supporting and leading Marketing and Communications activities for different
organisations. For further information and/or advice contact;
josamuel@mail.com
Tel +44 7921 097727/ +81 503 1360 396
Associated Articles (click link to view)
Communications Strategy - From Conceptualisation to Implementation