JAMES SANDERS 20009551 COSH09 1

How can/should you assess the quality of research?

Within this essay I will explore a range of research methods, in particular action research,

in order to understand what determines the quality of a piece of research. I will investigate

both qualitative and quantitative research methods as well as considering the importance

of validity and reliability when assessing research. I will also closely examine the action

research and apply it to my own practice in order to assess the quality of the research I

use on a daily basis. I will give consideration to the different stages of research in order to

ascertain the different motivations, choices and reasons for doing a piece of research

which inevitably affect its perceived quality.

In order to assess the quality of a piece of research consideration must be given to its

purpose, what it is going to used for? There are many reasons why research is carried out,

in some cases it is used to gain quantitative data. For example, I conducted a piece of

research on the area in which I work called a needs assessment. The assessment

concerned itself with the collection of data to provide information on the ethnic and socio

economic background of young people in the area. This data was then collated and

compared against the registration forms that young people who attend the youth club fill

out. This enabled us to see if a representative percentage of young people within the

community attend the youth club. This piece of research was very useful as it clearly

identified which groups of young people were accessing the youth service. Unfortunately it

did not capture why certain groups of young people were not represented within my youth

club. So, while using this quantitative method gave me clear, easily readable data on a

large number of young people it gave no meaning. It did not give me a reason why young
James Sanders 20009551 2
COSH09 Sue Wayman

people with disabilities or those from ethnic minority groups were not attending. Of course

I could make informed guesses but I was imposing my own meanings on the situation

rather than gaining information from the young people themselves. It could be said that

with the information gathered I managed to change nothing I just had a series of

percentages. Clough and Nutbrown (2002:12) said that “research which changes nothing-

not even the researcher-is not research at all.” With this in mind, although this piece of

research seemed to generate only numbers it also raised my awareness of the

unrepresented groups within my youth club and motivated me to think about ways I can

encourage them to access the youth service. It may also prompt a more qualitative form of

research such as an open-ended questionnaire or interview in order to find out young

people’s reasons for not attending the club. Using a mixture of methods in this way can

create quality research that produces meaningful information.

At the beginning of any piece of research there should be an issue or reason to do it. The

starting point for practitioner research is to devise an answerable question that is relevant

to an issue within practice (Fox et al. 2007). Even at this conceptual stage the research

can already take on a characteristic where it can become seen as less useful or even

invalid. In the case of my needs assessment I feel the need to inform myself of the

demographic of the community in order to address any miss represented groups within my

youth club. At the same time I need to conduct further research to ascertain why certain

groups do not attend my youth club.

Research is seen to fall in to two main categories of quantitative and qualitative. These

two methods of completing research achieve very different results and concern
James Sanders 20009551 3
COSH09 Sue Wayman

themselves with obtaining different types of data. Quantitative methodology is mainly

focused on the collection of numerical data. It has been said that, quantitative research is

“...supported by the positivist or scientific paradigm, leads us to regard the world as made

up of observable, measurable facts” (Glesne & Peshkin, 1992: 6). Where as qualitative

involves a more naturalistic approach in the analysis of words, pictures or objects to seek

to understand phenomena in a context-specific setting (Golafshani 2003). Each of these

methodologies has its strengths and weaknesses and great debate has been generated to

argue for and against both. Within social sciences it is generally believed that the

meanings given by qualitative research is of more value. However, during my experience

as a youth worker I have found there is continual pressure to conduct quantitative

research in order to evidence my work. For example, the continual collection of data on

how many young people attend the youth club, how many are not in education,

employment or training etc.

Quantitative and qualitative researches are often regarded as positive and non-positive

paradigms. Gray (2004) says that positivism is focused on facts rather than values and

that the scientific approach can form reasoned conclusions with quite indirect evidence,

often based on theory. They draw conclusions from observations rather than participation

(Zuber-Skerritt 2009). Using qualitative methodology links with the values of youth and

community work in that it is widely acknowledged that youth work is qualitative in nature

so cannot be measured in terms of numerical value (France 2001, Jeffs & Smith 2005).

However, qualitative research often generates large amounts of data, which can be

difficult to sift through and to find commonalities to draw results from (Bryman 2004).
James Sanders 20009551 4
COSH09 Sue Wayman

The non-positivist approach believes that you cannot take on a positivist approach, as

researchers cannot carry out observations objectively; humans are not neutral or value

free no matter how hard you try to be (Zuber-Skerritt 2009). This links to Habermas (cited

in Szczelkun 1999) theory of communicative action which is a cyclical process that

involves the researcher as an initiator who is looking for knowledge to serve their needs

but also being a product of what is around them. Just by engaging in a piece of research

will show that you have a particular interest in that subject (Clough and Nutbrown 2002).

Bell (2005) says that when choosing a topic for action research you are looking for an

outcome that will have practical value. Therefore you will have already decided on

possible outcomes before you start; you are bringing into it your own thoughts, feelings

and judgements (Zuber-Skerritt 2009). By using participants in the research process rather

than subjects (as in the scientific approach), you can make the research objective (Zuber-

Skerritt 2009). Stringer (1996) said that when he embarked on one of his research studies

using participants rather than subjects he discovered, through using rigorous methods

within his study, the complexity of people’s individual and social realities that he could not

hope to understand as an objective researcher. Participatory action research is of benefit

as it is not just the researcher that can learn from the process it is all of those that

participate (Gaventa and Cornwall 2001). When I carry out my research as a practitioner

researcher I hope that the organisation, the young people and my practice will benefit from

my findings.

Bryman (2004) highlights that qualitative research can sometimes fail to establish exactly

how the researcher carried out the action; for example, how interviewees were selected,

as well as the process for analysing the data collected.

James Sanders 20009551 5
COSH09 Sue Wayman

As a practitioner within the youth and community field I have found that reflective practice

is integral to the way in which we deliver and go about our work. The process of looking at

our work and reflecting on our actions in order to improve our practice has become second

nature. McLeod 1999:8 calls this “research carried out by practitioners for the purpose of

advancing their own practice” and is often referred to as practitioner research. When Jeffs

and Smith (2005) wrote about reflective practice they related the process to the

‘experiential learning’ cycle by Kolb (cited in Jeffs and Smith 2005:65). To take this further

would be to undergo Action research and is also explained in terms of a cyclical process

that involves experience, reflection, research, planning and action (Denscombe 1998).

‘Action research is about working towards practical outcomes, and also about creating

new forms of understanding’ (Reason and Bradbury 2001:2). Zuber-Skerritt (2009) says

that ‘action research is located in this non-positivist, phenomenological paradigm’ (Zuber-

Skerrit 2009:107). Within the field of youth and community work action research is

considered to be the ideal form of research. This is a very involved process that requires

education through self-reflection or experiential learning (McNiff 1988). Action research

and reflective practice are similar in their nature in that they are both concerned with

improving practice. Bell (2005) says that the difference between action research and

reflective practice is that action research is undertaken with specific, relevant research

methods and rigor. The difference has also been described as: ‘Reflective practice can be

used to identify problems, action research can seek to provide solutions’ (McMahon

1999:168). In theory, action research can only be of benefit to the practitioners that

engage with the process (Reason and Bradbury 2001). However, the outcomes of the

research may indicate that major changes need to be undertaken within the practice that
James Sanders 20009551 6
COSH09 Sue Wayman

may pose both practical and ethical dilemmas (Bell 2005). Therefore action research for

practitioner researchers must have clear aims, objectives and structure before

commencing (Bell 2005). Denscombe (1998) says that there are ‘four defining

characteristics of action research’ which are: Practical, Change, Cyclical and Participation

(Denscombe 1998:57).

Bryman (2004) highlights the importance of linking research to theory; firstly by defining

what sort of theory the research is alluding to, and secondly, whether the research starts

from the basis of an existing theory, defined as deductive research, or whether a theory is

developed from carrying out the research, described as inductive research. There are

three main forms of theory; grand theories which operate at a more general and abstract

levels, middle range theories which aim to explain observed regularities, and the notion

that background literature can act as theory in that it spurs further enquiry (Bryman 2004).

The phenomenological approach to action research draws on the experiences of people

and how they perceive the world, rather than the researcher having any pre-conceived

ideas about their reality (Zuber-Skerritt 2009). Gray (2004) says that phenomenology is

about gaining understanding from the values and culture of the participant’s experiences.

This approach can be flawed by the prejudices of the researcher and how they can

potentially bias the data (Gray 2004). As Zuber-Skerritt (2009) said, we bring our own

meaning and values to our research so to analyse data completely value free is very

unlikely. Also, although as humans we can empathise with others, we can never truly

understand their thoughts and feelings (Stringer 1996). The phenomenological approach

sits within the interpretivism theoretical perspective according to Crotty (cited in Gray
James Sanders 20009551 7
COSH09 Sue Wayman

2004). This is a non-positivist approach that is concerned with the reality of the individual

(Gray 2004). Interpretivism is linked to the constructivism theory of epistemology (Gray

2004). ‘Epistemology provides a philosophical background for deciding what kinds of

knowledge legitimate and adequate’ (Gray 2004:16). Constructivism is about how people

construct their own meanings from their interactions with the world (Gray 2004). There are

two elements to the constructivism theory of epistemology, socially constructed and

individually constructed (Gray 2004). Socially constructed theories are concerned with

groups and cultures, where individually constructed theories are about the individual’s

unique experiences of their lives (Van Dijk 2003). Van Dijk (2003) says that the socially

constructed theories will inform individually constructed theories. As individuals we all

learn about life from the perspective our groups (families, colleagues, community) and the

culture we grow up in (Van Dijk 2003). These constructed understandings will help to

inform our individual realities (Van Dijk 2003). Van Dijk (2003) wrote about the power,

social abuse, control and dominance that certain groups have over the discourses in

society. This is also why phenomenological research is difficult to achieve in its true from.

Everyone has their own individually constructed understanding, informed by their social

constructs, of the research they are to undertake which will make it challenging to put all of

that knowledge aside and be completely non-biased (Stringer 1996). I believe that you can

carry out research that is informed by the phenomenological theory but you will not be

able to use it ‘to the letter’. As a practitioner researcher, it is important to know what

theories and philosophies are informing your research so that you design the research

questions accordingly and have an understanding of the underlying issues and limitations

(Fraser and Robinson 2004). Also it will help at the analysis stage of the research to

evaluate what approach you took and how you could have improved it or would do it
James Sanders 20009551 8
COSH09 Sue Wayman

differently again (Fraser and Robinson 2004).

Reliability and validity are often used as markers or measures to indicate whether a piece

of research has been completed to a standard that is acceptable. Reliability links with

quantitative methodology, in order to be reliable and produce significant results findings

must be more than a one off. They must be inherently repeatable (Shuttleworth 2008).

This means other researchers must be able to perform the same experiment under the

same conditions and generate the same results (shulleworth 2008). By repeating research

a number of times researchers reinforce their findings. This method is widely used within

scientific research and “this prerequisite is essential to hypothesis establishing itself as an

accepted scientific truth” Shuttleworth (2008:online). Reliability can be tested in two ways

‘test and retest’ and ‘internal consistency’. When using test and retest you should get the

same result/score in test one and test two. Internal consistency estimates the reliability by

grouping questions in a questionnaire that measures the same concept. Reliability’s

association with scientific research means that any research that uses human judgment is

always questioned (shulleworth 2008). For example if a piece of research needed

observers to collate information from an observed group, it could be said that the reliability

of the test could be compromised as each observers judgment can vary widely. Even if the

same individual is used they could rate observations differently throughout a day

according to the time or their mood (shulleworth 2008). This can vary even more if

measured over a length of time or days.

Validity is one way in which the quality of research can be assessed. As Joppe (2000)

explains “validity determines whether the research truly measures that which it was
James Sanders 20009551 9
COSH09 Sue Wayman

intended to measure or how truthful the research results are”. Validity can be seen as the

strength of our conclusions and propositions based on how we carried out the research.

Cook and Campbell (1979) define it as the "best available approximation to the truth or

falsity of a given inference, proposition or conclusion." The idea of validity within research

links to how we carried out the study, for example, the randomization of the sample group,

the choice of candidates or even our impact on the subject we are researching. Although

reliability and validity are often related to quantitative data collection, validity does have its

links to qualitative research. There has to be some way that the quality or reliability of

qualitative research can be judged. Golafshami (2006) discusses a number of researchers

who have triumphed judging qualitative research by it’s rigor, trustworthiness and quality.

When assessing the quality of a piece of research it is not just the accuracy and

usefulness of its results that should be considered. According to universalists such as

Dingwall (1980) and Bulmer (1982) when researchers encounter ethical dilemmas the

good of their research participants should always be put above the success of the study.

There are a number of ways in which we can judge the ethical quality of a piece of

research. We can ask whether the participants have given informed consent and been

made aware of exactly what participating in the survey will involve. It is important that any

participation is voluntary and that people are in no way coerced into taking part. It is the

researchers responsibility to ensure that taking part in research in no way puts the

participant at risk of physical or psychological harm. It is also good practice for participants

to be given anonymity with their identities and information being kept confidential.

However, there are some social phenomenon that may only be studied by breaking some

of these ethical rules. This has led Fletcher(1966:31) to argue for ‘situation ethics’ where
James Sanders 20009551 1
COSH09 Sue Wayman 0

the ethics of a piece of research is considered on a case-by-case basis. For example, in

some cases it may be necessary to deceive people when partaking in disguised

observations. Fletcher would argue that in some situations the end justifies the means

while in others there may be no choice other than to break some ethical rules. While

Fletcher argues for flexibility in ethical decision-making it is clear that any quality piece of

research should show that all ethical dilemmas have been carefully considered and

wherever possible the good of the participant has been of utmost importance.

The perceived quality of any piece of research is dependant on many factors. The reason

or motivation to undergo research is the first area that can impact on the research. Within

my practice I am continually asked to conduct research that puts a value or a price on the

work that we do. How many young people have certain issues or needs that we can

evidence so we can prove to funders (the government) the value of our work. In turn a

piece of research’s quality is objective to the person who is assessing the piece of work.

Whether the research is fit for purpose to start off with, for example choosing the right

methodology for the research and carrying it out in a manner that gives a truthful

representation of the subject, ensuring its validity and reliability. As a practitioner I feel that

using action research is a way that you can ensure your research is of good quality, as

you should be coming from the point of changing practice and producing knowledge

(Hughes 2001). With this in mind I believe the best way to assess if research is of good

quality would be on it’s utility within practice.

Word Count 3160

References
James Sanders 20009551 1
COSH09 Sue Wayman 1

Bell, J (2005) Doing your Research Project: A guide for first-time researchers in education,
health and social science 4th Ed. Berkshire. Open University Press.

Bulmer (1982) sited in Bryman, A. (2004) Social Research Methods: 2nd Edition. Oxford:
Oxford University Press.

Bryman, A. (2004) Social Research Methods: 2nd Edition. Oxford: Oxford University Press.

Clough, P and Nutbrown, C (2002) a student’s guide to Methodology 2nd Ed. London.
Sage Publications Ltd

Cook, T.D. and Campbell, D.T. (1979). Quasi-Experimentation: Design and Analysis for
Field Settings. Rand McNally, Chicago, Illinois.

Colasi, L, (1997) The Layman's Guide to Social Research Methods. Avaliable from
http://www.socialresearchmethods.net/tutorial/Colosi/lcolosi1.htm Accessed
[12/12/2010]

Denscombe, M (1998) The Good Research Guide for Small Scale Research Projects.
Birmingham. Open University Press.

Dingwall (1980) sited in Bryman, A. (2004) Social Research Methods: 2nd Edition. Oxford:
Oxford University Press.

Fletcher(1966:31) sited in Bryman, A. (2004) Social Research Methods: 2nd Edition.

Oxford: Oxford University Press.

Fox, M., Martin, P. & Green, G. (2007) Doing Practitioner Research. London: Sage
Publications

Fraser, S and Robinson, C (2004) Paradigms and Philosophy in Fraser et al. Doing
Research with Children and Young People. London. Sage Publications Ltd.

Gaventa, J and cornwall, A (2001) Power and Knowledge in Reason, P and Bradbury, H.
Handbook of Action Research: Participative Inquiry and Practice. London. Sage
Publications Ltd.

Glesne, C., & Peshkin, P. (1992). Becoming qualitative researches: An introduction. New
York, NY: Longman.Gray, D, E (2004) Doing Research in the Real World. London. Sage
Publications.

Golafshani, N. (2003). Understanding reliability and validity in qualitative research. The

Qualitative Report, 8(4), 597-606. ,http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf
Accessed [10/12/2010]
James Sanders 20009551 1
COSH09 Sue Wayman 2

Jeffs, T. & Smith, M. (2005) Informal Education: Conversation, Democracy and Learning
3rd Edition. Nottingham: Educational Heretics Press.

Joppe, M. (2000) sited in Golafshani, N. (2003). Understanding reliability and validity in

qualitative research. The Qualitative Report, 8(4), 597-
606. ,http://www.nova.edu/ssss/QR/QR8-4/golafshani.pdf Accessed [10/12/2010]

Kumar,R (2005) Reasearch methodology:a step by step guide for beginners. London:sage

McLeod J (1999) Practitioner Research in Counselling London: Sage

McMahon, T (1999) Is Reflective Practice Synonymous with Action Research?

Educational Action Research, Volume 7, No 1, P163 – 169.

McNiff, J (1988) Action Research: Principles and Practice. Hampshire. MacMillan

Education Ltd.

Reason, P and Bradbury, H (2001) Introduction: Inquiry and Participation in Search of a

World Worthy of Human Aspiration in Reason, P and Bradbury, H. Handbook of Action
Research: Participative Inquiry and Practice. London. Sage Publications Ltd.

Robson, C. (2002) Real World Research: 2nd Edition. Oxford: Blackwell Publishers.

Shuttleworth, M. (2008) Validity and reliability How do we know research is correct?

Experiment-resources.com.Available from http://www.experiment-resources.com/validity-
and-reliability.html Accessed 5/12/2010
Stringer, E, T (1996) Action Research; A Handbook for Practitioner. London. Sage
Publications.
Szczelkun, S (1999) Summary of the Theory of Communicative Action. Royal College of
Art London. Available from: http://www.csudh.edu/dearhabermas/publsbm01.htm.
Accessed 05/12/10
Van Dijk, T, A (2003) Critical Discourse Analysis. Discourse and Society. Available from:
http://www.discourses.org/OldArticles/Critical%20discourse%20analysis.pdf. Accessed
8/12/10.
Zuber Skerritt O (1996) New Directions in Action Research London: Taylor and Francis
Zuber-Skerritt, O (2009) Action Learning and Action Research: Songlines through
Interviews. Rotterdam. Sense Publishers.