SIMILARITIES, DIFFERENCES AND EMERGING TRENDS IN HIGHER

EDUCATION RANKING CRITERIA: USING INTERPRETIVE

RESEARCH FOR THE PROCESS ANALYSIS OF CRITICAL
EVALUATION
Prof. Dr. Mujahid Ali
Bahauddin Zakariay University, Multan
Asif Yaseen
Bahauddin Zakariay University, Multan
Saifullah Qureshi
Bahauddin Zakariay University, Multan

ABSTRACT
This research ascertains the increased competitiveness in higher education by assessing
academic excellence that is exercised by investigating theoretical issues in methodological
subjects fundamentals to university ranking systems prevalent in higher education systems.
During recent times, different stakeholders like students, parents, industry, university
management and faculty are showing greater interest to know the process and criteria of Ranking
System. Thus, it is necessary to make a critical examination of best known practices prevalent in
the world.

PURPOSE
The purpose of this paper is to address and assess the specific context of different
international ranking systems by examining the different dimensions and their empirical outlook.
A conceptual model indicating the multi-dimensional index is proposed after finding
commonalities and highlighting variations in different criteria systems that supports the
facilitation of the paradigm shift for evaluating the competitiveness of higher education
institutions.

DESIGN/METHODOLOGY/APPROACH
Different ranking methods prevalent for ranking the universities of the world are
compared and interpretive research based on a single question is used to propose a system that
reflects the extended choice of new requirements for current and future university education
needs.

FINDINGS
This research will aim at identifying emerging trends in measuring the quality in higher
education.
Keywords: University ranking, ranking methodology, criteria, higher education

INTRODUCTION
Ranking System is a standardized way to establish comparability and studying
competitiveness among universities of the world. Systematic quality management in relation to
university ranking is a hot issue due to vested interest of different stakeholders so we are
observing that new means of ranking excellence in university education are coming into market.
These stakeholders are students, parents, employers, funding bodies, communities or
administrators (Harvey and Green, 1993). This is also important to consider that no one ranking
system is perfect due to inherent weaknesses in the use of data and data processing methodology.
This is due to examination of ranking system from two perspectives simultaneously, social
characteristics and multi-dimensional decision making matrix. Social characteristics relate to the
process that education is a service whose perceived quality cannot be measured objectively and
secondly the product of all ranking systems are marketing tools for all stakeholders so biasness
and fallacies are lying inherently. As per Macquarie Dictionary 3e, university is an institution of
higher learning, conducting teaching & research at the undergraduate and postgraduate level.
This definition focuses on Academic Excellence in terms of teaching and research. Quality of
lectures, faculty members and student support system (i.e. student engagement with learning,
social/emotional support system, resource of library & I.T) are perceived Academic Excellence

(Hill, Lomas, McGregor, 2003).

Similarly, some researchers measure quality of higher

education based on many quality dimension such as Reliability, Responsiveness, Access,

Competence, Courtesy, Communication, Credibility, Security, Tangibles, Performance,
Completeness etc (Owlia, Aspinwall, 1996). In the same way, focus on Quality of Research
while considering Academic Excellence, such number of Research papers published, research
papers cited by other researcher must be given due importance.

LITERATURE REVIEW
Though the process of ranking excellence in higher education is neither objective nor conclusive
one due to socio-political context and double Loop learning frames yet significant to making a
choice for higher education. HEIs are required to create education that should have relevance to
society. The scientists, scholars, researchers, faculty members and students are required to
promote the welfare of society and human resource development. During current times, HEIs
are becoming Double Loop Learning centers, providing education in social and political context.
These are considered the house of interactivity, learning mutuality and feedback. However, the
objective criterion is based on quantitative factors, such as number of research paper published,
citation per faculty member, number of faculty member having PhDs while subjective criteria
includes factors such as quality of education, quality of research, peer review, recruiter review
etc.

Some of the previous researches have shown that the objective measures of ranking are better
and accurate as compared to subjective measures (Taylor & Braddock, 2007). They argued that
in subjectivity there are chances of personal biasness as Peer Review or Recruiter Review. So, it
is better to allocate scores on the basis of objectivity. In the same way, it is difficult of allocate
scores to the excellence of education to a school. So, due to measuring and biasness problem,
objective measures are preferred. Ranking system must include both objective and subjective
basis keeping in view the following quality dimensions in higher education systems,

Reliability: State and extent of modernization in education.

Consistency: Standardization of teaching methodologies.

Responsiveness: Willingness to understand the needs of students and provide them the response.
Access: Availability of staff for guidance and advice physically and virtually.
Competence: Knowledge, skills and abilities of staff
Credibility: The degree of trustworthiness of the institution
Security: Confidentiality of information
Tangibles: State, sufficiency and availability of equipment and facilities
Performance: Primary knowledge/skills required for students
Completeness: Supplementary knowledge and skills, use of computer
Flexibility: The degree to which knowledge/skills learned are applicable to other fields
There is direct relationship between ranking process and Academic Excellence. It works as a self
improvement tool by showing their performance. It can increase the competitions among the
HEIs that bring excellence in quality of education and research because ranking systems are
more concerned with excellence (Taylor & Braddock, 2007). Schools rankings have influence
on the decision- making process of prospective students (Van Roon, 2003). More recruiters also
use these ranking while seeking new employees for their organizations (Wuorio, 2001).

The history of university rankings can be traced to more than thirty years (Schatz, 1993). Since
then, a lot of HEI ranking agencies came forward who ranked the HEIs as per their own criteria.
Some of them are Go vt. agencies (HEC Pakistan), some are magazines and newspaper (Times
Higher Education Supplement, Financial Times/Business Week), universities (Shanghai Jiao
Tong University) or some NGO (OIC). These different organizations use their own criteria to
rank the HEIs based on different factors such as quality of education, research output,
infrastructure and human resource etc. So, while ranking universities, it is necessary to give
more weight to excellence in education and the research as compared to other factors. Although
those factors also contribute to the university excellence but the above mentioned two points
should be considered critically. Some of the previous researches have shown that the objective
measures of ranking are better and accurate as compared to subjective measures (Taylor &
Braddock, 2007). They argued that in subjectivity there are chances of personal biasness as Peer
Review or Recruiter Review. So, it is better to allocate scores on the basis of objectivity. In the
same way, it is difficult of allocate scores to the excellence of education to a school. So, due to
measuring and biasness problem, objective measures are preferred.

Some of the ranking systems rank the universities globally such THES, SJTU. While some
systems, rank universities regionally or nationally such as OIC and HEC Pakistan. A brief
introduction of renowned prevailing University Ranking System is given as under:

THE TIMES HIGHER EDUCATION SUPPLEMENTS SYSTEM

Times Higher Education Supplement System (formerly known as THE-QS World University
Ranking) is a British magazine that annually publishes the World University Rankings. THE
publishes the ranking I collaboration with Quacquarelli Symonds, a consulting agency group.
But now, THE has an agreement with Thomson Reuters who will provide all the data for annual
World University Rankings from 2010 and beyond. They are also considering introducing a new
ranking methodology over coming month.
Presently, THE is considering the following criteria to rank World Universities.

Peer Review Score (40)

Employer Review (10)

International Faculty Score (5)

International Students Score (5)

Student to Faculty Ratio Score (20)

Citations per Faculty Member Score (20)

THE SHANGHAI JAIO TONG SYSTEM

SJT System is published by the Shanghai Jiao Tong University, Shanghai, China.
Shanghai Jiao Tong University first publishes the Academic Ranking of World University in
2003. SJT also ranks universities based on broad field of research. Presently, SJT is considering
the following criteria to rank World Universities.

o Alumni of an institution winning Nobel Prizes and Fields Medals (10)

o Staff of an institution winning Nobel Prizes and Fields Medals

(20)

o Highly cited researchers in 21 broad subject categories

o Papers published in Nature and Science*

(20)

(20)

o Papers indexed in Science Citation Index-expanded and Social Science Citation Index
(20)
o Per capita academic performance of an institution

(10)

U.S NEWSWEEK INTERNATIONAL

U.S Newsweek International is an American magazine that conducts ranking of
universities based on following criteria:
1.

Criteria from SJTU: (50)

Number of highly cited researchers in various academic fields

Number of articles in Nature and Science Journal

Number of articles listed in Social Sciences Citation Index and Arts & Humanities

Citation Index

2.

Criteria from THES: (40)

Percentage of international faculty

Percentage of international students

Citation per faculty members

Faculty to student ratio

3.

Criteria of Newsweek Library: (10)

Library holdings (number of volumes)

ORGANIZATION OF ISLAMIC CONFERENCE (OIC) RANKING

OIC was developed in 1970s for development and protection of Islamic Countries. Recognizing
the importance to education in OIC countries, OIC prepared some plan to promote higher
education in Islamic countries. However, it was necessary to have a system to measure the
performance of universities in Islamic countries. In 2006, a meeting was held in Kuwait to

discuss the procedure and mechanisms of ranking of universities. In February 2007, a Technical
Experts Meeting was held to propose criteria, procedure and mechanisms for university ranking
in OIC countries.

The drafted document containing proposed criteria, procedure and

mechanisms was distributed to all OIC member states in the seminar of OIC in April 2007 which
was finally amended and adopted. The major categories of criteria are given below:
o

Research quality and output

(50 )

Quality of Education Teaching

(35 )

International outlook

(7 )

Facilities

(3 )

Socio-economic impact

(5 )

HIGHER EDUCATION COMMISSION (HEC) OF PAKISTAN

Moving from global ranking bodies towards regional and then national bodies, our next ranking body are
HEC Pakistan, which is sole ranking authority of HEIs in Pakistan. HEC first published the ranking of
HEIs in 2005. Ranking criteria used by HEC consists of five key categories. These categories are given
below:

Students (17)

Facilities (15)

Finances (15)

Faculty (27)

Research (26)

In addition, there are also some other ranking agencies that perform the ranking of universities. These are
Financial Times (For ranking business schools), Business Week, and Global University Ranking by
Wuhan University, China etc.

RESEARCH METHODOLOGY
We have evaluated the following ranking systems for making comparison based on the
secondary data of these ranking organizations. We will evaluate the following university ranking
organizations:

Times Higher Education Survey

Shanghai Jiao Tong World University Ranking

HEC Pakistan University Ranking Criteria

OIC Ranking System

Literature review does not indicate strong evidences about empirical research regarding
assessment of quality of education in relation to ranking systems. Existing literature focuses on
ranking systems as marketing instruments for different stakeholders. Thus the existing work does
not empirically support our investigation. So our research design is an inductive approach
primarily focusing on qualitative domain by the integration of qualitative and quantitative
strategies. We have anticipated that the use of inductive approach will reveal the diversity within
categories and across the categories also. We have asked only one question from different
stakeholder groups that What is the perceived meaning of university ranking system to
you. The respondents themselves made a number of responses related to quality of learning
environment, social support systems.

THEORETICAL FRAMEWORK
Let us examine how the constructs of different international ranking systems measures
up.

THE TIMES HIGHER EDUCATION SUPPLEMENT SYSTEM

The most recent THES ranking (2009) assesses universities according to the following
criteria:

o Peer Review Score (40)

o Employer Review (10)
o International Faculty Score (5)
o International Students Score (5)
o Student to Faculty Ratio Score (20)
o Citations per Faculty Member Score (20)

THE CRITICAL EVALUATION

Academic peer review is based on online survey for which questionnaire are distributed
to academics worldwide. Respondents are asked to list up to thirty universities that they consider
best in their area. Score are allocated according to how many times; university names are
mentioned in these lists provided by the peer academics. Critically examination of criteria 1
shows that THE system is allocating higher weightage 40 scores based on peer review
assessment. The 40 weighting is very high for the peer cross-section of university researchers
that is totally based on subjective grounds.

More there are also chances of influences of

reputation while review. As Simon Marginson (2006, p.90) says, there is the halo effect:
Moving to criteria 2, it is aga in based on subjectivity. It is also based on a global survey of
employers who are recruiting the graduates. These employers may tend to prefer the graduates
of prestigious universities but not on the personal competences of the students.

There are

examples in which some executives said that the graduates of prestigious institutes will perform
better as compared to other universities.
Criteria 3 and 4 both are concerned with international orientation and contribute 10
scores at all. International faculty may have impact on the teaching & education excellence as
foreign qualified faculty member have diversified knowledge from different area of the world.
Furthermore, it is quite easy to measure it. But criteria 4, does not contribute towards educatio n
excellence & research work.
Moving towards criteria 5, student: faculty ratio contributes 20 scores at all. A lower
student: faculty ratio is a good indication of quality education. Small classes ensure better class
participation, effective control and feedback, improved communication and cooperation between
students and teachers and among students themselves. It is also an objective measure and easy to
calculate. However, assigning 20 weight to just student: faculty is not justified.
Finally, criteria 6 states that 20 of a universitys score is allocated based on citation.
Citation means a reference to one academic publication in the text of another. It can be
measured on per capita basis also. The more of a university publication are used as reference in
the publications of other universities, more scores that university gets. This is one of the core
functions of a university. This criterion truly reflects a universitys strength and performance.
Further, it is an objective measure that is easy to calculate. The only conspicuous thing is that

criterion 6 contributes only 20 of a universitys score. So, it is necessary to assign more weight
to citation and research output.

THE SHANGHAI JAIO TONG SYSTEM

CRITICAL EVALUATION
SJT is considered a better ranking system as compared to other ranking system and its acceptance
and reliability is increasing day by day. Academics generally prefer the ranking issued by SJT and are
satisfie d by their criteria and methodology. It is clear from the criteria that SJT is giving more weight age
to the quality of education and research output & research quality. So, due to these reasons their
acceptance is increasing because these two are the core functions of a higher education institute.
Secondly, as in THE system, scores are assigned based on subjectivity which have some criticism
due to personal biasness or incorrect data. But in SJT system, all the scores are assigned on objective
measure by calculating the research output or awards won by staff and student.
Evaluating criterion 1 and 2, both are relating and contributing towards the provision quality
education i.e. 10 & 20 respectively. So here, 30 scores are based on the quality of teaching and the
quality of faculty members.
Criterion 3 to 5 is concerned with research quality and research output. The aggregate scores
assigned to research are 60. This shows that SJT system of ranking is a research oriented system which
assigns more scores to the academics which are focusing on the quality & output of research.

OIC
CRITICAL EVALUATION

OIC is considered a good source of evaluation of academics in Islamic countries.

Critically examining the criteria used by OIC, it is clear the OIC is giving 85 weightage to the
core functions of universities i.e. quality of education & research which is 35 and 50
respectively. Besides allocating scores to research & quality of teaching OIC also focusing on
the international factors (international faculty members, international students, foreign PhDs staff
and international conferences) and gives 7 weights to it. It is a good component of overall scores
for university ranking.

Then, OIC assigns 3 scores to Facilities which an institute have such as availability and
access of students to library books and research journals. The last criteria add 5 scores to
academics based on socio-economic factors as how much is working to generate the incomes and
society development. This is a good criteria of academic excellence that the must use the
competencies of their students in generating revenues for the institute and giving them practical
exposure. Thus, OIC is a considered a good body of ranking higher education institutions.

HEC PAKISTAN
CRITICAL EVALUATION
HEC is also using a very comprehensive criterion for ranking the universities of Pakistan.
This system comprises of five broad categories students, Faculty, Finances, Facilities and
Research. Although HEC follows a comprehensive criteria for ranking Pakistani universities,
but still it contains certain criticisms. These are discussed under:
The first criticism on the HEC ranking methodology is regarding scores allocations.
Scores allocated to research and quality of education is quite low. So, there is need to increase
these points in order to induce university teachers and researchers to add maximum value in total
scores through research and quality of education.
Second, scores allocated to research are based on quantitative factor means they are
measuring just research output, not the quality of researches or quality of education. There is a
need to allocate score based on quantity but also on quality as allocated in SJT system.
Further, scores allocated based on category Student also includes sub-categories that
account for only quantity as number of students passed out, number of students have more than
60 marks, number of PhD produced etc. These all factors are measuring just quantity and have
no link with how much quality and competence they possess.

Table -1: Cross- Comparison Outlook

Major Category
Finances

Infrastructure

International Orientation

Others

Quality of Education

Research

Criteria
Funds Allocated
Number of books in library, Computers, internet,
Journal, equipment, universitys research
Institutes/Centers
International Student Score
Foreign PhDs Faculty Members
International Conferences Organized
International Exchange Programs
Peer Review
Employer Review
Size of Institution
International Faculty Score
Student to Faculty Ratio Score
International Aspect(Other)
Alumni winning Noble prices & Field Medal
Staff/Faculty of institution winning Awards/Noble
prizes & Field Medals
Highly cited researchers
Faculty having M.Phil, Training by Faculty, Funds
obtained through competitive Grants
Alumni Highly cited researcher
Faculty Members with Ph.D
Ratio of Ph.D Faculty to Total Faculty
Alumni winning Awards
Ratio of post graduate students to total number of
students
Rate of growth of post graduate students
Students wining international Olympiads
Research Quality
Research Performance
Rate to Growth for research quality
Research Volume

THE

SJT

HEC
15

15
5

OIC

3
1
1.5
1.5
1

40
10
10
5
20

2
3

10
20
20

6
8

8
3
9

3
4
3
2
3
3
17
16
5
5

Rate of growth for research performance

Patents

5
2
26

Citations per Faculty Member Score

Articles published in Nature & Science
Articles in Science Citation index-expanded, Social
Science Citation Index, and Arts & Humanities
Citation Index

Socio Economic Impact

Students
Total

20
20

20

Life Learning courses, entrepreneurship programs,

Contract & Consultancies incomes etc
Student Perspective

5
100

100

17
100

Table -2: Similarities Index

Major Category

Criteria
Research Quality
Research Performance
Rate to Growth for research
quality
Research Volume
Rate of growth for research
performance
Patents

THES

SJT

HEC

OIC
17
16
5
5
5
2

26

Research
Total

Citations per Faculty

Member Score
Articles published in Nature
& Science
Articles in Science Citation
index-expanded, Social
Science Citation Index, and
Arts & Humanities Citation
Index

20
20

20

20
40

26

50

100

Major Category

Quality of Education

Total

Criteria
THES SJT HEC OIC
International Faculty Score
5
2
Student to Faculty Ratio
Score
20
5
3
International Aspect(Other)
Alumni winning Noble prices
& Field Medal
10
Staff/Faculty of institution
winning Awards/Noble
prizes & Field Medals
20
2
6
Highly cited researchers
20
8
Faculty having M.Phil,
Training by Faculty, Funds
obtained through competitive
Grants
8
Alumni Highly cited
researcher
3
Faculty Members with Ph.D
3
4
Ratio of Ph.D Faculty to
Total Faculty
9
Alumni winning Awards
3
Ratio of post graduate
students to total number of
students
2
Rate of growth of post
graduate students
3
Students wining international
Olympiads
3
25
50
27
37

CONCLUSION: IDEAL RANKING SYSTEM

The above discussion and critical evaluation reveals that there are much similarities
present in each ranking mythology. However, each system has its own pros & cons. The need
here is to merge the most appropriate factor into one.
First, while allocating scores on research, quality of research should not be ignored. It
should focus also on quality of publication as well as the discipline must be considered. Because
it shows the true worth of the research work conducted by the faculty of the respective institute.
This clause is found in SJT system, in which they are measuring the quality of research as well
as its discipline. So, it is a good indicator of measuring and allocating scores to research.
Secondly, ranking bodies should allocate scores to higher education institutes on the basis
of funds generations also. Such as OIC gives score on the basis of contract & consultancies as
well as several entrepreneurship programs.

This clause is very important in order to give

students a practical exposure of industry.

Then, it is also very good to make universities

independent in arranging their resources.

An important factor in OIC ranking is allocation of scores on growth rate in research
performance and research quality. This clause is important for the university to boost their
research quality and performance and to set the targets, goals and clear direction in research
work. So, it is necessary for the institutions to work on one of the principles of Total Quality
Management that is Continuous Improvement.
International orientation is also an important measurement factor of teaching excellence.
As OIC contributes 5 scores on the basis of international orientation. Internationalization helps
in sharing cross-culture knowledge and experiences and gives students a better opportunity to
excel in their competencies.
Another important contribution that have significant impact on academic excellence is
Employer Review. THE system allocates 10 % of total scores on the employer review who are
appointing the graduates of a particular institute.

Employer review about the academic

excellence is measured by examining the performance of the graduates in that organization. It

reflects true professional comp etencies as well as the knowledge of the students. Although it is
certain criticism on the subjectivity and biasness however if measured correctly, it will have a
positive impact over the ranking of institute and academic excellence.

REFERENCES
Susan M. Keaveney, Customer Switching Behavior in Service Industries: An
Exploratory Study, Journal of Marketing, Vol. 59, No. 2. (Apr., 1995), pp. 71-82.
Ranking of Universities-Ranking Methodology
http://www.hec.gov.pk/QualityAssurance/Ranking_Methodology.htm
http://www.cepes.ro/publications/pdf/wcu_ranking2007.pdf
http://www.sesrtcic.org/files/article/232.pdf
http://www.sciencedev.net/docs/oic%20universities.pdf
Ranking

Methodology

Indicators

and

Weights

for

ARWUhttp://www.arwu.org/ARWUMethodology2010.jsp
http://www.usnews.com/articles/education/best-colleges/2009/08/19/methodologyundergraduate-ranking-criteria-and-weights.html
http://www.univforum.com/canadian- university-ranking.html
http://en.wikipedia.org/wiki/QS_World_University_Rankings
Lagrosen, Roxana & Leitner, Examination of the dimensions of quality in higher education:
Quality Assurance in Education, Volume 12 Number 2 2004 pp 61-69
S. Owlia & Aspinwall, A framework for the dimensions of quality in higher education: Quality
Assurance in Education, Volume 4 Number 2 1996 1220
Taylor & Braddock,

International

University

Ranking

Systems

and

the

Idea

of

UniversityExcellence: Journal of Higher Education Policy and Management Vol. 29, No. 3,
November 2007, pp. 245260
Melbourne Index, see Williams and Van Dyke (2004).
Engr

Muhammad

Ismail,

Ranking

of

universities:

http://qa.nust.edu.pk/downloads/Ranking_of_Universities_M_Ismail.pdf

Hill, Lomas & MacGregor, Students perception of quality in higher education,

Quality

Assurance in Education, Volume II, No. 1, 2003, pp 15-20.

Bickerstaffe & Ridgers, Ranking Business School, Journal of Management Development, Vol.
26, No. 1, 2007, pp. 61-66

K. Peter: Business school rankings: content and context, Journal of Management Development,
Vol. 26, No. 1, 2007, pp. 49-53
Anninos & Loukas N, University Performance Evaluation Approaches: The Case of
Ranking Systems,