Are university rankings reliable?
The
parameters used do not necessarily reflect quality. Large institutions benefit
from what is essentially a marketing exercise
University
rankings are a standard feature in most countries. The rankings resemble a
football league table and is always read like one. Rank seems to be the only
thing that counts, with relative position in the table attracting more
attention than the processes by which the rankings are achieved.
The
declared goal of all ranking agencies is to assess quality using a set of
indicators which could overlap with each other. Most ranking systems are a
three-part process: first, data is collected on indicators; second, the data
for each indicator is scored; and, third, the scores from each indicator are
weighted and aggregated. Rankings, therefore, are an aggregation of indicators,
leading to the total score. This weight-and-sum approach meets the common sense
test well and thereby makes university ranking highly marketable.
Chief criteria
Rankers
typically choose indicators relating to learning inputs; research; outcomes;
reputation. Each indicator is seen as a reasonable proxy for quality and ,
suitably aggregated and weighted, constitute a plausible, holistic “definition”
of quality. By selecting a particular set of indicators and assigning each a
given weight, the authors of these rankings often impose a specific definition
of quality on the institutions being ranked. Intriguingly, there is hardly any
agreement among the authors of these indicators as to what indicates quality,
even while the choice of indicators and the weight given to each indicator make
a considerable difference to the final output.
Additionally,
rankings are based on convenient data. The result is often that ‘teaching
quality’— a particularly relevant indicator — gets excluded because obtaining
independent, objective measures of teaching quality is difficult, expensive and
time-consuming. Therefore, ‘measured institutional quality’ is not immutable,
as an institution’s ranking is largely a function of what the ranking body
chooses to measure. No wonder rankings have been met with a mixture of public
enthusiasm and institutional unease. Very few league tables do a good job of
normalising their figures for institutional size or of using a “value-added”
approach to measuring institutions. As a result, they tend to be biased towards
larger institutions and institutions with good “inputs”.
Big impact
Underlying
the weight-and-sum methodology is the first assumption that all the indicators
are mutually supporting and that they all contribute, though not necessarily in
equal proportion, to the measurement of academic excellence. In other words,
the relationships between the indicators are assumed to be additive. Related to
this is the second assumption that the indicators compensate one another such
that a weakness in one indicator is made good by strength in another; for
instance, having more international students can compensate a poor showing for
citation. Third, summing of the raw scores from distributions with different
standard deviations does not in any way affect the raw total scores and distort
the overall.
Despite
these shortcomings, rankings have had an impact far beyond their arbitrary
design would warrant. This is because in the highly competitive culture of
today, people are trying hard to out-do one another in almost anything, and
universities are not spared of this questionable approach. Increasing
marketisation of higher education, coupled with greater mobility of students,
led to the creation of this psyche where perceived status and reputation are
seen as important marketing tools.
These
concerns should not be dismissed lightly, because rank consumers have no way of
knowing that what they get is often not what they have been promised.
University ranking has to be raised to a level of rigorous scientific research.
They must (i) clearly spell out what constitutes quality; (ii) empirically
identify minimally overlapping indicators to measure quality; (iii) give
weights in proportion to the relative importance of the indicator; and (iv)
figure out ways to actualise the given weights, without prejudice or bias.
The
writer was dean and director-in-charge, IIM- Lucknow, and director, Jaipuria
Institute of Management
Source | Business Line | 26 July
2017
Regards
Pralhad
Jadhav
Senior Manager @
Knowledge Repository
Khaitan
& Co
Upcoming Lecture | ACTREC - BOSLA Annual lecture series (125th birth anniversary of father of library
science, Padmashree Dr. S. R. Ranganathan) on Saturday, 12th August 2017 at Advanced Centre for Treatment,
Research and Education in Cancer (ACTREC), Kharghar, Navi Mumbai. (Theme | 'MakerSpace')
No comments:
Post a Comment