This article first appeared on the Wonkhe website (8th April 2018) under the heading ‘TEF won’t sweeten my rankings rancour’.

Which is the best university? It’s a seductive question to ask, but that doesn’t mean there’s a sensible answer. League tables, aka rankings, is the nonsensical answer you’re likely to get.

They weigh the wrong factors – a very narrow idea of best, based on counting what’s measured rather than measuring what counts. Traditionally, this has led to a dominance of rankings by research-led institutions.

But even if the factors weighed were the right ones, the rankings use poor proxies to measure them – as if research citations, for example, were an unambiguous marker of quality, rather than being hugely dependent on publication in English, in the right journals and in the right disciplines.

But even if they were the right proxies, the data is often of poor quality: out of date, non-comparative, self-reported.

But even if the data were good, what rankers do with it isn’t: aggregating and weighting arbitrarily.

But even if the methodology were sound, the way the results are presented suggests an equal distance between say, first and thirty-first place as between fortieth and seventieth. Anyone who has ever seen a bell curve knows that is misrepresentation.

But even if league tables didn’t make all these mistakes and more, their worst crime is to imagine that there is such a thing as a single best university, rather than many different ways in which universities can be good at different things. Indeed, it is the very diversity of the higher education sector that is its strength. It means the sector as a whole can paint a rainbow of objectives catering to the divergent needs of particular students, communities, employers, economies and societies.

No platform for rankings

You can’t ban league tables, sadly. If we want information about higher education to be transparent, then there are those who will put it in a pop chart. That will attract attention, because offering an answer to that “best university” question is sexy.

The answer might not be to have fewer league tables, but instead to have more: an infinity of rankings so that each person can pick the one that combines just the factors they want, weighted perfectly to their needs. No ranking would be authoritative, because the array would reflect the personal and diverse nature of the question.

THE’s latest rankings product (its Global Impact Ranking) is a step in the direction of infinity in that it adds another league table to the shop window, incrementally diminishing the value of the ever-increasing heap.

However, perhaps we should welcome the desire to rate universities according to criteria such as recycling, fair labour practice and admissions policies, even if the process is as flawed as all the others? After all, the sexiness of rankings does shine a light on issues that might get overlooked (especially when the desire to do well in other rankings distracts universities from considering what else matters).

TEF: just another ranking?

That was explicitly the government’s intention when it introduced its own form of ranking – the Teaching Excellence Framework (TEF), which the then-minister Jo Johnson said would “introduce new incentives for universities to focus on teaching”. The idea was to rank universities’ teaching quality to get them to improve it and to drive student choice based on quality.

The problem is that TEF repeats the mistakes of other rankings. It weighs the wrong factors: the metrics (as was later acknowledged with the change of the name to include student outcomes) have little to do with teaching. It uses poor proxies, such as measuring employment not employability. The data is poor: the NSS component was downgraded after an NUS boycott undermined it. The methodology is arbitrary: for example, benchmarking by disciplines, but not regions.

The list goes on, but TEF is unlike other rankings in at least three respects. First, being the government’s own ranking, TEF bears more responsibility than most. It purports to be a truer truth – an authority that it hasn’t earned.

Second, most league tables – even though they are rarely entirely open about their methodology – do tend to stick to it. TEF, however, recognises the failings of its metric methodology and adds a subjective element: the review panel.  It may be the best part of TEF, but it’s the least transparent and most susceptible to inconsistency.

Third, most league tables’ misrepresentation is a single hierarchical list. TEF retains the hierarchy, but shrinks distinctions to three categories: good (bronze), better (silver) and best (gold). This, of course, creates a cliff edge where a fine judgement between silver and bronze, say, translates into a presentational gulf.

Informing student choice

Interestingly, there is no “mediocre” or “bad” in this hierarchy, but that’s not how students see it. Bronze is no one’s idea of an endorsement. This highlights an absolutely critical issue about rankings – TEF included – which would be the case even if they were more rigorous in their approach: how do they inform student choice?

Human choices are rarely rational. They emerge from a soup of feelings and preconceptions, sprinkled with croutons of information fried in confirmation bias. When it comes to a complex decisions, such as which university to choose, we don’t devise a personal list of criteria, sourcing objective data on each, and then coolly and fairly appraising the options relatively. Instead we latch on to something that provides a basis for beliefs we already hold.

In other words, we use heuristics: rules of thumb that often bear little resemblance to nuanced realities, but which hurt our brains less. This is precisely the quality about league tables that makes them so sexy. They say, don’t you worry your head about the real differences between two institutions that are both good in their own way, we’ve made the whole process simpler. Misleading, but simpler.

The same is true of TEF. Rather than providing information that disrupts misplaced beliefs and encouraging students to examine what kind of educational experience will support their own learning, TEF short-circuits the thinking and provides a yes/no/maybe checklist.

The Government was right to shine a light on teaching (well, on learning), but not the seedy neon beam of TEF. There are other approaches and, as Dame Shirley Pearce proceeds with her review of TEF, I hope she will think boldly about options that promote diversity and innovation rather than aping league tables that suppose there is a single model of “good” and which play blind darts to see who gets closest.

The post Rancour, rankings and the rankness of TEF appeared first on Johnny Rich.

Over several months in 2012, I was fortunate enough to be invited to participate in The 1994 Group’s annual Policy Forum discussing some of the most pressing issues in higher education. At the end, they invited me to write a blog outlining the conclusions I had reached from the discussions. This was orginally published on the 1994 Group website.

The right tools for the job

Last year, Stefan Collini, the great Cambridge academic, published a much-publicised book titled What are universities for? On the first page – the first paragraph even – he abdicated from answering the question. Without setting out to do so, the 1994 Group Policy Forum, of which I was proud to play a part, found itself needing to rise more adequately to the challenge.

    Do universities exist for the intellectual enrichment of our nation, of our culture, rising through teaching into research, always unencumbered by quotidian concerns?

    Do they exist as labour market factories, churning out graduates pumped full of transferrable skills and marked ‘approved’ by qualifications?

    Are they partners with industry, the powerhouses of innovation that stoke the economy?

    Or perhaps universities are engineers of social change, meeting the promise of youth with opportunity?

    The answer – or my answer, at any rate – is that universities are all of these things and no doubt much else besides. (We should be grateful that have the simple word ‘university’ as shorthand.) And, given that we’re trying to wield a Swiss Army Knife, how do we design it to do all its jobs effectively.

    For starters, we need to create tools that are fit for purpose.

    The diversity of our sector – of the institutions, the courses, the students, the academics – instantly equips us with an armoury of tools. We need to protect that diversity from forces that seek to erode the richness and creative approaches that fill niches and answer needs.

    Marketisation, a term that strikes terror into many an academic’s heart, can enhance this. Look at the innovative approaches of BPP and Pearson who have been forced to seek out market gaps and cater for non-traditional needs.

    But before anyone mistakes me for a rabid free marketeer, I should point out that, as often as not, the market in HE is a blunt instrument driving conformity. University league tables, for example, are an insidious but addictive mirage enticing institutions to ape the trappings of our most revered universities. Those universities deserve reverence, but we shouldn’t build a sector encouraged to palely imitate a single blade, when what we need is a whole pocket-knife.

    These issues are set in high relief within The 1994 Group itself, which faces a real challenge to demonstrate its distinctiveness, to stand up for excellence in more than just research, but also in the wider student experience – the teaching and learning experience in particular.

    One example of just a bold and admirable demonstration would be for the 1994 Group – or any other mission group for that matter – to be the first to embrace the call of the Higher Education Academy and NUS to say that every teacher – or even every new teacher – we employ shall be qualified to teach (which may involve recognising and celebrating existing skills as much as demanding new qualifications).

    The University of Huddersfield made just such a commitment in 2011 by announcing its aim to ensure all its academic staff should achieve HEA Fellowship before the end of this year. They have made good progress towards this target and if one university can do it, why not a group of leading institutions?

    As well as tools that are fit, we need tools that work together. It’s no good if the corkscrew gets in the way whenever you want to remove a stone from a hoof.

    So it is with finance, for example. The funding changes of 2011 claimed to set the needs of students at ‘the heart of the system’, empowering them to decide what they want to study, elevating them to infallible customers. Meanwhile, their interests have been pitted against those of other those other HE consumers, employers, hungry for certain skills, but facing a menu offering only those they don’t need. [See note 1]

    One answer to this is better information, advice and guidance (IAG) for prospective students – and for the big recruiters (who, all too often, use the combination of Russell Group, 2:1 and STEM as the only markers of a good candidate. No wonder they struggle so hard on diversity issues). Perfect knowledge makes for a better market.

    Unfortunately, the Government has driven a wrecking ball through our IAG services. It is not necessary after the age of 16 for a school or college even to mention the word ‘careers’, let alone provide expert, independent and impartial guidance. Michael Gove in particular claims to be eager to educate students for careers, but has done almost everything in his power to avoid educating students about them.

    The Government has axed AimHigher, axed Connexions and axed Next Steps. In their place is the National Careers Service – a non-specialist website and helpline that most young people don’t even know about, let alone are they motivated to use it. Whenever IAG requires young people to be proactive, it will always favour those who would not get left behind in any system and it will abandon those who need help most: the ones who don’t know that questions need to be asked, let alone what those questions are and, still less, the answers.

    The responsibility of IAG has instead fallen on the universities as a by-product of their access arrangements. After forking out for tuition fee waivers (which reek of red herring) and for bursaries, many universities have spent their remaining funds for fair access on outreach activities in local schools. This is good news. As Les Ebdon of OFFA has said,

“Let there be no doubt – sustained, well-targeted outreach can be very effective and we want to see more of it.”

How to produce an access agreement for 2014-15, Office of Fair Access, January 2013

    Sadly, in areas of the country without such universities, we see forgotten pockets of young people. Also, with the best will in the world, universities are hardly impartial, nor even necessarily expert about the right options for a school-leaver. For some universities, outreach has been little more than an excuse to channel marketing costs through their fair access budget. [See note 2]

    Similarly the Key Information Sets and the whole of David Willetts’ data transparency agenda are not the right tools. Make no mistake, these are welcome initiatives. They will provide more indicators for universities to continue to drive up their standards, but they will not transform the landscape. A cynic might argue that they are ideological cover for charging students more – so long as you make sure students can know what they’re getting for their money, it’s justified. Or justifiable. Never mind that all this data is meaningless in a guidance vacuum, especially to the most disenfranchised in society.

    This is not the personalised support our young people need. If the Government won’t acknowledge that careers education is one of the best investments it can make on behalf of taxpayers, then everyone else who stands to lose out from poorly informed students needs to step up to the plate. Universities and employers need to collaborate to invest in a national initiative to take IAG into our schools and colleges. I run one such scheme, but it is far from the only one, nor sufficient on its own. [See note 3]

    Nevertheless, however right we get our IAG, we will always be King Canutes wailing at the tide if we do not go with the flow of demand and supply, which means channelling them, using the tidal force to drive change. We need to tie what universities can offer students to the needs of the labour market.

    One way to do that would be to fund student places through linking students’ ability to work and earn to the funding of the university that prepared them to do so (such as a tax charged directly to employers and hypothecated back to the graduate’s place of study, instead of doing the same indirectly in the form of student loan repayments). This would encourage universities to draw out (an expression from which we derive the word ‘educate’, by the way) employability skills as the natural product of what they do. It would work in the interests of students, of industry, of the economy and of all universities that genuinely foster and develop talent.

    This is what the marketisation of HE really should mean: getting our ducks in a row, aligning interests towards common goals. We need to agree on what universities are for, ensure that the market forces are there to sharpen and shape the right tools and ensure alignment so that tools don’t work against each other.

    The market is not immoral in itself, but neither is it a force for good. It is amoral. We need to decide what we think ‘good’ is and then deploy market forces to do the heavy lifting, because it’s a big and important job and we need all the help we can get.

[1] A leading finance sector recruiter boasted to me recently that they had managed to reduce the time it took for a new graduate employee to add value from a year to nine months. I could not help but wonder why it should be more than nine hours?

[2] See ‘Universities are misspending money that should be spent on access’, The Guardian, Mike Baker, 16/7/2012

[3] Push – see www.push.co.uk

The post What are universities for and how do we achieve it? appeared first on Johnny Rich.