�鶹

TEF data: so what can we count on?

Scrutiny of datasets informing the teaching excellence framework will ramp up, and rightly so, given the high stakes and risk of misuse

Published on

May 26, 2016

Last updated

May 26, 2016

John Gill

Twitter:

Data being analysed under magnifying glass (illustration)

Source: iStock

We live in the age of data. A word that once evoked the deadliest of dead-end jobs (I once spent a month doing data entry in an NHS deanery) now speaks of data scientists who rule the world from trendy cafes, powered by skinny soya lattes.

As the American statistician and management guru W. Edwards Deming put it: “In God we trust, all others bring data”. This is a mantra that’s revolutionising business, dismantling old industries and making billionaires of digital pioneers. It is also, inevitably, finding its way into public policy, and changing the fabric of academia, one of the most data-hungry professions around.

But data must be handled carefully, must be clean and meaningful, and understood and used in the right way – lessons we’re very clear about as the publishers of the Times Higher Education World University Rankings. And when it directly affects the prosperity of individuals or institutions, the integrity of data has to be protected as never before.

This is front of mind in the UK because of the forthcoming teaching excellence framework, which we now know will be tied to university funding (by informing tuition fees) despite intensive lobbying by universities and students to resist such a link.

�鶹

Last week’s White Paper softened the approach, giving a two year bedding-in period, but we should be in no doubt that more aggressive financial incentives are coming.

One result is that the scrutiny of the data fuelling the TEF will also ramp up, and in our news pages this week we highlight just one of the potential chinks.

�鶹

Our investigation reveals that the funding council has looked into claims that institutional staff have tried to get undergraduates to inflate their course ratings in the National Student Survey (one of the TEF metrics) on 30 occasions over the past seven years.

This isn’t a huge number in the scheme of things (and only four were serious enough to warrant NSS results being withheld from data sources such as Unistats), but it shows that there’s already pressure to a degree that is warping individuals’ actions.

That pressure will intensify dramatically with the introduction of the TEF.

The reality is that there will always be flaws with datasets of this kind, and the risk of misuse and abuse is equally inevitable.

�鶹

Most academics will also accept that imperfect data can be useful if handled in the right way (although it has to be said that government has a pretty ropey record in this regard, particularly when it comes to supposedly data-led reform of the NHS).

But while accountability and competition are indeed vital components in universities’ success, there is an underlying worry here: that a toxic culture develops in which everything is measured and the measurement becomes an end in itself.

Deming’s quote about the divine nature of data is counterbalanced by his “seven deadly sins” of management.

Among them are “lack of constancy of purpose”, “evaluation by performance, merit rating, or annual review” and “management by use only of visible figures, with little or no consideration of figures that are unknown or unknowable”.