As noted in my tweet yesterday, this week’s topic is the efforts by the UK to assess the quality of its academic research base. Paul Jump’s article in the Times Higher offers a wonderful introduction to the history and evolution of research assessment efforts in the UK up through to the current Research Excellence Framework. Please think about whether something like this could or should happen in your own country and whether the benefits of such activities outweigh their costs and whether their current format is better than previous incarnations. Some other questions to consider include:
- How effective is the current approach?
- What impact do such assessments have on academic researchers? What are the forces that would be aligned for and against research assessments?
- Do the transactions costs imposed on the scientific community in undertaking such an assessment outweigh their benefits?
- What are the implications for Mode 2 knowledge generation? Will assessment focused on ‘impact’ necessarily lead to more impact?
Impact has been one of the key elements of the Research Excellence Framework and, arguably, the main difference from all previous assessments. Sir Andrew Witty, the CEO of GlaxoSmithKline, has just released a new report on Releasing a British Invention Revolution, which was commissioned by the Department for Business, Innovation & Skills. The report is worth reviewing in any case, but on this specific question, impact was viewed as essential and an elements that should rise in future assessments (interestingly he raises these issues in the discussion on SMEs).
Another key concern if whether the REF threatens academic independence and places increased pressure on academics? In a survey by the union UCU, 4% of academics and, in particular, ‘more than 10 per cent of academics at eight UK universities’ have been ‘threatened with redundancy’ according to THE.
As a word of warning: always consider the source and reliability of the data that underpins policy arguments. ‘Convenience’ samples should, as a general rule, not be trusted. Although the survey was nominally open to non-union academics, UCU is obviously better placed to encourage its own union members to participate. One might expect union members to be more sceptical of the REF than others. Further, only a fraction of academics aware of the survey responded, and one might again expect that those who responded to a survey on the REF would feel more threatened. In general, one should beware of surveys which claim many thousands of responses. The implication that a survey with 7500 responses is credible is deeply misleading. There are 180,000+ academics at UK higher education institutions. So is a sample of 4% of all UK academics useful? A properly representative random sample of 1000-2000 academics would be expected to have a margin of error 2-3% (the same is true for a UK population of 60,000,000!) and we would be able to draw credible inferences from the results of such a random sample. The problem is that this was nothing like a random sample and there is no reason to believe any of the numbers in the survey.
Regardless of the lack of representativeness of the survey and the problems drawing inferences about University X or Y, the survey does at least demonstrate that a some unknown minority of academics do have serious fears about the impact of the REF on their careers. In some departments and universities, this has become extremely contentious, as illustrated by the case of the History Department at Lancaster University.
Finally, there are inevitably concerns that, however much it might be claimed otherwise, interdisciplinarity will always be disadvantaged in a framework centred around disciplines (or Units of Assessment as known in the REF). In so doing, the REF can be seen as undermining Mode 2 knowledge generation activities and encouraging more traditional silo approaches. On the other hand, one might expect Mode 2 knowledge generation activities to create the sort of impact that the REF seeks to encourage. In a recent speech on Universities and the Poorest Billion by our Vice-Chancellor, Professor Sir Leszek Borysiewicz, he acknowledges the concerns of many academics with regard to undertaking such Mode 2 activities, but leaves little doubt as to where he comes down in this debate:
But our academics need structural support and they must believe that the University is itself committed such that it recognises their activity as core to the mission. Academics worry whether exercises such as the Research Assessment Exercise in Australia or REF in the UK would be unsympathetic – which inherently they are not! However, putting it bluntly, if saving lives interferes with research assessment models, then which one needs to compromise?