The methodology of the RAE

It is curious that scientific papers generally have sections on methodology, evidence, modes of measurements, actual data, discussions and conclusions. The aim is to be non authoritative. A quotation attributed in some form to Galileo and Einstein is: `Authority is the enemy of true science.'

By contrast, the  RAE embeds `authority' at the centre of assessment. It assumes there is a well understood notion of `excellence', and that a few chosen people are well placed to determine this. Their judgements are given as if ex cathedra, contrary to long standing academic traditions, which were developed for good reasons. Also, the `excellence' determined is that of departments, or research units, not of individuals, except collectively. Yet famous people in science, such as Francis Crick, have urged that research is done by individuals. All those in a high rated department get much more funding than all those in the next rated department. There is no mode for the RAE to support talented individuals, or exploration for its own sake, a common method of progress in science. Further, what is a `research unit', in this internet age?

The RAE gives its criteria in terms of `national excellence' as against `international excellence'. Does this imply the UK is determined by our masters to be second rate? The glory of mathematics and science is their independence of geographic, national, and political boundaries! In 40 years of refereeing and reviewing, I have been asked always how good is the work, but not: is it of national or international excellence? A phrase used recently in a newspaper in regard to British culture was `it shows a cringeing respect for all things foreign'.

There are sugestions that the intention of the Government is to be more selective. No evidence is given that this will increase the innovation of British Science. Let us make the following analogy. The American Rose Growers Association have informed me that to grow a champion rose, a grower should invest in 20,000 seedlings, to budding stage. Governments would presumably regard this as inefficient!

The Panels do not ask for information on what new lines have been started? or developed? on what explorations of possible new worlds have been undertaken? Without such explorations, we are living on capital, on our reserves.

It seems to me that those who work out these systems, and possibly also those who apply them on the Panels, have read nothing on the progress of science, such as the work of Thomas Kuhn  on the `Structure of Scientific Revolutions'. The RAE Guidance document  does not contain the word `pioneering' and it contains the word `innovation' once only, and that with regard to `technical'. The word `excellence' is well used - once again, Orwell's comments on the impoverishment of language are relevant. How will pioneering and innovative scientific departures take place in the UK under this financial reward system?

There is no place in the RAE assessment for long term projects, since the Panels  have a 4 or 5 year time view. Long term projects go up and down in their publication profiles, as major milestones are reached, followed by consolidation. Initially, such projects may seem small time, far from `mainstream'. Big research themes rarely come like Venus Anodyamene, fully formed from the sea, though an individual (not a committee) may have such a vision. The Panels find it easiest to judge `excellence' by publication in `top journals', which are of course run by the `top people'. Thus does authority propagate itself! Even `mainstream' is shown by experience to flap around like a flag in a storm!

Einstein said: "To punish me for my contempt for authority, fate has made me an authority myself.." As an authority, he told de Sitter that the notion of an expanding universe was `bad physics'. In the current UK regime, that would surely have been enough to close de Sitter's department!

A report on a Bangor research proposal in 1989 wrote: "There were substantial reservations, however, about how interestingly or significant the results would be outside your immediate school, as the impact of this type of work does not appear to have been widely felt so far, in spite of the quite impressive publication record cited." I thought we were asking for funds to help develop this new world we had found! We had major publications in 1981-89, involving collaborations with Philip Higgins, Edmund Robertson, David Johnson, Jean-Louis Loday, among others, as well as a correspondence with Alexander Grothendieck, 1982-88! The time lag for influence, and recognition of the influence, of a new mathematical idea is long (cf., Galois, Grassmann, ...). Our first paper on the nonabelian tensor product of groups was published in 1984, and the bibliography on this topic has now reached 99 items. But this topic is only a glimpse of the world of multiple groupoids and their possible applications.

Groupoids were defined in 1926, and my book of 1968 emphasised their utility in 1-dimensional homotopy theory, convincingly,  as I thought. Yet this idea has not been taken up generally in texts in algebraic topology. Abstract groups are currently much more important in the scientific community than abstract groupoids. How long will this last? Subsequent work on higher dimensional analogues of groupoids has proved difficult, as is most exploration, and not too easy to take up, since there are, apparently, few easy pickings (what have I missed?). Lie groupoids are becoming increasingly important, since the pioneering work of Ehresmann and Pradines.

Perhaps also the fact that a research proposal is judged on sociological rather than mathematical grounds reflects on the lack of a serious development of what one might call `mathematical criticism' or `analysis of methodology', of what constitutes `good mathematics'.  So there is left the naive analysis: `Good mathematics is what the top people do!'. This type of view has been strongly attacked by Grothendieck in his long but unpublished book: `Recolte et Semaille'. David Corfield has discussed related issues in a recent book.

The history of science and mathematics shows that major developments have arisen from young and ambitious people working on their own, and following through an idea unappreciated by the established, and therefore ripe for the picking. A new idea does not get published in the `top' journals, or attract research grants from Research Coucils,  unless it immediately fits with their current paradigms, and therefore is less original! This aspect of the dynamics of research progress seems foreign to the RAE. Anyone who doubts this aspect should read books by Lovelock on `Gaia', or consider the history of Wegener's theory of `continental drift'. It is surely too cynical to suggest that the RAE replace the word `excellence' by the word `kosher', but that does suggest that the methodology of research assessment, as far as it can be determined, is related to that used in the past to judge heresies.  

Here is a quotation from G.-C. Rota (for which I am grateful to David Corfield):

"What can you prove with exterior algebra that you cannot prove without  it?" Whenever you hear this question raised about some new piece of  mathematics, be assured that you are likely to be in the presence of something important. In my time, I have heard it repeated for random  variables, Laurent Schwartz' theory of distributions, ideles and Grothendieck's schemes, to mention only a few. A proper retort might be: "You are right. There is nothing in yesterday's mathematics that could not also be proved without it. Exterior algebra is not meant to prove old facts, it is meant to disclose a new world. Disclosing new worlds is as worthwhile a mathematical enterprise as proving old conjectures."
(Indiscrete Thoughts, p.48)

Which of these last will be more highly rated in a Research Assessment? Which is easiest to `measure'? What happens to those who try to disclose new worlds? Does `Flatland' have an answer? Professor Alan L. Mackay, FRS, described, in a lecture at Imperial College in 1985, the history of 5-fold symmetry in crystals: `First they said it is not true. Next, they said it is true, but unimportant. Eventually, they said: It is true, it is very important, and we have known it for years!'

Certainly two successive RAEs have not rated as internationally excellent our work in `Higher dimensional algebra', which in UK mathematics (as against computer science) is currently studied outside Bangor mainly, but not entirely, at Cambridge. Some outside the UK see this topic as a potential major mathematical theme of the 21st century! See the counters on the above site for the international interest (over 17,200 page views since May, 2000), and do a web search for other applications.

Is the RAE patterned so as to reduce diversity?  to deter exploration,and pioneering work? to stop individuals developing new areas? Does it seek value for money? Where is the comparative monitoring of this? What is the evidence for its methodology? It is notable that there are no routes for debate.

The idea of funding research according to measurements of qualities is not new. Sir Peter Swinnerton Dwyer was asked about this  at a meeting of Heads of Departments of Mathematics in the 1980s when he was Chairman of the University Grants Committee. He replied that there was a question not only of the amount  that should be allocated according to this judgement, but even of its sign!

A radical medical view is: `Diagnosis without treatment is unethical.'  The RAE Panels are not there to help and advise, but solely to assess. Yet they regularly bring in new criteria! A recent one is `sustainability and vitality'. Of course `sustainability' is to some extent in the RAE's disposal. They do not ask if the Assessment Unit has a long term record of a sustained research programme. The meaning of``vitality' is unclear.

The current methodology realises the old adage: `The best is the enemy of the good.' This is seen in the closures of good departments in the UK, for `inadequate' research attainments, as judged by by RAE Panels, whose judgements include no justification , no explanation of methodology or context, and no appeal. To what in wider terms should this situation be compared? It is certainly an emphasis on AUTHORITY as a control of research, which is generally supposed to date to prescientific eras!  

In mathematics, one can believe that Galois, Cantor, Grassman, ... , would have done badly in any supposed contemporary RAE. In the work of Heinrich Brandt, we have a short 1926 paper on groupoid as a generalisation of the notion of group. Even now, groupoids are not widely accepted as the natural context for many ideas on symmetry. But this paper is growing in influence. Who would have forseen this?

Part of a report of the Engineering Professors' Council meeting with Barry Sheerman MP, Chairman of the Education and Skills Select Committee

"On the Effects of the RAE and its impact on University management he seemed a little surprised at the EPC comments on the impact of the RAE exercise and the resultant emphasis on research with its consequent downgrading of teaching. He felt that the survey on student satisfaction might go some way to redress the balance. Although he felt that it would not be possible for his Committee to have an enquiry into the effects of the RAE, he did raise the possibility of organising an evidence session where the committee could question a number of VCs and the staff from HEFCE who would be running the RAE."

One can ask `What is professionalism in research?', and in attempting an answer one can turn to the  great: I like the comment of Dirac quoted in Out of Line, that `one should follow a mathematical idea wherever it leads.....'. One aspect of professionalism is to follow through an idea so that all its implications are tested and published: if it does not work out as well as hoped, that may be for a reason that someone else can spot, and pursue successfully.

Finally we give a quotation from the the Autobiography of Thomas Young, referred to in the book `The last man who knew everything', Andrew Robinson, Pearson Education Inc, 2006, p.224.

"It is indeed so impossible to forsee the capabilities of improvement in any science, that it is idle to form any general opinion of what would be the comparative advantage of the employment of time in any one investigation rather than another, for almost all the authors of important discoveries and even of inventions, are led as much by accident as by system to their success."

Compare with a recent article.  in the Oserver, Dec 21, 2008

October 13, 2006

Updated 22 December, 2008

Return to Popularisation and teaching page

Return to home page