A dinosaur hangs up his claws: why I am leaving the EPSRC College of Peers

The EPSRC is the main research funding body in the UK for computer science and software engineering. I wrote my first EPSRC proposal in 1981 and have reviewed proposals, served on panels and helped with making funding decisions for at least 25 years. Over that time, I guess I must have spent hundreds of unpaid hours reviewing proposals and being involved in panel meetings. However, I will do no more reviews – I have now resigned from the EPSRC College of Peers.

My resignation has been prompted by a small thing – a ‘requirement’ for members of the College of Peers to complete an online training course that I discuss below. However, I think that this is indicative of a culture change that has occurred in the research council over the past few years, where the EPSRC seems to me to have lost sight of what should be its principal objectives – to support high-quality research and to nurture the research community.

Let me start with the most recent piece of nonsense. Earlier this month, I received an email from the EPSRC inviting me to take an online training course for proposal reviewers. It included the following paragraph:

“We believe this training provides valuable information on EPSRC and our strategies and priorities, and essential guidance on best practice on reviewing proposals as well as training on roles and responsibilities for being part of an EPSRC peer review panel. For this reason we are making this training a mandatory requirement for all college members.”

The training course is a generic, on-line course (i.e. it is not tailored to the discipline of the College member) and those taking the course are required to successfully answer 12 out of 15 multiple choice questions. It is not clear what happens if you fail.

I am under no illusions that I know everything about reviewing proposals and I am sure that I could learn from experienced reviewers of proposals in computer science and software engineering. However, I have NEVER seen a generic on-line training course which is not superficial and trivial and I see no way in which any such course might change my reviewing practice. Asking experienced reviewers to complete such things is simply a consequence of a quality system that favours process over thought and which believes that so long as all boxes are ticked, then all is well.

I object on principle to the notion that you can assess how good a reviewer is by multiple choice questions which are almost certainly put together by someone who has never reviewed a proposal in their life. I object in principle that the EPSRC expect me to spend an hour of my time on trivia, without reimbursement, that will be of no benefit to anyone.

I object in principle to the notion that my reviews should take EPSRC priorities into account, whatever these priorities are. What I think of these priorities is not important. I recognise that when funding is limited there have to be priorities but implementing these is a funding decision and there is no need for reviewers to know anything about them. Reviewers should use their knowledge and judgement to assess the scientific merit of a proposal and that is all – they are rarely qualified to do otherwise.

Of course, many of my colleagues will simply ignore the email and nothing will happen. The nature of these things is such that no-one will pay any attention to whether or not this course has been completed and reviewing will continue as before. I could do the same thing and, it might be argued, objecting as an insider is more likely to be effective than as an outsider.

Sadly, this view is not supported by recent history. The EPSRC have made a number of policy changes over the past few years, which many of us thought were unwise- we pointed out the flaws and we were ignored. Let me turn now to some of these policy changes that I believe have been damaging to the research community, particularly to early career researchers and researchers who have taken career breaks.

  1. The ban on resubmission of rejected proposals – introduced supposedly to reduce the number of proposals for review. This is simply daft. None of us get things right first time but this policy hits younger researchers with less experience of proposal preparation and those adventurous and innovative proposals that are exciting but flawed. Inevitably it leads to incrementalism and a preference for proposals from experienced researchers who already understand the system.
  2. The shift in PhD funding to Doctoral Training centres (CDTs). Not only is putting all your eggs in a small number of baskets unwise, it leads to researchers being trained in a small number of areas which may or may not be important in future. It means that there is less diversity in the expertise of new academics which can lead to difficulties in teaching important parts of the curriculum (such as software engineering) which are not fashionable and so not a topic for a CDT. As far as I am aware, there is no empirical evidence that these centres improve the quality of the PhD experience for research students.
  3. The focus on few large grants rather than many smaller grants. Again, this makes it harder for early career researchers to get started and reduces the diversity of the project portfolio. To be successful, this requires an ability to foresee the future and know what will be important. None of us have such an ability. The empirical evidence that exists here suggests that research from smaller projects received more citations. (Big Science vs. Little Science: How Scientific Impact Scales with Funding). Disclaimer: I have personally benefited from this policy over the past 10 years – but this does not mean I think it is money well-spent.
  4. The abolition of project studentships. Bidders for research projects can no longer request support for a PhD student but must instead request support for a more expensive, salaried RA. The EPSRC have a First Grant scheme for young researchers (one of their better ideas) but this has a relatively low limit on funding. It was ideal for funding a PhD student and this was a great way for a young researcher to get started. Now it is impossible to fund an RA for 3 years on the First Grant scheme and so early career researchers are forced into shorter term projects.

I could go on – but these are the most significant policy changes that have, I believe , seriously damaged the CS and IT research community. Of course, difficult decisions have to be taken when resources are limited and for sure some people will be unhappy. But these decisions which threaten the long-term health and diversity of the research community are profoundly wrong.

The latest nonsense on training, unimportant in itself, is, for me, the straw that has broken the camel’s back. It is indicative of an organisation that does not really understand or know the academic research community, whose concerns are to reduce it own workload rather than support the most exciting research and which favours tick boxes over professionalism. I know there are many well-meaning people involved in the EPSRC – both academics and staff who are often underpaid and overworked. Full-time staff are understandably unwilling to challenge the culture of the organisation and I suspect any academics who do so are quickly eased out. The EPSRC has been taken over by ‘professional managers’  who pay lip-service to consultations with the community and who do not care about nurturing that community.

I believe that it is time to say that we have had enough – if we wish to preserve our community in the long-term, we need to withdraw our support for this flawed organisation and agitate for change. I have therefore resigned from the EPSRC College of Peers and will no longer work unpaid for them (nor will I apply for research support as I have said in this post). I hope that others will join me.

Afterword

The email from the EPSRC’s Peer Review Manager acknowledging my resignation stated “However, although you may no longer be a College member, we may still call upon you for the occasional review where appropriate”. So it seems that training isn’t necessary after all!

15 thoughts on “A dinosaur hangs up his claws: why I am leaving the EPSRC College of Peers

  • July 17, 2013 at 10:00 am
    Permalink

    100% agree with your 4 criticisms (which many of us told the EPSRC we thought were major errors)

    And of course, the impact section of proposals…

    The afterword is chilling – the lunatics have taken over the asylum!

    Reply
  • July 17, 2013 at 10:07 am
    Permalink

    You summed up beautifully what i have wanted to say for the last 3 or so years. Whenever i broached the subject with the EPSRC re: doctoral training centres, ban on resubmission etc, i was told that this was just sour grapes because i am at a new university…this isnt the reason I have done well regarding grants, i just thought their ideas were flawed.

    Reply
  • July 17, 2013 at 10:36 am
    Permalink

    Well said Ian. Before I saw this I wrote to the EPSRC asking to be removed from their mailing lists. (Easier for me because, being retired, I am no longer under pressure to bring in grant money.)

    I have thought for some time that EPSRC is not fit for purpose, not least because like many organisations set up to do something useful it has drifted into trying to preserve itself by telling its masters what it thinks they want to hear, rather than trying do do what is needed to produce great research (like many university central administrators — with notable exceptions).

    One example is their repeated failure (at least until I stopped looking) to acknowledge the achievements of EU research funding, e.g. in the conditions for grants for technology transfer. Such funds should be allocated on the basis of the merit of the technology not who funded the research.

    I think this sickness has now spread to many university academic departments also — e.g. evaluating young academics according to where they publish rather than what they write.

    Vice chancellors join in, constantly talking about ‘our competitors’ rather than ‘our collaborators in providing outstanding national higher education and research’.

    It’s a deep sickness of our time, well documented in the BBC TV trilogy “The trap” probably still availalable on Youtube, summarised here:

    http://en.wikipedia.org/wiki/The_Trap_%28TV_series%29

    Reply
  • July 17, 2013 at 11:00 am
    Permalink

    Very well said, Ian. The move away from many decisions about small grants which seemed to be good indicators of quality, to a small number of large decisions, which inevitably become political or arbitrary, is a fundamental problem. The CDTs are an exercise in wishful thinking that everyone is being forced to play in, at great cost in terms of wasted time, and stresses within organisations. While some will be excellent, many of the resulting projects will be much less effective (cost or research) than project studentships.

    Reply
  • July 17, 2013 at 12:03 pm
    Permalink

    I agree with you on all points – especially about CDTs and the shift to large grants. It’s been many years since I’ve bothered with EPSRC funding – it always seems too hard to use it to fund the smart people I want to work with.

    Reply
  • July 17, 2013 at 10:20 pm
    Permalink

    Excellent post. Even though I have greatly benefitted from EPSRC myself and am grateful for their funding, a number of their recent decisions have left me rather concerned and perplexed.

    Reply
  • July 18, 2013 at 10:55 am
    Permalink

    Paul Golby has been chairman of EPSRC since April last year; you should take a look at his article in the Times Higher that addresses some of these issues. So this is maybe a good time to voice our concerns.

    Reply
  • July 18, 2013 at 9:53 pm
    Permalink

    In defense of the EPSRC, pushing questionnaires to college to make people re-engage with the core tenets of reviewing is not a bad thing. Yes, the additional criticisms that Ian makes are valid (albeit the resubmission thing is in practice not a bad thing), but reminding the folk reviewing proposals what they are meant to be looking at is a *good* thing. There are too many random reviewers out there who look at negative rather than positives.

    cheers
    ian

    ps Those reading may want to take my comments with a pinch of salt as I don’t know what the questionnaire asked, because when I tried to take the questionnaire my password was rejected from their moodle site, after having logged into JES.

    Reply
    • July 18, 2013 at 10:11 pm
      Permalink

      I am not sure about the assertion that ‘resubmission thing is in practice not a bad thing’. You can’t tell. We simply don’t know which excellent proposals that just missed funding were not resubmitted.

      Over the past year or so, I have reviewed proposals for Swedish, Finnish and US funding bodies – all expected me to be professional and understand their criteria; none suggested that I had to complete some trivial multiple-choice test.

      Reply
  • July 18, 2013 at 10:04 pm
    Permalink

    Crowcroft, at this point I do call you out; impact is important as something to think about. There is a duty to the taxpayer to explain why anyone should fund the research you want to do. Curiosity driven research is wonderfully important, but you need to explain why this abstruce thing should be examined from this viewpoint, so that other academics will gain further insight. It need be no more than that for some research, but for much of the stuff that we and others do, kicking us to ensure we talk to the right people before the proposal goes in is a positive thing.

    Reply
  • August 1, 2013 at 12:28 pm
    Permalink

    Ian,

    I found the link to your blog via the THE article on the recent EPSRC review (to which I contributed ‘evidence’ on behalf of the CDBU ).

    Thank you for such an excellent summary of the many problems with EPSRC. I can only hope that others will join you in resigning from the peer review college. (My hopes aren’t high, however…)

    I resigned from the EPSRC peer review college a number of years ago in response to their introduction of the “Pathways to Impact” requirement. I am not going to rehearse my arguments here about just why I resigned ( and all that is wrong with Pathways to Impact, National Importance, the frankly unethical “demand management” scheme, the focus on “research leaders” (and I say that as an EPSRC Leadership Fellow) and rapid shift away from responsive mode funding, CDTs, the inability to resubmit, etc…). For the small number of people who may be interested, this blog post and the links therein summarise my key misgivings.

    There was also a rather heated debate in the (lengthy) comments thread under this post by Athene Donald , which was posted last year in response to the “Science For The Future” campaign.

    Once again, thank you for highlighting the many problems that exist with EPSRC. It appears that Paul Golby may well be rather more willing to engage with the EPSRC-funded community than the previous Chair. I was also encouraged that the independent review discussed in the THE article was certainly not a “whitewash”. There is also an upcoming review on EPSRC’s peer review mechanisms. Perhaps there is room for some optimism…

    Reply
  • August 1, 2013 at 12:40 pm
    Permalink

    I should also have mentioned in the post above that because I no longer review EPSRC grant proposals, I also don’t submit proposals to them.

    Reply
  • August 13, 2013 at 11:35 pm
    Permalink

    Ian

    Fantastic posting. Of course, I expected no less from you. I am someone who studied your books as core text in my Computer Science degree. I love your entire post in fact. Unfortunately, I am subjected to multiple wasteful surveys in my core duties as a CS lecturer by HR. Well done on taking time to put fingers to keyboard…..

    regards
    Kevin

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *