Since the year 2000 I have been doing this: working as an evaluator / reviewer / auditor for the European Commission. The European Union awards a large amount of research funding to European consortia, and they need reviewers (experts) to judge the quality. There are usually many more proposals than there is funding available, and so these experts have the task to put the incoming proposals in a ranking - the top ranked proposals will then receive the funding.
This evaluation activity is done several times a year, and each time a new call is launched for proposals, the European Commission organises these reviews and invites experts all across Europe to participate in those reviews/evaluations. For the evaluators, this is a great honour to be invited, and it is also beneficial: they get to see the very latest state of research interests, and they can expand their network with other evaluators. Naturally all the proceedings are treated confidentially, so no evaluator can use the information obtained from reading those proposals.
Information about the current EU workprogrammes (Framework Programme 7 = FP7) is on the
CORDIS website. This site has also a section where everyone with any considerable professional or academic expertise can
register as expert. It is recommended to frequently update the profile there, because that is the expert repository from where the European Commission selects the evaluators so that their expertise matches the requirements for evaluating projects and proposals.
When the Commission invites the experts, they first send out an email for probing the availability at a given time. This is usually 4-6 months before the event and gives them an overview on how many experts would be available. About 3 months before the evaluation event an official letter will be sent with an invitation to participate in the evaluation. The expert then has to sign a contract, declaring any conflict of interest and complying with the confidentiality requirements. This then constitutes a binding agreement, and the expert can then book the travel - this will be reimbursed by the Commission later.
About 2 weeks before the actual meeting (which usually takes place in Brussels), the experts are given access to the material that they are to evaluate. Either the proposals are being sent by postal mail, or they are made available online. It is then the task of the evaluator to do a thorough review of the material and give a score, based on set criteria. These criteria may differ for each call, as the procedures slightly change over the years and are also different for each discipline. Th scores are then submitted back to the Commission before the meeting.
The meeting where all the evaluators come together can last a few days or a whole week. The purpose there is to come to a consensus: for each proposal, a number of reviewers (typically 3) provides a score, and these score may differ. In scheduled consensus meetings, these evaluators now discuss why they believe that their score is appropriate. They need to be prepared to change their score, because for the final ranking, the score is not simply the average of all evaluator scores, but is the score to which each evaluator agrees. There are mechanisms in place in case the evaluators have very stubborn opinions - another evaluator would join, and then they would try to reach consensus. They all sign the score sheet in the end, indicating their agreement of the score. One of them is assigned to be a rapporteur, writing the notes and comments into the final form that will be forwarded later to the applicants for the funding.
After all the individual project consensus meetings have been held, a final meeting with all evaluators takes place. The goal here is to achieve a ranking of all the proposals. Since not every evaluator will have seen every proposal, there is still a chance before that meeting to read the other proposals of interest. In this meeting, the highest ranking proposals are put to special scrutiny, because there could be objections by other evaluators who had been critical in their reviews. Important is that the same standard is applied in the overall scoring for each proposal.
I think that this process works quite well and is very well taking into account the difficulties in achieving a fair scoring. I noticed that over the past 10 years the scoring had been somewhat simplified: in 2000 there were 5 criteria, with different weighting. The latest evaluation I attended, only had 3 criteria, with equal weighting.
