PERSPECTIVE

Evaluate the Evaluations!

RAGHAVENDRA GADAGKAR

Centre for Ecological Sciences and Centre for Contemporary Studies, Indian Institute of Science, Bangalore, Karnataka 560 012, India. ragh@iisc.ac.in

https://doi.org/10.29195/DSSS.01.01.0005

Bad practices seem to spread very rapidly and go to near fixation, making it very hard to subsequently replace them with good practices. Hence they linger long after their ‘badness’ has become widely recognized. Evaluating scientists and scientific papers by the Impact Factors (IF) of the journals in which they are published is one of the most pernicious of such lingering bad practices. In retrospect, it seems shocking that the practice of using IF has become so widely and uncritically accepted. The odds are so heavily stacked against the practice that I would have guessed that it won’t take off the ground. As has been mentioned in many forums, IF measures the impact of the journal and not of the paper, citation practices vary from discipline to discipline and ‘IF pressure’ is sure to lead to bad publishing practices. The bad practices associated with the evaluation procedures go well beyond the use of Impact Factors. Since evaluations are best done by peer groups, the business of eliminating bad practices and ushering in good practices is best attempted as a self-organized process by academics themselves with as wide a participation as possible. Science academies have a critical role in functioning as conscience keepers to usher in good practices and as gatekeepers to keep out bad practices. I believe that science academies around the world are not doing as good a job in this regard, as they potentially can. Recently, three prominent academies, Academie des Sciences of France, Leopoldina of Germany and The Royal Society, London have issued an excellent, joint statement about what they consider good and bad practices.

Although the points they make have been repeated time and again, coming as a joint statement from three of the world’s prominent science academies brings with it a certain amount of authority and credibility. Besides, there are some points in this statement that are especially worthy of note. Apart from unambiguously pointing the finger at the excessive use of bibliometric data, the statement suggests reducing the number and frequency of evaluations in the first place; evaluating, training and nurturing the best evaluators, and cautions that the new so-called ‘Altmetrics’ may not be much better after all. I believe that this statement should be widely read by all scientists and hence I am reproducing it below. Nevertheless, it is a statement by three ‘foreign’ academies. Hence, I would urge our own three science academies to seriously study the matter in the Indian context and issue our own statement and bring to bear the pressure of their authority on the conduct of evaluations in India. Clearly, we need an urgent evaluation of the evaluation process itself

1

October 27, 2017

Statement by three national academies (Académie des Sciences,

Leopoldina and Royal Society) on good practice in the evaluation

of researchers and research programmes

1. Introduction

The large increase in the size of the international scientific community, coupled with the

desire to ensure the appropriate and efficient use of the substantial funding devoted to

supporting scientific research, have understandably led to an increased emphasis on

accountability and on the evaluation of both researchers, research activities and research

projects (including recruitment, as well as the evaluation of grants and prizes). Given that

there is a large diversity of procedures currently used in evaluations which have accumulated

over time, it is now necessary to provide some guidelines for best practice in the evaluation of

scientific research. Peer review, adhering to strict standards, is widely accepted as by far the

best method for research evaluation. In this context, the present statement focuses on the

evaluation of individual researchers.

Such an assessment by competent experts should be based on both written (journal articles,

reviews, books, book chapters, patents, etc.) and other contributions and indicators of esteem

(conference presentations, awards, public engagement activity, peer review activity, datasets

shared, seminars, etc.). As a careful evaluation of scientific content and quality by experts is

time consuming and costly, the number of evaluations should be limited and only undertaken

when necessary, in particular for decisions on competitive academic appointments or funding

large projects.

With the increase in the number of evaluations and the emergence of easily accessible

electronic databases, the use of bibliometric measures has become an additional tool.

However, there has been too much reliance on bibliometric indices and indicator-based tools

as measures of performance by many evaluation committees and exercises, leading to the

REPRODUCED WITH PERMISSION

2

danger of superficial, over-simplified and unreliable methods of evaluation. This bad practice

involving the misuse of metrics has become a cause for serious concern.

Of particular concern are the widely used journal impact factors (IF) which are an estimate

of the impact of the journal itself rather than the intrinsic scientific quality of a given article

published within it – a point that has been made on several occasions and notably in the San

Francisco Declaration

(1)

. Outstanding and original work can be found published in journals of

low impact factor and the converse is also true. Nevertheless, the use of impact factors as a

proxy for the quality of a publication is now common in many disciplines. There is growing

concern that such “IF pressure” on authors has increased the incidence of bad practice in

research and the ‘gaming’ of metrics over the past two decades, in particular in those

disciplines that have over-emphasized impact factors. Also, the so–called ‘altmetrics’ – a new

form of impact measure – while adding an important and hitherto overlooked dimension to

the measurement of impact, suffers from some of the same weaknesses as the existing

citation-based metrics.

There is a serious danger that undue emphasis on bibliometric indicators will not only fail

to reflect correctly the quality of research, but may also hinder the appreciation of the work of

excellent scientists outside the mainstream; it will also tend to promote those who follow

current or fashionable research trends, rather than those whose work is highly novel and

which might produce completely new directions of scientific research. Moreover, over-

reliance on citations as a measure of quality may encourage the formation of aggregates of

researchers (or “citation clubs”) who boost each others citation metrics by mutual citation. It

thus becomes important to concentrate on better methods of evaluation, which promote good

and innovative scientific research.

2. Principles of good practice in the evaluation of researchers and research

activities

Essential elements for the evaluation of researchers can be summarized as follows:

2.1. Selection of evaluation procedures and evaluators

Evaluators

Since the evaluation of research by peers is the essential process by which its quality and

originality can be estimated, it is crucial to ensure that the evaluators themselves adhere to the

highest standards and are leaders in their field. The selection of evaluators should be based on

their scientific excellence and integrity. Their scientific achievements should be widely

recognised and their curriculum vitae and research achievements should be easily accessible.

Such an open process will ensure the credibility and transparency of the evaluations.

3

Evaluation processes

Since the number of excellent evaluators is limited, the number of evaluation processes

should be reduced in order to avoid over-use of first-class evaluators. There is a concern that

different agencies and institutions have carried out an excessive number of routine evaluations

over the last decades, putting too much pressure on the best evaluators. First-rate evaluators

are increasingly reluctant to commit to time-consuming and unproductive evaluation

exercises. It is of great importance to reduce the number of evaluations and to confine them to

the core issues of research that only peers are able to judge. Evaluators provide a “free

resource” as part of their academic duty and this resource is over-exploited. Evaluating bodies

must recognise that good evaluation is a limited and precious resource.

A page limit for submissions to all evaluation processes is needed. Excessively long

submissions are counter-productive: evaluators need to be able to concentrate on the

essentials, which is problematic with very lengthy submissions.

Rotation of evaluators is essential to avoid excessive or repeated influence from the same

opinion leaders. The panel of experts should be adapted to reflect the diversity of disciplines

or scientific domains. Although gender and geographical distribution will be factors in the

selection of evaluating groups, excellence must remain the primary criterion.

2.2. Ethical guidelines and duties of evaluators

Evaluators should clearly declare possible conflicts of interest before the evaluation

process. The confidentiality of expert reviews and of the discussions in the evaluation panel

must be strictly respected to protect both the evaluators and the evaluated persons.

While reviewers have often learned the practice of evaluation by experience and self-

teaching, this competence cannot be taken as given. Methods and approaches to evaluating

and reviewing should become part of all researchers’ competence as should the ethical

principles involved. Evaluators should be made aware of the dangers of “unconscious bias”.

There should, as far as possible, be equivalent standards and procedures for all research

disciplines.

The evaluation procedures must also include mechanisms to identify the cases of biased

or otherwise inappropriate reviews and exclude them from consideration.

2.3. Evaluation criteria

Evaluations must be based under all circumstances on expert assessment of scientific

content, quality and excellence. Publications that are identified by the authors as their most

important work, including major articles and books, should receive particular attention in the

evaluation. The simple number of publications should not be a dominant criterion.

Impact factors of journals should not be considered in evaluating research outputs.

Bibliometric indicators such as the widely used H index or numbers of citations (per article

4

or per year) should only be interpreted by scientific experts able to put these values within

the context of each scientific discipline. The source of these bibliometric indicators must be

given and checks should be made to ensure their accuracy by comparison to rival sources of

bibliometric information. The use of bibliometric indicators should only be considered as

auxiliary information to supplement peer review, not a substitute for it.

The use of bibliometric indicators for early career scientists must in particular be avoided.

Such use will tend to push scientists who are building their career into well-

established/fashionable research fields, rather than encouraging them to tackle new scientific

challenges.

For patents a clear distinction should be made between the stages of application, delivery

and licensing.

Success in raising research grant funding should, where relevant, be only one and not the

dominant factor in assessing research performance. The main criteria must be the quality,

originality and importance of the scientific research.

3. Short summary of the main recommendations

Evaluation requires peer review by acknowledged experts working to the highest ethical

standards and focusing on intellectual merits and scientific achievements. Bibliometric data

cannot be used as a proxy for expert assessment. Well-founded judgment is essential. Over-

emphasis on such metrics may seriously damage scientific creativity and originality. Expert

peer review should be treated as a valuable resource.

Reference

1. http://www.ascb.org/files/SFDeclarationFINAL.pdf