Publishing is changing and evaluation along with it
Professor Outi Merisalo has witnessed major changes in scholarly publishing over the past few decades. Digital publishing, for example, has improved access to research. At the same time, competition among researchers has intensified and the pressure to publish has increased.
– Publication pressure has led to an increase in the number of publications and publication channels. This is directly reflected in the panels’ work: there is more to evaluate than before, and gaining an overall understanding of the publishing landscape has become increasingly demanding, Merisalo explains.
Merisalo also points to the growing prevalence of formal peer review as a significant development. Clear evaluation criteria and the requirement for impartial assessment have strengthened the quality of scholarly publishing. Nevertheless, the system also faces challenges.
– It can be difficult to find knowledgeable reviewers who are willing to commit to the task. A shortage of reviewers may significantly prolong publication processes, Merisalo notes.
Reliable evaluation requires looking beneath the surface
According to Merisalo, the most challenging aspect of panel work is obtaining reliable information about how publication channels operate in practice. Today, their websites provide more information than before about matters such as peer review, editorial boards and publishing policies. However, an evaluation cannot be based solely on a publication channel’s own description of its activities.
– In addition to websites, we examine texts published by the channel and draw on the experiences of researchers who have published in it. With experience, you learn to identify the information and details that help form a reliable overall picture of how a publication channel operates, Merisalo says.
Publication channels that do not openly disclose their operating principles are particularly difficult to evaluate. It is also difficult to assess the quality and development of a newly established channel based on only a few years of activity.
– To qualify for Publication Forum Level 1, a publication channel must have sufficiently established operations, and enough information about its practices must be available, Merisalo states.
English is not the only language of science and scholarship
Merisalo emphasises that evaluating publication channels requires an understanding of the publishing cultures of different academic disciplines. In research on Romance and classical languages, for example, German, French, Spanish and Italian play an important role in scholarly communication.
– The significance of a publication channel cannot be assessed solely on the basis of English-language publishing. Evaluations must take into account the languages used in different fields and their importance to the academic community, Merisalo stresses. The expertise of researchers from different fields is essential for recognising precisely these kinds of discipline-specific characteristics.
Artificial intelligence raises new questions for evaluation
The Publication Forum’s evaluation criteria have been revised several times. According to Merisalo, the current criteria work well overall. They cover factors including peer review, research integrity, the international reach and accessibility of a publication channel, and indicators of predatory publishing. The next challenge is to address questions concerning the use of artificial intelligence.
– We do not yet know enough about the effects of artificial intelligence. We need new ways to identify and limit the potential harms associated with its use, Merisalo says.
Evaluation must remain in the hands of the research community
Merisalo considers the Publication Forum’s reliance on the expertise of the research community to be one of its key strengths. The evaluation process combines researchers’ qualitative expertise with quantitative data on publication channels. The panel secretariat supports the evaluation process, enabling panel members to focus on the content, quality and practices of publication channels.
– The expertise of the research community will continue to be needed as publishing evolves. The Publication Forum has been able to respond to developments such as the rise of predatory publication channels and to support the development of responsible evaluation practices. Researcher-led evaluation will therefore continue to play an important role in the future, Merisalo emphasises.