You work on the social impact of research. I’d like to hear how you think about this issue. Iceland’s population is about a twentieth of that of Japan. When people talk about the impact of research on society, how broad is this “society”? Is it domestic society, European society, or the world community? In Japan, if we talk about the social impact of research, most researchers imagine Japanese society.

Eiríkur Sigurðarson: First, let’s distinguish between scientific impact and social impact. Scientific impact is not necessarily only local. For example, medieval Icelandic studies and Icelandic literature research are actually global research fields. Even research on more local historical phenomena or periods can have a scholarly impact that is global.

In Iceland, it’s usually the local population that is the focus of discussions when it comes to social impact. Perhaps in some research on specific local populations, it would be the impact on smaller communities. This issue—the kind of society people have in mind when they say “social impact”—relates to the language of publications. In Iceland, as in many other places, there is a pull towards publishing in English. Usually, you get more research points at the University of Iceland if you publish in English than in Icelandic. Still, people publish a lot in Icelandic. Also, some, but not many, even write Ph.D. theses in Icelandic.

Do students aiming to be researchers tend to go overseas?

Sigurðarson: Because we have such a small population, only 850 thousand people, and a fairly small number of higher education institutes, almost every lecturer, scholar, and professor at our universities has a degree from abroad. You actually spend time there as a Ph.D. student, and even as a postdoc. So in that sense, we are very international, and we have contacts all around. It’s only recently that we’ve started hiring academics with Ph.D.s from Iceland because our Ph.D. programs are only about fifteen or twenty years old.

Research and Faculty Evaluations in Iceland: Present Situation and the Future

What is the research and faculty evaluation system like in Iceland? Such systems are a major topic of interest and discussion in Japan.

Sigurðarson: There are two systems. One is a national system that evaluates the quality of higher education in Iceland. It operates on a five-year cycle and is organized by the Icelandic Research Center, the country’s central research council. The actual quality evaluations of universities are done by an entirely international board appointed by the Ministry of Education.

Does it evaluate individual faculty members or institutions?

Sigurðarson: In the end, it’s an institutional evaluation in the sense that the quality board issues a report for the University of Iceland as a whole. But for this evaluation, faculty members have to submit a self-evaluation report. So it’s also an evaluation of individual faculty members.

In addition to the national system, there’s the evaluation of research within the university. Every scholar has to submit a report every year, detailing all publications, presentations, public engagements, and so on in the year before. Based on this, they are rated. For example, if you publish in a top-class national journal, you get five to fifteen points. If you publish in a high-impact international journal, you get twenty research points. You can get up to one hundred points for a monograph, if it’s one of the leading publishers, like Cambridge University Press or Oxford University Press. And for a presentation, like the one I gave yesterday at my workshop, I guess I get three points. (Laughs).

Happy that it was useful!

Sigurðarson: Then you get a certain number, I think the average within universities is about 35 points per year and in the humanities, we’re slightly above average.

And then you actually get a bonus payment in September for the research points you earn. So it’s a very direct incentive and strongly influences how you behave. I don’t think it’s an entirely bad system. It’s very transparent, nothing hidden there—just a list of what you can get points for. And it’s rather open, so we get a lot of points in the humanities. However, some find it difficult to actually thrive within the system, such as researchers or teachers in the languages. This is because they are basically teaching faculty. The University of Iceland was basically a teaching university, but a few decades ago, the decision was made to change it into a research university. Part of this was adopting this publishing incentive system. And it has worked. Publications have been skyrocketing, as have citations.

Of course, this system’s problems have also been discussed a lot. The system pushes researchers to do something, possibly leading them away from other activities that would be more useful to society. A problem that has been particularly discussed lately is that this system is used for decision-making in different contexts. For example, when Ph.D. students apply for funding within the university, the research points of the supervisor count. This is also used when the university is distributing funding to its faculties. A considerable amount of the funding is divided based on research activities, affecting how research is done. So now the universities are trying to scale this point system back.

With that said, in Iceland, the Association of Professors and the Association of University Lecturers sign employment contracts with the Treasury. And this evaluation system is a part of these contracts. So it is not something the university can change on its own. Scholars are very much for this system—maybe also for changing it, perhaps for scaling back how it’s used, but not for abolishing it. So I think we’re going to keep it.

The Research Network ENRESSH: Organization and Activities

You are participating in the European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH). Could you talk a bit about this network?

Sigurðarson: ENRESSH is a European network of people doing research on research evaluations, on the social impact of research, and so on. The people who lead and started the network are actually scholars in the humanities. A part of the group is practitioners like us, from research administration.

The network is divided into different work groups. So, for example, there is one work group heavily focused on bibliometrics. They are working on, for example, how to incorporate books into research evaluations. There is also a group working on the language of publications. They produce something called the Helsinki Initiative, which is about respecting domestic languages, not just English, when it comes to research publications in social science and humanities. This is all really valuable for humanities and social sciences research.

And then there is a special interest group working on young scholars. It does research from the perspective of the metrics debate and social impact debate, asking how they can get recognition to have some kind of career in the social sciences and humanities. It also asks about the career obstacles they face.

So the problems of metrics and career are closely linked. You’re part of the group researching social impact. What are its accomplishments?

Sigurðarson: We’ve collected social impact case studies from the social sciences and humanities. Most of them were from the social sciences, but a considerable bit from the humanities as well. The work has concentrated a lot on these cases. A number of papers have already been published based on them. There’s one on social impact pathways. Another compares what they call impact capacity between Eastern Europe and Western Europe. In other words, between countries where the impact debate hasn’t really taken off and countries where it has and people are used to it. Right now, the group’s doing social impact research on Nordic countries, and there are also many other papers being produced.

Is ENRESSH involved in policymaking?

Sigurðarson: The network is funded by COST, a funding scheme now within the European Framework program. So, you could say that it is a part of the European Union. And one of the objectives of this network is to produce policy papers. It networks scholars within Europe working on some particular field, collects the research they are doing, and then often compares countries in this regard. Among its output is supposed to be something that is policy relevant.

So is there some kind of coordinator that translates the research into policy?

Sigurðarson: The coordinating and mediating work is actually part of our project. The network will produce some recommendations for how to include humanities and social sciences in framework program activities and European Research funding activities, as well as suggestions on how to evaluate the contribution of humanities and social sciences. So these should be concrete and applicable recommendations, more targeted, I think, at a bit lower policymaking level. ENRESSH’s target is the people who are actually figuring out how to run the programs and evaluate research.

Monographs are a case in point. It is difficult to have them properly recognized and evaluated in the current systems. So the idea is for us to produce concrete recommendations on how to include monograph publications in both pre-funding and post-funding evaluations. As for social impact, we wrote articles based on our papers from the 2017 RESSH conference in Antwerp. In 2019, they will be published in a special section of the journal Research Evaluation. The ENRESSH policy group will submit recommendations that include their content to the commission. It will also lobby people who grant research funding and innovation funding in Europe, as well as other actors who are dealing with impact evaluations at the European level.

Is there a particular instance that comes to mind where this lobbying yielded concrete policies?

Sigurðarson: Not yet. I don’t think I can point to anything that’s already happened. But I know the ENRESSH policy group has been active in Brussels.

There’s a tendency for a group like this to focus very much on the social sciences. This is probably because most of the scholars involved in the field of research evaluation are social scientists. As a humanities scholar, you can feel slightly marginalized just by the number of things going on. Of course, though, overall it’s a very comfortable environment. (Laughs).

From Indicators and Metrics to “What is the Good We Are Doing?”

Today’s research evaluation problems appear to have come from research evaluation’s means becoming the objective. In other words, meeting research metrics has become the goal of research, and, accordingly, research inspiration is tending to atrophy. People say there’s a gap between evaluation for resource allocation, on the one hand, and evaluation for promoting quality research, on the other. What do you think should be done so that research evaluation is not an obstacle to research but instead advances it?

Sigurðarson: A major thing I’m trying to promote is properly understanding the priority order of the questions we ask about research evaluations. This is explained and analyzed very well by Jerry Z. Muller in his book The Tyranny of Metrics. So, the question goes from indicators and metrics to the question of, “What is the good we are doing?”

This is because metrics and indicators are only applicable in some cases, not all. If the debate’s priority is metrics and indicators, you risk losing out, or ignoring, many valuable aspects of research that do not fit the metrics debate. Therefore, first one should ask, “What is the good that this researcher is doing?” Then you discuss how we will evaluate the methods and quality of their work and whether we can use metrics and indicators. It would be just a waste of time to apply indicators to all cases. So our objective is to reverse the priorities.

Then, we should divide research into (1) fields where we can use some kind of metrics or tools, (2) fields where we need to use peer review, and (3) fields where we simply cannot use any formal methods. In (3), we are left with an informal policy debate on what we actually want for society. In such cases, one applies the likes of Martha Nussbaum’s capabilities approach to discuss the social impact of the research.⁽²⁾

This is the most effective method that we should use first. This debate can’t be avoided. When it comes to the value of the humanities for society, humanities scholars need to have their own voice, they need to be able to speak on their own terms.

Shifting the focus away from metrics and indicators sounds very useful for research evaluation that promotes research. Going back to the basics.

The Role of URAs in Research Evaluation

What do you think is the role of research administrators and research managers in research evaluation?

Sigurðarson: It depends a lot on people’s position and the context. Say you are a research administrator assisting people writing funding applications, and the fund is asking for cogent argument about the research’s use to society. In that case, your role might be precisely assisting them in writing or describing—and indeed just understanding—the social impact of their research. If one follows the debate regarding social impact even a little bit, one can express it using more appropriate terms.

In my experience discussing this with scholars, there is a tendency to understand the question very narrowly. When asked to describe their research’s social impact, some think, “Okay, they are asking me how I am going to benefit the economy,” or that the question is, “How is this research going to have a direct influence on policy?” Even some people say, “No, I’m researching medieval Icelandic literature, and there’s nothing in what I’m doing that’s going to change anything in the economy.” So one role of URAs would be to assist people in describing their social impact more broadly, to understand in a wider context what the social impacts might be.

If you’re working on the humanities and social sciences in an institute that is more focused on the hard sciences in general, then a role would be to keep arguing, at the institute’s policy level, for the social good produced by the social sciences and humanities. Talking about what would be lost by your institute if we stopped or reduced funding or supporting the humanities.

I have been working a lot with Ph.D. students in the humanities in Iceland. If working with young researchers, that is, early career researchers, is part of your remit, your role would also be helping them develop ideas and formulate arguments about what they’re doing. Maybe pointing to some questions that they might ask themselves about their research, that they may not have thought of themselves, as well.

Do you think that using the social impact approach is effective even for young researchers moving on to something other than research? In other words, when they have to articulate what they have been doing up through the present.

Sigurðarson: Yes. One can ask, “What is the social benefit of you as an individual getting a research education, writing a thesis, and developing yourself as a researcher?” If they gain skills that they can use in other contexts, it would be a loss to society if these weren’t developed. That personal development, that kind of capacity building of yourself as an individual and the people around you, is beneficial to society.