Extracted main text — title through conclusion, appendix excluded. This is what our citation measures are computed over, published so the extraction can be checked by eye.
70,681 characters · 28 sections · 65 citation commands
A Supervised Machine Learning Approach for Assessing Grant Peer Review Reports
\definecolor{snsfblue}{HTML}{3D7D9F} \definecolor{snsfgreen}{HTML}{71B294} \definecolor{snsfred}{HTML}{D9534F}
\newgeometry{left=3.4cm, right=3.4cm, top=15mm,bottom=25mm}
\thispagestyle{empty}
\restoregeometry
\doublespacing
\setcounter{page}{1}
Research funders rely on external peer review to evaluate proposals for scientific merit, methodological rigor, feasibility, and potential impact. Despite the central role of peer review in grant evaluation and funding decisions, analyses of peer review reports are rare and challenging. Science has become more open, with journals increasingly publishing peer review reports and authors' responses alongside the articles. In contrast, grant peer review reports are confidential to allow reviewers to comment freely and to protect ideas presented in grant applications. Interestingly, a recent study suggested that "researchers trust grant reviewers far less than they trust journal peer reviewers..." langfeldt24, likely due to a perception that the grant evaluation process lacks transparency.
Analyzing the content of grant peer review reports could provide several benefits, including an objective assessment of review quality to gauge adherence to guidelines, ensuring thoroughness, fairness, and consistency in panel discussions, and improving decision-making. Such analyses may enhance the transparency of evaluation processes and identify areas for improvement in reviewer training. Prior work has investigated potential biases in peer review based on the gender and racial disparities of authors and reviewers erosheva2020nih,squazzoni2021peer. Various methodologies have emerged in recent years for analyzing peer-review reports, focusing on content, sentiment, and argument structures. Early studies primarily used linguistic and sentiment analysis to explore the relationship between reviewer language and review scores, offering foundational insights luo22. While straightforward, these methods struggle to capture more nuanced information. More advanced approaches combine machine learning with qualitative techniques, such as clustering analysis to assess alignment with evaluation criteria hren2022. Recently, machine learning models like random forests and transformers have been applied for more detailed classification of peer review texts williams23, severin23, kuznetsov2024nlp. A key development is the increasing use of argument detection models, which leverage deep learning to classify and segment argumentative propositions within peer reviews Fromm2021, Lawrence2020. The latest innovations focus on Large Language Models (LLMs), such as GPT-4 or Llama, for generating labels and automating quality assessments, significantly improving efficiency and accuracy Pelaez2024.
In this study, we integrate human annotation with transformer-based machine learning to create a tool for evaluating the characteristics of grant peer review reports submitted to the Swiss National Science Foundation (SNSF) between 2016 and 2024. We present each step in detail, report initial findings, and discuss the potential of machine learning as a tool for gaining insights into the grant peer review process. The code scripts for the conducted analyses are publicly available on GitHub\footnote{\url{https://github.com/snsf-data/ml-peer-review-analysis}.} and the fine-tuned models are shared on the Hugging Face Hub\footnote{\url{https://huggingface.co/snsf-data}.} to enable others to benefit from our work.
The approach involves three key steps: (1) human annotation; (2) machine learning and method selection; (3) and validation and robustness. The steps are summarized in Figure (ref) and outlined below. The training of the machine learning models was performed locally without access to the internet to prevent any potential data leakage or network interference.
The study relies on a corpus of 47,522 external peer review reports written in English for 13,653 applications submitted to the SNSF's Project Funding scheme between October 2016 and April 2024. Project Funding is the largest SNSF funding scheme, open to research proposals from all academic disciplines. In 2023, the SNSF funded 2,928 grants with CHF 1.2 billion, of which 1,009 grants (CHF 580 million) were in the Project Funding scheme.\footnote{See the SNSF's DataPortal for more details: \url{https://data.snf.ch/key-figures/funding-instruments?s5=0&s2=1}} The SNSF evaluation procedure consists of several evaluation steps. The peer review reports analyzed in this study constitute the “External review,” which is the first part of the SNSF's scientific evaluation process (see Figure (ref) in Appendix for details). At least two external experts review research proposals against a set of evaluation criteria specified by the SNSF: (1) Scientific relevance, originality and topicality, (2) Suitability of methods and feasibility, (3) Applicant’s scientific track record and expertise. Proposals tagged as `use-inspired' by the applicants are assessed by taking into account their broader impact as an evaluation sub-criterion: (1a) Broader impact. External reviewers provide their assessment in text boxes for each criterion and an overall comment, which all address the proposal's specific strengths and weaknesses, resulting in up to 5 distinct text comments in total. (see Figure (ref) in Appendix for details on the evaluation criteria).
Alongside the structured written report, external reviewers rate the proposals on a numeric scale. The SNSF evaluation panels then discuss and rate the proposals taking into account the external peer review reports. Funding decisions are based on the panel's ranking of proposals heyard2022.
We cleaned and anonymized the raw texts of the review reports. This included removing redundant characters such as HTML code and the names of applicants, co-applicants and project partners. The data management plan underlying this work provides a detailed description of data protection.\footnote{ \url{https://doi.org/10.46446/DMP-peer-review-assessment-ML}} We limited the analysis to reviews submitted in English. Thus, our analysis does not include 10.2% of the originally submitted reviews conducted in different languages (mostly German and French), all of which come from the humanities and social sciences.
We conducted eleven distinct annotation rounds, during which we developed and refined a comprehensive annotation codebook with instructions.\footnote{ \url{https://doi.org/10.46446/Codebook-peer-review-assessment-ML}} The final version of the codebook includes twelve distinct categories, each relevant to the criteria specified by the SNSF. These categories are applied at the sentence level of each review and are grouped into three dimensions: 1) Evaluation Criteria, 2) Focus, and 3) Statement Type and Reasoning.
The dimension Evaluation Criteria examines to what extent the reviewer applied the criteria outlined above. Focus captures the aspect the reviewer is focusing on: the applicant (quantitative and general comments), the proposal (such as the project plan), or the methodological framework. Statement Type and Reasoning evaluates whether the statement is positive or negative, whether it is substantiated with evidence, and whether the reviewer offers suggestions for improvement. Table (ref) provides a detailed overview of the categories.
Each annotation round relied on randomly sampled sentences from the entire text corpus. Table (ref) in the Appendix provides a detailed overview of each annotation round. The sampled sentences are representative of the three main research domains.\footnote{See the SNSF discipline list: \url{https://www.snf.ch/SiteCollectionDocuments/allg_disziplinenliste.pdf}.} 38.3% of randomly sampled review sentences are from proposals submitted to the research domain of Mathematics, Information Technology, Natural- and Engineering Sciences (MINT). A similar share of review sentences (36.9%) comes from a review submitted to the Life Sciences (LS). The remaining 24.7% of review sentences are based on proposals in the Social Sciences and Humanities (SSH). These percentages correspond to the distribution of reviews across disciplines in the full-text corpus.
The human annotators classified each sentence across the twelve categories. The categories are not mutually exclusive and were annotated independently of each other. The only exception is Rationale, which was only considered if the annotator first identified a positive or a negative statement.
The linguistic backgrounds of the annotators, two native English speakers and two non-native English speakers, reflect the context in which the data is created: while the language of the peer review reports is English, they are written and assessed by both native and non-native English speakers. The annotation codebook, with clear instructions and regular inter-coder reliability tests, ensured consistency and rigor throughout the process (see Table (ref)).
The final annotation set comprised 3,000 sentences. The first 2,000 sentences were sampled randomly from the entire corpus of English reviews. Stratified sampling was used for the remaining 1,000 sentences. We randomly selected 200 sentences from each of the five text boxes reflecting the SNSF evaluation criteria (see Figure (ref)) to obtain more sentences for under-represented categories, such as Applicant: Quantity.
To validate annotation, we randomly assigned each sentence to three of the four annotators. Their annotations were aggregated based on majority agreement: if at least two of three annotators identified a particular characteristic in a sentence, it was classified as an example of that category. If only one or no annotator identified the characteristic, the sentence was classified as not belonging to that category. This consensus-based method reduces annotator bias benoit16, improving the reliability of the annotated dataset.
Further, a comprehensive analysis of agreements and disagreements between annotators was conducted after each annotation round. This process facilitated a nuanced understanding of annotation discrepancies, enabling continuous refinement of the coding guidelines and improvement of annotation consistency. Appendix (ref) lists five example sentences from each category with full agreement among annotators, while Appendix (ref) provides an overview of the descriptive statistics for the sample of annotated sentences.
Next, we used the human-annotated sample of 3,000 sentences to train machine learning models that predict the category labels for the remaining review sentences in our dataset and for future incoming reviews. For this purpose, we relied on a pre-trained language model vaswani17, tunstall22 and fine-tuned it for the text classification task based on our annotated review sentences.
We focus on the so-called BERT models (Bidirectional Encoder Representations from Transformers) developed at Google devlin18. The BERT model is pre-trained using masked language modeling and next-sentence prediction based on a large text corpus. It learns via bidirectional representations, conditioning on both left and right context in the text sequence in all layers of the model devlin18. Due to its strong performance and efficiency, BERT has become a foundational model for many NLP tasks, including text classification acheampong2021transformer. The text corpus consists of the BookCorpus zhu15 and English-language Wikipedia. Variations of the original BERT model modify the pre-training by improving the optimization or the training data. Given the scientific nature of our dataset, which consists of peer review reports, we follow the developments of SciBERT, beltagy19, a version of BERT pre-trained on scientific texts from SemanticScholar. SciBERT has been further enhanced by citation graphs, leading to the SPECTER model cohan20. Specifically, we use SPECTER2 singh22, an updated version of SPECTER, which outperforms the original BERT model on various NLP tasks in the scientific domain wolff2024enriched, Okasa2024. We deploy the SPECTER2 model available on Hugging Face\footnote{\url{https://huggingface.co/allenai/specter2_base}.} via the transformers library in Python wolf20. To test the effectiveness of SPECTER2, we also fine-tuned the original BERT model as a baseline, along with RoBERTa, a more robustly optimized version of BERT Liuetal2019.
The pre-trained SPECTER2 model can be used as-is to extract text representations via the CLS token cohan20. We not only extracted embeddings through the CLS token for input into a classifier, but also fine-tuned SPECTER2 on our classification task using the dataset of 3,000 annotated sentences.\footnote{We perform the fine-tuning using a GPU unit with 4 GB of memory (NVIDIA RTX A2000).}
Formally, consider $\{(X_i,y_i)\}_{i=1}^n$ to be an i.i.d. random sample of size $n$ with $X_i$ being the $i$-th text sequence (sentence) and $y_i \in \{0,1\}$ being the corresponding outcome (category) with binary class labels. We are interested in predicting the conditional probability of the outcome given the input text sequence, i.e. $P[y_i=1 \mid X_i=x]$. We encode each input sequence $X_i$ into a vector representation via the text embedding function of SPECTER2, $\mu(\cdot)$, as follows $$h_i=\mu(X_i;\theta)$$
where $h_i$ is the contextualized representation for the text sequence $X_i$ based on the CLS token and $\theta$ being the model parameters. We then feed the embeddings into a fully connected linear output layer with weights $W$ and bias $b$ to get the output scores, i.e. the logits, and apply a sigmoid activation function, $\sigma(\cdot)$, to obtain the class probabilities as $$\hat{y}_i=\sigma(W\cdot h_i + b)$$
with $\hat{y}_{i} = \hat{P}[y_i=1 \mid X_i=x]$ being the predicted class probability for input $X_i$. The prediction of the class label for a text sequence $X_i$ is then obtained via thresholding as $$\hat{y}_i^{\mathcal{C}} = \mathbb{I}(\hat{y}_i \geq 0.5 )$$
with $\mathbb{I}$ being an indicator function. In the course of the fine-tuning, the optimal values for parameters $\theta$, $w$ and $b$ are determined by optimizing the cross-entropy loss $$\mathcal{L}(\theta, w, b) = -\frac{1}{n}\sum_{i=1}^n \big( y_i \cdot \log(\hat{y}_i) + (1-y_i)\cdot \log(1-\hat{y}_i)\big).$$
As such, the parameters of the linear output layer, $W$ and $b$, are trained from scratch, whereas the parameters of the pre-trained SPECTER2 model, $\theta$, are fine-tuned.
Such fine-tuning requires selecting an optimizer and its hyperparameters to guide the learning process. We used the Adam optimizer with decoupled weight decay (AdamW), which improves generalization and offers computational advantages over stochastic gradient descent with momentum loshchilov17. We follow recommendations from devlin18, using a learning rate of 2e-5, weight decay of 0.01, 3 epochs, and a batch size of 10. Table (ref) in the appendix provides a detailed overview of the fine-tuning hyperparameters.
Several methods are available for classifying texts into various categories minaee2021deep. To identify the most effective approach, we empirically compared three approaches: binary, multi-task, and multi-label classification.
The binary classification approach divides the task into 12 separate binary classification problems, with each model dedicated to a specific category. As a result, we fine-tune 12 separate models, one for a each outcome category. While this method is computationally intensive due to the need to fine-tune multiple models, it allows for category-specific adaptation, tailoring the model to the particular characteristics of each category, including the distribution of outcome labels. However, this approach does not account for potential correlations between the categories during estimation sun2019.
Multi-task classification makes use of multi-task learning, where we keep the shared encoder model and add category-specific adapters for binary classifications. The encoder parameters $\theta$ are shared across the categories, and only the last hidden layer $h_i$ together with the output layer parameters $W$ and $b$ are optimized for the binary classification of the given category via an adapter sun2019. The computational demand is, however, reduced substantially compared to binary classification, as only one encoder model needs to be fine-tuned. On the downside, the full encoder parameters are not optimized for each given category.
Multi-label classification frames the problem in a way that naturally fits our data, where a single review sentence can belong to multiple non-exclusive categories, i.e. $y_i = (y_{i,1},\dots, y_{i,C})$ is a binary vector with $y_{i,c}\in \{0,1\}$ for $C=12$ categories. In this approach, we fine-tune the model such that the number of output nodes matches the number of categories to predict Foster2024. The key advantage is computational efficiency as only one model needs to be fine-tuned. In addition, the correlation among the categories is taken into account. In contrast, the problem of high imbalance in the labels becomes aggravated since many category combinations appear infrequently in the training data.\footnote{The annotated data contains 177 unique combinations of categories, of which 63 appear only once.}
To evaluate the classification performance, we randomly divided the annotated set of 3,000 sentences into training (2,500) and test (500) sets while stratifying by the outcome category.\footnote{Due to the high class imbalance the stratification for the multi-label classification was not feasible.} We used the training set of 2,500 sentences to fine-tune the model and the untouched test set of 500 sentences to evaluate the prediction accuracy. To ensure a robust evaluation not dependent on a single data split, we performed five-fold cross-validation, stratified by outcome category, within the training set. Thus, we fine-tuned the model on four folds fo size 2,000 and evaluated it on the remaining fold of size 500, such that each fold served as a validation set once. We report the average performance metrics over the five folds (as recommended by hastie09).
We calculated the following performance metrics: accuracy, precision, recall, and the F1 score rainio2024. The accuracy is defined as the proportion of correctly classified texts: $$\text{Accuracy}= \frac{1}{n} \sum_{i=1}^n \mathbb{I}(\hat{y}_i^{\mathcal{C}} = y_i) ]$$
while precision and recall measure the proportion of correctly classified texts of a given category out of all classified texts of the given category, and out of all actual texts of the given category, respectively: $$\text{Precision}= \frac{\sum_{i=1}^n \mathbb{I}(y_i=1 \wedge \hat{y}_i^{\mathcal{C}}=1)}{\sum_{i=1}^n \mathbb{I}( \hat{y}_i^{\mathcal{C}}=1)} \quad \text{and} \quad \text{Recall}= \frac{\sum_{i=1}^n \mathbb{I}(y_i=1 \wedge \hat{y}_i^{\mathcal{C}}=1)}{\sum_{i=1}^n \mathbb{I}(y_i=1)}.$$
Our preferred evaluation measure is the F1 score, which is the harmonic mean of precision and recall: $$\text{F1}= \frac{2 \cdot \text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}$$
and takes the class distribution into account, i.e. the prevalence of the categories. Uneven proportions in the class labels pose a challenge for any classifier, as there are only a few examples to learn from (the class imbalance problem johnson2019). Therefore, we not only evaluate the metrics for both labels combined, the so-called micro-average, but also separately per label, i.e. in the subsets of $y_i=0$ and $y_i=1$, and compute the average of label-specific metrics, the so-called macro-average. While micro-average is dominated by the more prevalent class label, macro-average weights both class labels equally and, as such, better reflects the overall classification quality, particularly in cases with high class imbalance. manning1999foundations.
Sentences could be assigned to no category or one or more categories. When aggregating annotations based on majority agreement, 18.1% of sentences were not assigned to any of the twelve categories; 19.8% fell into a single category, and most sentences fell into two (22.9%) or three (22.7%) categories, respectively. 16.4% of the sample was classified into four or more categories (see Figure (ref) in Appendix for details).
For each of the 12 categories, we first assessed the percentage of sentences where all three coders assigned the same label. Agreement ranged from 64% (Proposal) to 98% (Applicant: Quantity), with an average of about 80% agreement across all categories. Figure (ref) provides a more detailed assessment of coding agreement and the prevalence of each category, revealing a considerable variation among them. Categories reflecting the Evaluation Criterion aspect had prevalence rates ranging from 6% to 8% for Suitability and Feasibility, respectively, and from 17% to 18% for Track Record and \textit{Relevance, Originality, and Topicality}. For the categories addressing the \textit{Focus} aspect, a high prevalence was observed for \textit{Proposal} (41%) \textit{Method} (16%), and the \textit{Applicant} (20%) in general, not considering the quantitative aspects. In contrast, only 1.6% of the sentences mentions explicitly the quantitative aspects of the applicant (\textit{Applicant: Quantity}), such as the number of publications. The prevalence for the \textit{Statement Type and Reasoning} categories also varied substantially. The \textit{Positive} references are more prevalent (37.1%) than \textit{Negative} ones (15.2%), while not all of them provide a \textit{Rationale} for the statement.\footnote{The annotators identified a \textit{Rationale} only if they marked the sentence as either \textit{Positive} or \textit{Negative}. When considering only the sentences classified as \textit{Positive} or \textit{Negative} by at least two annotators, the proportion of sentences with a \textit{Rationale} (also agreed upon by at least two annotators) rises from 18.4% to 35.5%. Further details can be found in Appendix (ref) and Figure (ref).} Only 4.5% of the 3,000 sentences provided suggestions.
We relied on the SPECTER2 model, pre-trained on scientific texts and augmented by citation graph, which we fine-tuned based on our set of annotated sentences. Table (ref) compares the performance in terms of the F1 score for binary, multi-label and multi-task classification based on the test set of 500 review sentences.\footnote{Appendix contains detailed performance metrics for the three classification approaches (see Tables (ref), (ref), and (ref)). We report the accuracy, balanced accuracy, class-specific F1 scores, macro-averaged F1 scores, micro-averaged F1 scores, precision, recall, and class-specific precision and recall. The main results always refer to macro-averaged F1 scores, which are not weighted by the class shares and thus provide a measure of classification performance that weights both class labels equally, as opposed to micro-averaged F1 scores.} Fine-tuning with separate binary classifiers provided the best classification performance across the twelve categories. The macro F1 scores for the binary classifiers range from 0.71 (Rationale) to 0.93 (Applicant: Quantity), with an average of 0.85 across all twelve classifiers. The average F1 score for the multi-label approach was lower (0.73) and the range of F1 scores much larger (0.48 for Feasibility; 0.90 for Track Record and Applicant). With an average F1 score of 0.62, the multi-task approach failed to produce reliable predictions. The performance of the models that share the information among the categories appears to suffer from the high-class imbalances in the category labels.
The fine-tuning of the models above relied on a single training-test split, as suggested by tunstall22. In this case, the test set refers to a subset of annotated data reserved for evaluating the fine-tuned model's classification performance, ensuring an unbiased assessment of the model's performance. To assess whether the allocation of sentences into the test and training set might affect the overall conclusions, we conducted a five-fold cross-validation. Figure (ref) compares the macro F1 score metrics for the five-fold cross-validation for the binary, multi-label, and multi-task classification. The dots show the range of the F1 scores across the five folds; the number above each range depicts the average F1 score across all folds. Mirroring the results from Table (ref), Figure (ref) reveals that the binary classification approach outperforms the multi-label and multi-task classifiers. The average macro F1 score across all categories for the binary classifiers was 0.83, while the average F1 scores for multi-label (0.71) and multi-task (0.60) are lower. Moreover, the variation of F1 scores across categories is considerably smaller for the binary approach. The five-fold cross-validation further supports that optimal performance is achieved through binary classifiers specifically fine-tuned for each category.
Overall, using separate binary classifiers for each category appears to be the most effective approach severin23, despite the greater computational efficiency of multi-label and multi-task methods. Therefore, we proceed with our analyses using the binary classification approach.
Following the binary classification approach, we further assessed the suitability of the underlying pre-trained model and its impact on classification performance. Therefore, alongside SPECTER2, we also repeated fine-tuning for the baseline BERT model and the robustly optimized RoBERTa model.
The results reveal similar performances of BERT and RoBERTa compared to the SPECTER2 model, both in terms of the test and cross-validation accuracy (see Table (ref) and Figure (ref)). Indeed, the average F1 score across all categories is 0.85 for all considered models in the test set. This result suggests that the annotated data used for fine-tuning is the decisive element rather than the model itself. This is also supported by the accuracy results provided in Table (ref), which shows that almost all F1 scores were close to or above 0.8 except for the Rationale category, which is inherently difficult to classify, even for human annotators. These accuracy levels exceed those found in related studies classifying peer review texts, despite much larger annotated sets ghosal22.
We examined variation in performance across classification methods and categories by relating performance metrics to the prevalence of each category in the training set and the findings from a survey distributed to the annotators. Figure (ref) shows the relationship between the prevalence of categories (in %) and the F1 Score for the three classifiers based on Pearson's correlation coefficients. There were moderate to strong positive correlations between the F1 Score and the prevalence of the category for the multi-label and multi-task classifiers, but not for the binary classifier. This supports choosing the binary classifier, as its performance is unrelated to category prevalence or label imbalance.
We distributed a short questionnaire to the four annotators during the annotation process. It assessed annotators' perceived difficulty of identifying each of the twelve categories to better understand the annotators' confidence in their decisions han2023expert.\footnote{The survey started with the following statement: “Reflecting on the previous coding rounds, we would like you to assess the ease or difficulty you experienced in classifying the categories featured in our coding tasks. Your personal perspective is highly valued, and we encourage you to share your honest opinion.”} For each category, the survey asked: “On a scale ranging from 1 (very straightforward) to 10 (very challenging), how easy did you find classifying \dots” We collected the views of all four annotators and calculated the average score.
The survey reveals substantive differences in the perceived difficulty in identifying the categories. It ranged from 1.75 for Track Record (fairly straightforward) to 7.25 for Rationale (challenging). The average difficulty was 3.73; the median 3.50. Apart from Rationale, the difficulty of all other categories was below 4.75. Assessing Rationale was the most challenging task.
Figure (ref) relates the perceived difficulty of annotating each category to the F1 Score (left panel) and the prevalence of the categories (right panel). Categories that annotators found easier to classify have higher F1 scores ($r=-0.73$). For instance, Rationale was the most difficult to annotate and had the lowest F1 score, whereas the applicant's Track Record was straightforward for annotators and classifiers. Overall, categories that are easier to annotate align with better classification performance. In contrast, the prevalence of a category only weakly correlated with the perceived difficulty ($r=0.20$, right panel of Figure (ref), indicating that annotators did not perceive infrequent categories (such as Applicant: Quantity or Suggestion) as more difficult than frequent categories, such as Positive or Proposal.
We assessed the face validity of the classifiers' performance via explainable machine learning molnar20. Following severin23, we conducted a keyness analysis to identify terms that predict each category.
We applied the twelve binary classifiers to the full set of review reports consisting of 1,612,405 sentences. For each category, we compared words and phrases in sentences where the relevant review characteristic was present against sentences without it, based on model predictions. Multi-word expressions were automatically identified through collocation analysis. A keyness analysis then used chi-squared tests to evaluate each term and phrase between the two document groups benoit18. Table (ref) displayed the 25 terms and multi-word expressions with the highest keyness values for each category.
The keyness analysis identifies specific terms associated with each category, providing insight into the content. Common terms in sentences classified under the Proposal category include project, proposal, topic, and adjectives like timely, ambitious, and \texttt{topical}. Predictive terms for the \textit{Positive} category include \texttt{expertise}, \texttt{excellent}, \texttt{strong}, \texttt{timely}, and \texttt{outstanding}, while \texttt{lack}, \texttt{unclear}, \texttt{weakness}, and \texttt{concern} are common in the \textit{Negative} category. The term \texttt{unk}, a placeholder for the applicant and co-applicant names to maintain anonymity is the most predictive term for the \textit{Applicant} and \textit{Track Record} categories. Additionally, terms such as \texttt{published}, \texttt{h-index}, \texttt{citations}, \texttt{first_author}, \texttt{google_scholar}, and \texttt{scopus} show high keyness values for \textit{Applicant: Quantity}.
Further, we compared the prevalence of categories in the annotated sample of 3,000 sentences with the predicted prevalence in Project Funding review sentences from 2016 to 2024. A reliable classification should show correspondence between these aggregated percentages muellerfujimura. Figure (ref) confirms that aggregated predictions for 1,612,405 review sentences align with the proportions in the annotated sample of 3,000 sentences ($r=0.99$).
In order to gain substantive insights into the content of the grant peer review reports, we compare the prevalence of each category at the level of reviews submitted between 2016 and 2024. For each review and category, we measure prevalence by dividing the number of sentences classified as belonging to the given category by the total number of review sentences. A value of 30% implies that three out of ten sentences address a given category. As sentences often contain more than one category, the prevalence across the 12 categories can exceed 100%.
Figure (ref) reveals considerable variation across reviews and categories. In the Evaluation Criterion dimension, the Track Record of an applicant is most prevalent (19.6% on average across all reviews), followed by Relevance, Originality, Topicality (17.5%), whereas Suitability (8.5%) and Feasibility (6.6%) are less prevalent. The highest Focus of the reviews is on the \textit{Proposal} (39.3%), while the focus on methods is lower (16.3%). More sentences mention the \textit{Applicant} in general (20.8%) in comparison to small percentage addressing quantity-related aspects of the applicant (1.1%). Turning to the \textit{Statement Type and Reasoning}, we observe that \textit{Positive} sentences outweigh \textit{Negative} sentences (42.4% vs. 17.2%). The average review provides a \textit{Rationale} in 24.4% of the sentences that are classified as positive or negative.\footnote{Since \textit{Rationale} is only considered for \textit{Positive} or \textit{Negative} statements in the annotation, we aggregate the predictions for \textit{Rationale} category as follows: if a sentence's prediction is either \textit{Positive} or \textit{Negative}, we retain its \textit{Rationale} prediction. If the sentence's prediction is neither \textit{Positive} nor \textit{Negative}, we disregard its \textit{Rationale} prediction. Subsequently, we calculate the review-level prevalence by considering only the sentences classified as \textit{Positive} or \textit{Negative} in the denominator of the average.} \textit{Suggestions} are infrequent in grant peer review reports (4.1%).
While the averages provide a measure of central tendency, the histograms reveal considerable variation across the reports. For example, while the prevalence of Positive sentences is symmetrically distributed around the mean, Negative sentences are skewed, with 10,157 reviews containing fewer than 5% of negative sentences, while only 946 reviews include more than 50% of negative review sentences.
We performed several robustness tests and validation analyses.
During annotation, it is crucial to determine the amount of labeled data needed to train classifiers with sufficient accuracy. Since estimating the minimum required annotated texts is challenging a priori, we use an approximation method -- a so-called ablation study -- to assess the added value of additional annotated samples for classification Foster2024, bucher2024. We split the annotated sample of 3,000 sentences into a training set of 2,500 sentences and a test set of 500. We further divide the training set into chunks of 500 sentences. Then, we perform sequential fine-tuning on the pre-trained SPECTER2 model, gradually adding chunks of 500 sentences to the training set, up to 2,500 sentences. At each step, we evaluate classification accuracy on the test set.
Figure (ref) reveals two main findings. First, for well-balanced categories like Positive and those with moderately imbalanced classes like Applicant, high classification accuracy was achieved even with a small sample of annotated data, and additional data only marginally increased accuracy. Second, adding training sentences substantially improved classification accuracy for more imbalanced categories, such as Feasibility or Suggestion. Since the greatest accuracy improvements came from the highly imbalanced categories, additional training data is particularly valuable for underrepresented classes. However, even for these imbalanced classes, accuracy gains become minimal once around 2,000 annotated sentences are used Foster2024, bucher2024. Given this evidence and the cost of obtaining further training data, we stopped additional annotations at 2,500 sentences in the training set.
Although the sentence level is a common unit of analysis for text classification ghosal22,severin23, we also explored whether providing a larger context would improve classification accuracy. Specifically, we focused on Rationale, as its classification accuracy was significantly lower than that of other categories and often required more than one sentence to convey its full meaning. We additionally annotated Rationale to capture if the rationale is provided in the surrounding sentences, i.e., one sentence before and after the target sentence. We then defined a new category, Rationale and Context, combining annotations for the target sentence with those of surrounding sentences (see Figure (ref) in Appendix (ref) for the prevalence statistics). Finally, we repeated the fine-tuning step for this category by providing the target sentence and the surrounding sentences as input to the transformer model.
The classification accuracies were comparable with an F1 score of 0.71 for both Rationale and Rationale and Context (Table (ref) in Appendix (ref)). Figure (ref) provides further evidence based on five-fold cross-validation, suggesting that the model fine-tuned with only the target sentence tends to outperform the context-inclusive model. Adding surrounding sentences does not improve the classification accuracy for Rationale.
To evaluate how the composition of training data affected classification performance, we compared classifiers trained on the complete set of sentences with majority annotator agreement to those trained exclusively on sentences with unanimous agreement. Classifiers trained on full-agreement data performed slightly better in categories with a lower inter-coder agreement, such as Proposal and Positive, while categories like Applicant: Quantity and Feasibility saw a notable decline in accuracy (Table (ref) in Appendix (ref)). To further examine this, we conducted a cross-validation exercise where training folds included only full-agreement sentences, while validation folds covered all 500 sentences. The results indicated that restricting the training set to full-agreement sentences did not significantly improve overall classification performance. Therefore, using a larger training set based on majority voting seems advantageous in balancing training data quality and quantity.
Using few-shot learning, Large Language Models (LLMs) can be employed to annotate additional data, which are then used to fine-tune smaller transformer models Pelaez2024,Gupta2024, or directly used for text classification brown2020language. We explored the potential of generating additional training data using the Llama model touvron2023llama, in particular the Meta-Llama-3-8B-Instruct version llama3modelcard. Specifically, we tested the model's ability to classify the twelve categories by developing tailored prompts based on the Hugging Face chat template.\footnote{\url{https://huggingface.co/docs/transformers/main/chat_templating}.} As part of the prompt, we provided one example of a sentence representing a given category and one example of a sentence not representing any category. A detailed prompt structure is depicted in Figure (ref) in Appendix.
We evaluated the classification performance based on the same set of 500 sentences used in the test set in the main classification analysis. The LLM performed poorly, particularly in classifying sentences for imbalanced categories, leading to low F1 scores. The results suggest that LLM annotation does not achieve the accuracy of human annotation and lags behind the accuracy of fine-tuned BERT models (see Table (ref)) as has been previously documented by sun2023text or bucher2024. Consequently, we neither include artificially generated data in our models nor use LLMs for text classification directly.
Peer review is a fundamental component of the research lifecycle, serving as a quality control process where experts evaluate submissions for publication, promotion or funding. Although peer review is a central element in evaluating grant proposals, research into the content of grant peer review reports is scarce. Such research could examine the quality of submitted reports and thus address the decreasing trust in the review process langfeldt24, promote fairness and consistency in panel discussions, increase process transparency, and identify areas for reviewer training.
The tools used for such tasks must be carefully developed, tested and transparently described. In this study, we have presented a pipeline for developing classifiers to aid in analyzing the contents of grant peer review reports. Our results show that machine learning can contribute to the assessment of review texts, but some characteristics are analyzed more reliably than others. In addition to the fine-tuned classifiers, which we share open-source, the step-wise procedure described in this paper may enable other groups to develop their own peer review text classification approaches.
Assessment of peer review reports using machine learning methods has gained considerable attention in recent years sizo19. Building on existing work Hua2019, Fromm2021, ghosal22, hren2022, GuoetAl2023, severin23, williams23, we established and validated a classification pipeline by defining review characteristics relevant to funding organizations, creating an annotation codebook, manually annotating a set of review sentences, and fine-tuning pre-trained transformer models, thus enabling the classification of reviews at scale. Furthermore, we compared various methods and approaches to test the robustness of our findings. Our approach offers a reliable framework for systematically assessing and potentially improving peer review content by employing a rigorous annotation process and transformer-based classifiers.
Developing an effective annotation codebook is a critical process, requiring careful refinement to ensure accuracy in identifying key characteristics of peer review reports in line with the funder's requirements hug20. Annotators in this study were academics familiar with the peer review process, though not domain experts. It has been observed that instructed non-expert annotators typically adhere more closely to the guidelines in the annotation codebook and demonstrate performance comparable to that of expert annotators snow2008cheap. Annotators must navigate technical jargon, implicit meanings, and vague and unclear texts. Working in a team of annotators can partly overcome these challenges. The quality of human annotations is crucial, as machine learning classifiers can only perform as well as the data they are trained on han2023expert. When human annotators find specific categories challenging to classify, machine learning models, even advanced ones, will also struggle.
As reflected in the lower inter-coder agreement, categories that proved inherently difficult to annotate include categories like Proposal and Rationale. Narrow and clearly defined categories, such as Applicant: Quantity and Suggestion, showed high levels of agreement among annotators. In contrast, broader categories allow more room for interpretation. The category Proposal is particularly broad, encompassing sentences that may address various sections of the grant proposal, including the research plan, descriptions of the proposed topic, or the literature review. Annotators also reported difficulties distinguishing this category from references made to proposed methods, especially in the context of social sciences and humanities.
For the category Rationale, disagreements often arose from differing interpretations of what constitutes a satisfactory justification for a positive or negative statement. This subjective aspect was unavoidable and may reflect similar challenges that reviewers may encounter when articulating their reasons for a supporting or a critical statement. Another challenge affecting classification accuracy relates to class imbalance within the training data, i.e. low shares of specific categories, as machine learning models require a balanced set of examples for effective learning. Although LLMs offer the potential for handling imbalanced categories by generating additional labeled data (e.g. Gupta2024; Pelaez2024), human annotation remains superior when dealing with rare or nuanced categories.
Selecting the appropriate classification approach is also crucial, as different methods offer varying levels of performance depending on the complexity of the task. In this study, binary classification outperformed multi-label and multi-task approaches. The classification accuracy achieved shows F1 scores that align with findings from recent studies, such as severin23, and even surpass those from other analyses like ghosal22. Notably, models trained on domain-specific data did not significantly outperform more general models, as also documented by Foster2024, suggesting that comparing different modelling approaches can be more effective than over-optimizing the model choice. The choice of the unit of analysis can also influence performance, and sentence-level analysis employed in similar studies ghosal22, severin23 proved more effective than classifying longer sequences in this context. The similar performance of the different models might indeed stem from the unit of analysis being a single sentence, where the additional value of pre-training on the scientific texts is potentially limited. Finally, researchers should select classifiers based on quantitative metrics, such as accuracy thresholds, and also qualitative assessments, like identifying systematic deviations from human annotations, to ensure trustworthy substantive analyses. Importantly, the classifiers should not be used for automatic classification of grant peer review reports without human oversight and the level of the classification accuracy should be critically evaluated before deployment.
Beyond findings related to the annotation and classification process, this study offers insights into the characteristics and prevalence of elements in grant peer review reports submitted to a national research funder hug2024referees. The peer review reports predominantly focused on the proposal and methods rather than the applicant. This finding aligns with the funding scheme for which the review reports were written: SNSF Project Funding is a project- rather than a person-centered funding scheme. It is reassuring that quantitative assessments of applicants based on the number of publications or bibliometric indicators such as citation counts, the h-index or impact factors were uncommon, in line with the Declaration of Research Assessment (DORA, see bladek2014dora) signed by the SNSF in 2014. It will be interesting to compare peer review reports for project-centered funding scheme with reports submitted to person-centered funding schemes that promote early career researchers. Further, the findings reveal that grant peer reviews tend to be more positive than negative, a result in line with previous studies of grant proposals hren2022 but contrary to evidence from journal peer review severin23. While over 30% of sentences from a typical journal peer review contained suggestions severin23, the average predicted prevalence of suggestions in our sample of grant peer reviews was only 4%. Notably, most of the positive or negative statements (67%) in the subset of 3,000 annotated sentences came with a rationale for the statement, i.e. justification for the praise or criticism was provided in the target sentence or the surrounding sentences (see Figure (ref)). Overall, these first descriptive findings offer valuable insights into grant peer review practices, illustrating the potential of further analyzing whether reviewers adhere to the scheme's objectives and funders' guidelines. Detailed analyses are ongoing and will be the subject of future articles.
In conclusion, the pipeline developed in this study lays the groundwork for more detailed and systematic assessments of grant peer review reports. It enables research funding agencies to systematically evaluate and refine practices, with the ultimate aim of improving the evaluation process. Researchers may benefit from a clearer and more structured understanding of the peer review process, which could lead to increased engagement and adherence to review guidelines.
Due to data privacy laws, no raw data from this paper can be publicly shared. For a detailed description of data protection please refer to the data management plan underlying this work: \url{https://doi.org/10.46446/DMP-peer-review-assessment-ML}. The annotation codebook developed in this work is available online: \url{https://doi.org/10.46446/Codebook-peer-review-assessment-ML}.
The code scripts for fine-tuning the classification models and the models themselves resulting from this work are available open-source on GitHub: \url{https://github.com/snsf-data/ml-peer-review-analysis}, archived on Zenodo: \url{https://doi.org/10.5281/zenodo.14215058} and Hugging Face Hub: \url{https://huggingface.co/snsf-data}, archived on Zenodo as well: \url{https://doi.org/10.5281/zenodo.14217855}.
This work was financed by the Swiss National Science Foundation (SNSF). Matthias Egger, Katrin Milzow, Anne Jorstad, Michaela Strinzel and Gabriel Okasa are employed by the SNSF.
\singlespacing
\doublespacing