Argument mining as rapid screening tool of COVID-19 literature quality: Preliminary evidence

Background: The COVID-19 pandemic prompted the scientific community to share timely evidence, including preprints that had not yet been peer reviewed. Purpose: We developed an artificial-intelligence system for analyzing scientific literature using recent advances in argument mining. Methodology: Scientific-quality criteria were drawn from two selected Cochrane systematic reviews. Four independent reviewers blindly scored 40 papers for each review on a 1–5 scale. These scores were compared with automatic analysis by MARGOT, which detected claims and supporting evidence. Results: MARGOT performed differently across the two reviews, with fair-to-moderate agreement between its most relevant metrics and both Cochrane and expert scores. Discussion and conclusions: The discrepancy may reflect a limitation of MARGOT, while agreement between human reviewers also suggests that the reviews differ in complexity. The preliminary results motivate broader evaluation with more topics and specialized reviewers.