Social news factuality verification using large language models

Abstract

Social media platforms have greatly accelerated the spread of news, but this rapid information flow also amplifies the risk of misinformation. Traditional automatic detection methods that rely solely on textual features often struggle with nuanced, emerging content. This paper present a novel pipeline that verifies the factuality of social news by clustering related articles into events and using large language models (LLMs) to extract and verify claims against trusted news sources. The approach groups social-media posts and mainstream reports on the same event, extracts atomic claims with a model like GPT-4, and checks each claim’s truthfulness by comparing it to the cluster’s reliable news. This pipeline was evaluated on a newly constructed Vietnamese news dataset of 1,765 articles (including 723 social-media items), manually annotating claims as true or false. The LLM-based method achieved high accuracy (≈88.9% F1-score on claim verification and 92.1% F1-score on overall news verification). These results demonstrate that carefully prompted LLMs, combined with event-level clustering of evidence, can outperform traditional methods (e.g., bidirectional encoder representations from transformers (BERT)-based classifiers) in verifying news. The paper discusses advantages of clustering over simple retrieval, scalability considerations for LLMs, and prospects for multilingual and knowledge-enhanced verification.