Chinese paper classification based on pre-trained language model and hybrid deep learning method

Abstract

With the explosive growth in the number of published papers, researchers must filter papers by category to improve retrieval efficiency. The features of data can be learned through complex network structures of deep learning models without the need for manual definition and extraction in advance, resulting in better processing performance for large datasets. In our study, the pre-trained language model bidirectional encoder representations from transformers (BERT) and other deep learning models were applied to paper classification. A large-scale chinese scientific literature dataset was used, including abstracts, keywords, titles, disciplines, and categories from 396 k papers. Currently, there is little in-depth research on the role of titles, abstracts, and keywords in classification and how they are used in combination. To address this issue, we evaluated classification results by employing different title, abstract, and keywords concatenation methods to generate model input data, and compared the effects of a single sentence or sentence pair data input methods. We also adopted an ensemble learning approach to integrate the results of models that processed titles, keywords, and abstracts independently to find the best combination. Finally, we studied the combination of different types of models, such as the combination of BERT and convolutional neural networks (CNN), and measured the performance by accuracy, weighted average precision, weighted average recall, and weighted average F1 score.