Fine-tuning language models for social sentiment: insights from startup user feedback

Abstract

Startup growth and sustainability are heavily impacted by user comments. This research has done a comparative study between large language models (LLMs), small language models (SLMs), and traditional machine learning (ML) methods in classifying user comments on startups. The performances of bidirectional encoder representations from transformers (BERT), Indonesian bidirectional encoder representations from transformers (IndoBERT), distilled bidirectional encoder representations from transformers (DistilBERT), and ML models; logistic regression, support vector machine (SVM), random forest, and extreme gradient boosting (XGBoost) were evaluated with a diverse dataset and beyond extensive preprocessing. Out of all the models this research trained, IndoBERT outperformed with an accuracy and recall 90.99% and F1-score of 89.94%, due to its intermediate pretraining for Indonesian language task. DistilBERT, from the SLM family of models and only requiring a fraction of resources to make predictions performed significantly better (over 90% in both accuracy and recall). These outcomes indicate that LLMs, particularly those targeting specific languages, are significantly superior in performance for sentiment analysis. The results seem to imply that models like these can be utilized to study the phenomenon and further enhance user engagement with startups. Future work will involve the further development of more language specific BERT versions and expanding LLMs in other ways within startup spaces.