A Novel Classification Framework Using Transformer-Based Encoding and Low-Rank Tensor Fusion for Enhanced Classification and Efficiency
DOI:
https://doi.org/10.26877/jgz0xe27Keywords:
Sentiment Analysis, low-rank tensor, Social Media AnalyticsAbstract
This paper proposes a transformer-based framework for sentiment analysis, designed to improve both accuracy and computational efficiency across diverse datasets. The model incorporates a low-rank tensor fusion mechanism to reduce computational complexity, optimizing the transformer encoder’s performance. Through an extensive evaluation on three benchmark datasets—Airlines, CrowdFlower, and Apple—our approach demonstrates superior performance in sentiment classification tasks, achieving accuracy levels of 93.2%, 91.5%, and 92.1%, respectively. The framework utilizes standard performance metrics, including precision, recall, and F1-score, showing consistent improvements of 5-10% over traditional models. Additionally, the model's efficiency is highlighted by its reduced processing time (120 ms per sample), making it suitable for real-time applications. The ablation study reveals that components such as pre-trained embeddings and attention mechanisms significantly contribute to its performance. The results underscore the model's robustness in handling varying sentiment distributions and highlight its scalability for large-scale sentiment analysis tasks. This study provides valuable insights into the practical application of transformer-based models in sentiment analysis, offering an efficient solution for processing diverse social media data in real-time.
References
[1]. X. Gong, W. Ying, S. Zhong, and S. Gong, "Text sentiment analysis based on transformer and augmentation," Frontiers in Psychology, vol. 13, p. 906061, 2022. doi: 10.3389/fpsyg.2022.906061.
[2] S. T. Kokab, S. Asghar, and S. Naz, "Transformer-based deep learning models for the sentiment analysis of social media data," Mater. Today: Proc., vol. 14, p. 100157, 2022. doi: 10.1016/j.matpr.2021.07.102.
[3] K. Jindal and R. Aron, "A systematic study of sentiment analysis for social media data," Mater. Today: Proc., vol. 47, pp. 5893-5899, 2021. doi: 10.1016/j.matpr.2021.01.123.
[4] S. T. Al-Otaibi and A. A. Al-Rasheed, "A review and comparative analysis of sentiment analysis techniques," Informatica, vol. 46, no. 6, pp. 3991-3999, 2022. doi: 10.31449/inf.v46i6.3991.
[5] L. Mathew and V. R. Bindu, "Efficient classification techniques in sentiment analysis using transformers," in Int. Conf. Innov. Comput. Commun., Springer, pp. 849-862, 2022. doi: 10.1007/978-981-16-2594-7_69.
[6] J. Dai, H. Yan, T. Sun, P. Liu, and X. Qiu, "Does syntax matter? A strong baseline for aspect-based sentiment analysis with RoBERTa," arXiv Preprint, 2021. Available: https://arxiv.org/abs/2104.04986.
[7] S. Çalı and Ş. Y. Balaman, "Improved decisions for marketing, supply, and purchasing: Mining big data through an integration of sentiment analysis and intuitionistic fuzzy multi-criteria assessment," Comput. Ind. Eng., vol. 129, pp. 315-332, 2019. doi: 10.1016/j.cie.2019.01.016.
[8] G. Choi, S. Oh, and H. Kim, "Improving document-level sentiment classification using the importance of sentences," Entropy, vol. 22, no. 12, p. 1336, 2020. doi: 10.3390/e22121336.
[9] P. Chauhan, N. Sharma, and G. Sikka, "The emergence of social media data and sentiment analysis in election prediction," J. Ambient Intell. Humanized Comput., vol. 12, no. 2, pp. 2601-2627, 2021. doi: 10.1007/s12652-020-02383-2.
[10] N. Kulkarni, D. Gokhale, and A. Rani, "A comparative study of word embedding techniques to extract features from text," Turk. J. Comput. Math. Educ. (TURCOMAT), vol. 12, no. 12, pp. 3550-3557, 2021.
[11] J. Gou, B. Yu, S. J. Maybank, and D. Tao, "Knowledge distillation: A survey," Int. J. Comput. Vis., vol. 129, pp. 1789–1819, 2021. doi: 10.1007/s11263-021-01453-z.
[12] K. Han et al., "Transformer in transformer," in Adv. Neural Inf. Process. Syst., vol. 34, MIT Press, pp. 1-13, 2021.
[13] S. Amiriparian et al., "The MuSe 2024 multimodal sentiment analysis challenge: Social perception and humor recognition," 2024. Available: https://doi.org/10.48550/arXiv.2406.07753.
[14] J. Cabezas, D. Moctezuma, A. Fernández-Isabel, and I. Martin de Diego, "Detecting emotional evolution on Twitter during the COVID-19 pandemic using text analysis," Int. J. Environ. Res. Public Health, vol. 18, no. 13, p. 6981, 2021. doi: 10.3390/ijerph18136981.
[15] L. Christ et al., "The MuSe 2023 multimodal sentiment analysis challenge: Mimicked emotions, cross-cultural humour, and personalisation," in Proc. 4th Multimodal Sentiment Analysis Challenge and Workshop, pp. 1–10, 2023.
[16] D. She et al., "WSCNet: Weakly supervised coupled networks for visual sentiment classification and detection," IEEE Trans. Multimedia, vol. 22, pp. 1358-1371, 2020. doi: 10.1109/TMM.2019.2936402.
[17] N. C. Dang, M. N. Moreno-García, and F. De la Prieta, "Sentiment analysis based on deep learning: A comparative study," Electronics, vol. 9, no. 3, 2020. doi: 10.3390/electronics9030483.
[18] K. Fountoulakis et al., "Self-reported changes in anxiety, depression, and suicidality during the COVID-19 lockdown in Greece," J. Affect. Disord., vol. 279, pp. 624–629, 2021. doi: 10.1016/j.jad.2020.10.017.
[19] S. Frenda et al., "The unbearable hurtfulness of sarcasm," Expert Syst. Appl., vol. 193, p. 116398, 2022.
[20] Gandhi, K. Adhvaryu, S. Poria, E. Cambria, and A. Hussain, "Multimodal sentiment analysis: A systematic review of history, datasets, multimodal fusion methods, applications, challenges, and future directions," Inf. Fusion, vol. 91, pp. 424–444, 2023. doi: 10.1016/j.inffus.2022.10.013.
[21] R. Geethanjali and A. Valarmathi, "A novel hybrid deep learning IChOA-CNN-LSTM model for modality-enriched and multilingual emotion recognition in social media," Sci. Rep., vol. 14, p. 22270, 2024. doi: 10.1038/s41598-024-73452-2.
[22] H. Zhu, L. Li, H. Jiang, and A. Tan, "Inferring personality traits from attentive regions of user-liked images via weakly supervised dual convolutional network," Neural Process. Lett., vol. 51, no. 3, pp. 2105-2121, 2020. doi: 10.1007/s11063-020-09743-6.
[23] B. Jang et al., "Bi-LSTM model to increase accuracy in text classification: Combining Word2vec CNN and attention mechanism," Appl. Sci., vol. 10, no. 17, 2020. doi: 10.3390/app10175841.
[24] Z. Liu et al., "Modality translation-based multimodal sentiment analysis under uncertain missing modalities," Inf. Fusion, vol. 101, p. 101973, 2024. doi: 10.1016/j.inffus.2023.101973.
[25] L. Zhu et al., "Multimodal sentiment analysis based on fusion methods: A survey," Inf. Fusion, vol. 95, pp. 306-325, 2023. doi: 10.1016/j.inffus.2023.02.028.
[26] J. M. Tshimula, B. Chikhaoui, and S. Wang, "COVID-19 detecting depression signals during stay-at-home period," 2021.
[27] M. Valstar et al., "FERA 2015-second facial expression recognition and analysis challenge," in Proc. IEEE ICFG, 2015.
[28] M. Viviani et al., "Assessing vulnerability to psychological distress during the COVID-19 pandemic through the analysis of microblogging content," Future Gener. Comput. Syst., vol. 125, pp. 446–459, 2021. doi: 10.1016/j.future.2021.07.003.
[29] W. Guo et al., "LD-MAN: Layout-driven multimodal attention network for online news sentiment recognition," IEEE Trans. Multimedia, vol. 23, pp. 1785-1798, 2021.
[30] G. Wang, G. Yu, and X. Shen, “The effect of online investor sentiment on stock movements: an LSTM approach,” Complexity, vol. 2020, no. 11, p. 4754025, 2020, doi: 10.1155/2020/4754025.
[31] Y. Wang, M. Wang, and W. Xu, “A sentiment-enhanced hybrid recommender system for movie recommendation: a big data analytics framework,” Wireless Communications and Mobile Computing, vol. 2018, no. 9, p. 8263704, 2018, doi: 10.1155/2018/8263704.
[32] T. Yue, R. Mao, H. Wang, Z. Hu, and E. Cambria, “KnowleNet: Knowledge fusion network for multimodal sarcasm detection,” Information Fusion, vol. 100, p. 101921, 2023, doi: 10.1016/j.inffus.2023.101921
