Comparative Performance of Transformer Models for Cultural Heritage in NLP Tasks
DOI:
https://doi.org/10.26877/asset.v7i1.1211Keywords:
Artificial Intelligence, Machine Learning, NLP Cultural Knowledge, Transformer Models, QA Goddess Durga DatasetAbstract
AI and Machine Learning are crucial in advancing technology, especially for processing large, complex datasets. The transformer model, a primary approach in natural language processing (NLP), enables applications like translation, text summarization, and question-answer (QA) systems. This study compares two popular transformer models, FlanT5 and mT5, which are widely used yet often struggle to capture the specific context of the reference text. Using a unique Goddess Durga QA dataset with specialized cultural knowledge about Indonesia, this research tests how effectively each model can handle culturally specific QA tasks. The study involved data preparation, initial model training, ROUGE metric evaluation (ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-Lsum), and result analysis. Findings show that FlanT5 outperforms mT5 on multiple metrics, making it better at preserving cultural context. These results are impactful for NLP applications that rely on cultural insight, such as cultural preservation QA systems and context-based educational platforms.
References
[1] L. Xu, L. Lu, and M. Liu, “Construction and application of a knowledge graph-based question answering system for Nanjing Yunjin digital resources,” Herit. Sci., vol. 11, no. 1, pp. 1–17, Dec. 2023, doi: 10.1186/S40494-023-01068-2/TABLES/6.
[2] F. Jin, Q. Chang, and Z. Xu, “MuseumQA: A Fine-Grained Question Answering Dataset for Museums and Artifacts,” ACM Int. Conf. Proceeding Ser., pp. 221–226, Dec. 2023, doi: 10.1145/3639479.3639525.
[3] Z. Fan and C. Chen, “CuPe-KG: Cultural perspective–based knowledge graph construction of tourism resources via pretrained language models,” Inf. Process. Manag., vol. 61, no. 3, p. 103646, May 2024, doi: 10.1016/J.IPM.2024.103646.
[4] R. C. Climent, D. M. Haftor, and M. W. Staniewski, “AI-enabled business models for competitive advantage,” J. Innov. Knowl., vol. 9, no. 3, p. 100532, Jul. 2024, doi: 10.1016/j.jik.2024.100532.
[5] M. Dumas et al., “AI-augmented Business Process Management Systems: A Research Manifesto,” ACM Trans. Manag. Inf. Syst., vol. 14, no. 1, pp. 1–19, Mar. 2023, doi: 10.1145/3576047.
[6] J. R. Rodriguez Barboza et al., “Posthumanist Technologies in Business: AI and Cloud Computing for Global Optimization and Ethical Challenges,” Adv. Sustain. Sci. Eng. Technol., vol. 6, no. 4, p. 02404021, Oct. 2024, doi: 10.26877/asset.v6i4.1064.
[7] B. Meskó and E. J. Topol, “The imperative for regulatory oversight of large language models (or generative AI) in healthcare,” npj Digit. Med., vol. 6, no. 1, p. 120, Jul. 2023, doi: 10.1038/s41746-023-00873-0.
[8] S. M. Williamson and V. Prybutok, “Balancing Privacy and Progress: A Review of Privacy Challenges, Systemic Oversight, and Patient Perceptions in AI-Driven Healthcare,” Appl. Sci., vol. 14, no. 2, p. 675, Jan. 2024, doi: 10.3390/app14020675.
[9] N. Susetyo et al., “Natural Language Processing in Higher Education,” Bull. Soc. Informatics Theory Appl., vol. 6, no. 1, pp. 90–101, Jul. 2022, doi: 10.31763/BUSINTA.V6I1.593.
[10] Y. Pande, “Project EngiBot: Engineering Insights through NLP- driven Chatbot,” Int. J. Res. Appl. Sci. Eng. Technol., vol. 11, no. 11, pp. 689–692, Nov. 2023, doi: 10.22214/ijraset.2023.56590.
[11] J. Su (苏嘉红) and W. Yang (杨伟鹏), “Unlocking the Power of ChatGPT: A Framework for Applying Generative AI in Education,” ECNU Rev. Educ., vol. 6, no. 3, pp. 355–366, Aug. 2023, doi: 10.1177/20965311231168423.
[12] J. Qadir, “Engineering Education in the Era of ChatGPT: Promise and Pitfalls of Generative AI for Education,” in 2023 IEEE Global Engineering Education Conference (EDUCON), IEEE, May 2023, pp. 1–9. doi: 10.1109/EDUCON54358.2023.10125121.
[13] X. Wang, L. Li, S. C. Tan, L. Yang, and J. Lei, “Preparing for AI-enhanced education: Conceptualizing and empirically examining teachers’ AI readiness,” Comput. Human Behav., vol. 146, p. 107798, Sep. 2023, doi: 10.1016/j.chb.2023.107798.
[14] T. L. M. Suryanto, A. P. Wibawa, H. Hariyono, and A. Nafalski, “Evolving Conversations: A Review of Chatbots and Implications in Natural Language Processing for Cultural Heritage Ecosystems,” Int. J. Robot. Control Syst., vol. 3, no. 4, pp. 955–1006, Dec. 2023, doi: 10.31763/ijrcs.v3i4.1195.
[15] I. D. Raharjo and Egia Rosi Subhiyakto, “Implementing Long Short Term Memory (LSTM) in Chatbots for Multi Usaha Raya,” Adv. Sustain. Sci. Eng. Technol., vol. 6, no. 4, p. 02404018, Oct. 2024, doi: 10.26877/asset.v6i4.934.
[16] S. Hawanti and K. M. Zubaydulloevna, “AI chatbot-based learning: alleviating students’ anxiety in english writing classroom,” Bull. Soc. Informatics Theory Appl., vol. 7, no. 2, pp. 182–192, Dec. 2023, doi: 10.31763/BUSINTA.V7I2.659.
[17] P. D. Larasati, A. Irawan, S. Anwar, M. F. Mulya, M. A. Dewi, and I. Nurfatima, “Chatbot helpdesk design for digital customer service.” Accessed: Dec. 20, 2024. [Online]. Available: https://pubs2.ascee.org/index.php/aet/article/view/684
[18] P. A. Alia, D. Kartika Sari, N. Azis, B. Gunawan Sudarsono, and P. Agus Sucipto, “Implementation Artificial Intelligence with Natural Language Processing Method to Improve Performance of Digital Product Sales Service,” Adv. Sustain. Sci. Eng. Technol., vol. 6, no. 3, p. 0240301, Jun. 2024, doi: 10.26877/asset.v6i3.521.
[19] T. B. Brown et al., “Language Models are Few-Shot Learners,” Adv. Neural Inf. Process. Syst., vol. 2020-December, May 2020, Accessed: Dec. 19, 2024. [Online]. Available: https://arxiv.org/abs/2005.14165v4
[20] C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” J. Mach. Learn. Res., vol. 21, pp. 1–67, 2020.
[21] M. A. Hasan et al., “NativQA: Multilingual Culturally-Aligned Natural Query for LLMs,” Jul. 2024, Accessed: Dec. 19, 2024. [Online]. Available: https://arxiv.org/abs/2407.09823v2
[22] L. Colucci Cante, B. Di Martino, M. Graziano, D. Branco, and G. J. Pezzullo, “Automated Storytelling Technologies for Cultural Heritage,” Lect. Notes Data Eng. Commun. Technol., vol. 193, pp. 597–606, 2024, doi: 10.1007/978-3-031-53555-0_57.
[23] E. M. B. Nagoudi, A. Elmadany, and M. Abdul-Mageed, “AraT5: Text-to-Text Transformers for Arabic Language Generation,” 2022, [Online]. Available: https://doi.org/10.48550/arXiv.2109.12068
[24] A. Prasetya Wibawa, H. K. Fithri, I. Ari, E. Zaeni, and A. Nafalski, “Generating Javanese Stopwords List using K-means Clustering Algorithm,” Knowl. Eng. Data Sci., vol. 3, no. 2, pp. 106–111, Dec. 2020, doi: 10.17977/UM018V3I22020P106-111.
[25] J. Santoso, E. I. Setiawan, C. N. Purwanto, and F. Kurniawan, “Indonesian Sentence Boundary Detection using Deep Learning Approaches,” Knowl. Eng. Data Sci., vol. 4, no. 1, pp. 38–48, Jun. 2021, doi: 10.17977/UM018V4I12021P38-48.
[26] L. Xue et al., “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Stroudsburg, PA, USA: Association for Computational Linguistics, 2021, pp. 483–498. doi: 10.18653/v1/2021.naacl-main.41.
[27] T. Wolf et al., “Transformers: State-of-the-Art Natural Language Processing,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Stroudsburg, PA, USA: Association for Computational Linguistics, 2020, pp. 38–45. doi: 10.18653/v1/2020.emnlp-demos.6.
[28] A. Mastropaolo et al., “Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks,” pp. 336–347, 2021, doi: 10.1109/ICSE43902.2021.00041.
[29] Z. Chi et al., “mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Stroudsburg, PA, USA: Association for Computational Linguistics, 2021, pp. 1671–1683. doi: 10.18653/v1/2021.emnlp-main.125.
[30] J. Frei and F. Kramer, “Annotated dataset creation through large language models for non-english medical NLP,” J. Biomed. Inform., vol. 145, p. 104478, Sep. 2023, doi: 10.1016/j.jbi.2023.104478.
[31] A. I. Regla and M. A. Ballera, “An Enhanced Research Productivity Monitoring System for Higher Education Institutions (HEI’s) with Natural Language Processing (NLP),” Procedia Comput. Sci., vol. 230, pp. 316–325, 2023, doi: 10.1016/j.procs.2023.12.087.
[32] L. Ranaldi and F. M. Zanzotto, “Discover AI Knowledge to Preserve Cultural Heritage,” Sep. 2021, doi: 10.20944/PREPRINTS202109.0062.V1.
[33] H. R. Will, N. Masini, and D. H. R. Spennemann, “Will Artificial Intelligence Affect How Cultural Heritage Will Be Managed in the Future? Responses Generated by Four genAI Models,” Herit. 2024, Vol. 7, Pages 1453-1471, vol. 7, no. 3, pp. 1453–1471, Mar. 2024, doi: 10.3390/HERITAGE7030070.
[34] S. Pawar et al., “Survey of Cultural Awareness in Language Models: Text and Beyond,” Oct. 2024, Accessed: Dec. 19, 2024. [Online]. Available: https://arxiv.org/abs/2411.00860v1
[35] C. Qu, L. Yang, C. Chen, M. Qiu, W. B. Croft, and M. Iyyer, “Open-Retrieval Conversational Question Answering,” SIGIR 2020 - Proc. 43rd Int. ACM SIGIR Conf. Res. Dev. Inf. Retr., pp. 539–548, 2020, doi: 10.1145/3397271.3401110.
[36] J. Li et al., “Graphix-T5 : Mixing Pre-trained Transformers with Graph-Aware Layers for Text-to-SQL Parsing,” 2022, [Online]. Available: https://doi.org/10.48550/arXiv.2301.07507
[37] T. Shao, Y. Guo, H. Chen, and Z. Hao, “Transformer-Based Neural Network for Answer Selection in Question Answering,” IEEE Access, vol. 7, pp. 26146–26156, 2019, doi: 10.1109/ACCESS.2019.2900753.
[38] X. Chen, P. Cong, and S. Lv, “A Long-Text Classification Method of Chinese News Based on BERT and CNN,” IEEE Access, vol. 10, pp. 34046–34057, 2022, doi: 10.1109/ACCESS.2022.3162614.
[39] L. Liu, X. Su, H. Guo, and D. Zhu, “A Transformer-based Medical Visual Question Answering Model,” in 2022 26th International Conference on Pattern Recognition (ICPR), IEEE, Aug. 2022, pp. 1712–1718. doi: 10.1109/ICPR56361.2022.9956469.
[40] S. Zhu et al., “Multilingual Large Language Models: A Systematic Survey,” 2024, Accessed: Dec. 19, 2024. [Online]. Available: https://github.com/tjunlp-lab/Awesome-Multilingual-LLMs-Papers
[41] B. Weng, “Navigating the Landscape of Large Language Models: A Comprehensive Review and Analysis of Paradigms and Fine-Tuning Strategies,” Apr. 2024, Accessed: Dec. 19, 2024. [Online]. Available: https://arxiv.org/abs/2404.09022v1
[42] M. Nitu and M. Dascalu, “Natural Language Processing Tools for Romanian-Going Beyond a Low-Resource Language”, doi: 10.55612/s-5002-060-001sp.
[43] C. Wang, P. Liu, and Y. Zhang, “Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA,” 2018.
[44] Y. Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” no. 1, 2019, [Online]. Available: http://arxiv.org/abs/1907.11692
[45] A. Mastropaolo et al., “Using Transfer Learning for Code-Related Tasks,” IEEE Trans. Softw. Eng., vol. 49, no. 4, pp. 1580–1598, Apr. 2023, doi: 10.1109/TSE.2022.3183297.
[46] C. Mallikarjuna and S. Sivanesan, “Question classification using limited labelled data,” Inf. Process. Manag., vol. 59, no. 6, p. 103094, Nov. 2022, doi: 10.1016/j.ipm.2022.103094.
[47] E. Alsentzer et al., “Zero-shot interpretable phenotyping of postpartum hemorrhage using large language models,” npj Digit. Med. 2023 61, vol. 6, no. 1, pp. 1–10, Nov. 2023, doi: 10.1038/s41746-023-00957-x.
[48] B. Pecher, I. Srba, and M. Bielikova, “Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance,” Feb. 2024, Accessed: Dec. 20, 2024. [Online]. Available: https://arxiv.org/abs/2402.12819v2
[49] A. Auriemma Citarella, M. Barbella, M. G. Ciobanu, F. De Marco, L. Di Biasi, and G. Tortora, “Rouge Metric Evaluation for Text Summarization Techniques.” 2024. doi: 10.2139/ssrn.4753220.
[50] M. Fuadi, A. D. Wibawa, and S. Sumpeno, “idT5 : Indonesian Version of Multilingual T5 Transformer,” 2023, [Online]. Available: https://doi.org/10.48550/arXiv.2302.00856
[51] D. Uthus, S. Ontañón, J. Ainslie, and M. Guo, “mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences,” vol. 5, 2023.
[52] M. Guo, J. Ainslie, and D. Uthus, “LongT5 : Efficient Text-To-Text Transformer for Long Sequences,” pp. 724–736, 2022, [Online]. Available: https://doi.org/10.48550/arXiv.2112.07916
[53] Z. Zhang, X. He, V. Iyer, and A. Birch, “Cultural Adaptation of Menus: A Fine-Grained Approach,” Aug. 2024, Accessed: Dec. 20, 2024. [Online]. Available: https://arxiv.org/abs/2408.13534v1
[54] S. Dwivedi, S. Ghosh, and S. Dwivedi, “Navigating Linguistic Diversity: In-Context Learning and Prompt Engineering for Subjectivity Analysis in Low-Resource Languages,” SN Comput. Sci., vol. 5, no. 4, pp. 1–9, Apr. 2024, doi: 10.1007/S42979-024-02770-Z/TABLES/2.
[55] E. Merdivan, D. Singh, S. Hanke, and A. Holzinger, “Dialogue Systems for Intelligent Human Computer Interactions,” Electron. Notes Theor. Comput. Sci., vol. 343, pp. 57–71, May 2019, doi: 10.1016/J.ENTCS.2019.04.010.
[56] T. Shen et al., “ViDA-MAN: Visual Dialog with Digital Humans,” MM 2021 - Proc. 29th ACM Int. Conf. Multimed., pp. 2789–2791, Oct. 2021, doi: 10.1145/3474085.3478560/SUPPL_FILE/DE3228.MP4.
[57] J. Tian et al., “ChatPLUG: Open-Domain Generative Dialogue System with Internet-Augmented Instruction Tuning for Digital Human,” Apr. 2023, Accessed: Dec. 20, 2024. [Online]. Available: https://arxiv.org/abs/2304.07849v3
[58] S. Longpre et al., “The Flan Collection: Designing Data and Methods for Effective Instruction Tuning,” Proc. Mach. Learn. Res., vol. 202, pp. 22631–22648, 2023.
[59] V. Bouras et al., “Chatbots for Cultural Venues: A Topic-Based Approach,” Algorithms 2023, Vol. 16, Page 339, vol. 16, no. 7, p. 339, Jul. 2023, doi: 10.3390/A16070339.
[60] P. K. Rachabatuni, F. Principi, P. Mazzanti, and M. Bertini, “Context-aware chatbot using MLLMs for Cultural Heritage,” MMSys 2024 - Proc. 2024 ACM Multimed. Syst. Conf., pp. 459–463, Apr. 2024, doi: 10.1145/3625468.3652193.
[61] M. Tsepapadakis and D. Gavalas, “Are you talking to me? An Audio Augmented Reality conversational guide for cultural heritage,” Pervasive Mob. Comput., vol. 92, p. 101797, May 2023, doi: 10.1016/J.PMCJ.2023.101797.
[62] K. Sathiyabamavathy and A. K. P., “Role of Chatbots in Cultural Heritage Tourism: An Empirical Study on Ancient Forts and Palaces,” https://doi.org/10.1177/24559296241253932, vol. 9, no. 1, pp. 9–28, Jun. 2024, doi: 10.1177/24559296241253932.
[63] J. Enciso-Anaya, K. Tamil, and Q. Truong, “Virtual Museum Tours of the de Saisset Museum,” Comput. Sci. Eng. Sr. Theses, Jun. 2024, Accessed: Dec. 20, 2024. [Online]. Available: https://scholarcommons.scu.edu/cseng_senior/306
[64] T. Komarac and Đ. Ozretić Došen, “Understanding virtual museum visits: generation Z experiences,” Museum Manag. Curatorsh., vol. 39, no. 3, pp. 357–376, May 2024, doi: 10.1080/09647775.2023.2269129.
[65] Z. Wang, L. P. Yuan, L. Wang, B. Jiang, and W. Zeng, “VirtuWander: Enhancing Multi-modal Interaction for Virtual Tour Guidance through Large Language Models,” Conf. Hum. Factors Comput. Syst. - Proc., May 2024, doi: 10.1145/3613904.3642235/SUPPL_FILE/PN7938-SUPPLEMENTAL-MATERIAL-2.PDF.
[66] R. Patria and P. H. Merdeka, “Creative Strategies in the Recovery of Endangered Languages,” J. Lit. Lang. Acad. Stud., vol. 2, no. 02, pp. 57–61, Aug. 2023, doi: 10.56855/JLLANS.V2I2.671.
[67] D. K. Nanduri and E. M. Bonsignore, “Revitalizing Endangered Languages: AI-powered language learning as a catalyst for language appreciation,” Apr. 2023, Accessed: Dec. 20, 2024. [Online]. Available: https://arxiv.org/abs/2304.09394v1
[68] I. Ozkaya, “Application of Large Language Models to Software Engineering Tasks: Opportunities, Risks, and Implications,” IEEE Softw., vol. 40, no. 3, pp. 4–8, May 2023, doi: 10.1109/MS.2023.3248401.
[69] L. Wang et al., “Document-Level Machine Translation with Large Language Models,” EMNLP 2023 - 2023 Conf. Empir. Methods Nat. Lang. Process. Proc., pp. 16646–16661, Apr. 2023, doi: 10.18653/v1/2023.emnlp-main.1036.
[70] W. J. Hutchins, “Machine Translation: History of Research and Applications,” Routledge Encycl. Transl. Technol. Second Ed., pp. 128–144, Jan. 2023, doi: 10.4324/9781003168348-7/MACHINE-TRANSLATION-JOHN-HUTCHINS.
