Story visualization using image-text matching architecture for digital storytelling

Yturrizaga-Aguirre A.; Silva-Olivares C.; Ugarte W. · Proceedings of the 2022 IEEE Engineering International Research Conference, EIRCON 2022 (2022) · DOI: 10.1109/eircon56026.2022.9934817

Resumen

Currently, the techniques for generating images from text used to visualize stories have serious limitations in terms of image quality, which prevents quantifying their impact in real life scenarios. An example of this occurs in the field of education, where digital storytelling is used as a tool to incite teaching. For this reason, we propose to design a web interface that allows primary school children to write a short story and obtain, as a result, a sequence of coherent and representative images of said content, emulating a conventional process of educational digital storytelling. We describe the use of an Image-text matching architecture based on NLP and Image Retrieval for the story visualization task focused on digital storytelling. To evaluate the performance of the architecture, the quantitative metrics: WuPalmer and cosine similarity were used, in addition to qualitative metrics.

Instituciones: Universidad Peruana de Ciencias Aplicadas