Image Captioning with Reinforcement Learning

Verma A.; Agarwal S.; Arya K.V.; Petrlik I.; Esparza R.; Rodriguez C. · Proceedings of the 2023 IEEE International Conference on Computer Vision and Machine Intelligence, CVMI 2023 (2023) · DOI: 10.1109/cvmi59935.2023.10464514

Resumen

Image captioning involves generating a natural language description that accurately represents the content and context of an image. To achieve this, image captioning utilises various machine learning techniques and fields, such as computer vision and natural language processing. In the field of image captioning, a lot of advances have been made with encoder-decoder models and reinforcement learning algorithms. However, there are still problems of imbalance between testing and training, as reinforcement learning only handles single comparator metrics such as CIDEr, SPICE, and BLEU and could not perform better in multiple metrics at once. Which is why a lack of diversity can be seen in generated captions. This idea proposes a general technique for collaborative updating that can bridge the gap between evaluation measures and test metrics to produce captions that are more human-like. To increase the precision of image captions, the approach involves using a compiled reward system that considers multiple evaluation metrics to compare the generated sentence with the provided sentences. We will evaluate the model's performance and the reward updating process on standard datasets like MS COCO.

Instituciones: Universidad Nacional Mayor de San Marcos; Universidad Nacional Federico Villarreal