سیستم خودکار زیرنویس تصویر
Abstract
1-Introduction
2-Relevant Work
3-BanglaLekha-Image Captions: The Data Set
4-Model and Training Details
5-Results and Discussion
6-Conclusion
7-References
Abstract
Automatic image caption generation aims to produce an accurate description of an image in natural language automatically. How- ever, Bangla, the fifth most widely spoken language in the world, is lagging considerably in the research and development of such domain. Besides, while there are many established data sets related to image annotation in English, no such resource exists for Bangla yet. Hence, this paper outlines the development of “Chittron”, an automatic image captioning system in Bangla. To address the data set availability issue, a collection of 16, 000 Bangladeshi contextual images has been accumulated and manually annotated in Bangla. This data set is then used to train a model that integrates a pre-trained VGG16 image embedding model with stacked LSTM layers. The model is trained to predict the caption when the input is an image, one word at a time. The results show that the model has successfully been able to learn a working language model and to generate captions of images quite accurately in many cases. The results are evaluated mainly qualitatively. However, BLEU scores are also reported. It is expected that a better result can be obtained with a bigger and more varied data set.
Introduction
The stark reality is that most of the works in image captioning have concentrated almost exclusively on English language [1, 2, 3]. Additionally, the relevant data-sets, e.g. the MSCOCO [4], have a prominent western preference which leads to a two-pronged problem: (1) the language in which captions are generated is English only, and (2)the data set is not representative of the cultural peculiarities of non-western countries. These very problems exist for generating image captions in Bangla, particularly for images which have a decidedly Bangla geocultural flavor. A simple example of this can be seen in Figure 1, where a web service is used to generate captions. The service uses the im2txt model trained on the MSCOCO data set and quite clearly the model fails to recognize the image in Figure 1a as a boy wearing a lungi, a very common male garb in Bangladesh. In fact, it incorrectly identifies the subject as a female since the attire is identified as women gown.