Object Detection and Position using CLIP with Thai Voice Command for Thai Visually Impaired

Because apps on smartphones or portable devices for Thai visually impaired people that accept Thai voice commands are limited, a development of system for finding objects in their home or fallen object on the floor to help Thai people with blind and visual impairments to make their daily lives more comfortable is a challenge for this research. The proposed system was designed to detect the obj ect and locate the position of the finding obj ect using the vision-language pre-training model as CLIP (Contrastive Language-Image Pre-training) with the voice command in the Thai language. The voice is transformed to Thai text and then translated to the English text fed to CLIP along with the image captured by a smartphone or Bluetooth webcam; after that the bounding boxes on the image are generated by a segmentation algorithm. Then, the zero-shot CLIP was used to predict the text description of the image to find the object position. The Thai text of the finding object location was generated by CLIP zero-shot detector module and then transformed into the Thai voice. The results showed that the developed system could detect and locate the finding objects using CLIP combined with the Thai voice command to declare the correct obj ect position in Thai voice, which is very useful for the Thai VIB people.