VIRAS: Visually Impaired Robot-Assisted Shopping

The ILT is developing an autonomous shopping cart for blind and visually impaired people in close collaboration with four Swiss associations for the blind.

Chantal Keller, Prof. Dr. Dario Schafroth

Blind and visually impaired people in focus

The Institute for Lab Automation and Mechatronics (ILT) at the Eastern Switzerland University of Applied Sciences has initiated the project VIRAS (Visually Impaired Robot-Assisted Shopping). It encompasses the development of an innovative shopping aid for blind and visually impaired people. VIRAS is a self-driving shopping cart that allows those affected to independently conduct their shopping in a supermarket.

To assess the need, a nationwide survey was carried out among various associations for the blind in Germany and Switzerland. The results clearly show a very strong interest within this group of people in receiving technical assistance while shopping in the supermarket. Currently, most affected people avoid shopping by themselves. Often, a third person is tasked with this. The use of the VIRAS shopping cart is well-received by 87% of the respondents. If the shopping cart were made available, almost 90% of visually impaired and blind people would use it to make their purchases independently.

4% of the Swiss population - tending to increase - are affected by a visual impairment. In light of this, the project can not only contribute to inclusion but also lead to a significant improvement in the quality of life of those affected. By integrating autonomous navigation functions into shopping carts, blind and visually impaired people have the opportunity to navigate supermarkets independently. This significantly enhances their self-reliance and leads to a more self-determined life.

The Swiss National Association of and for the Blind, the Schweizerischer Blindenbund, the Schweizerische Caritasaction der Blinden, and the Schweizerischer Blinden- und Sehbehindertenverband promote the development and implementation of the VIRAS project through their financial and ideological support.

The VIRAS team is always in close contact with the associations and discusses possible approaches. By involving the affected group of people, the development of the VIRAS shopping cart is tailored to their specific needs in a realistic way.

Challenges in Shopping

The need for a shopping cart for blind and visually impaired people arises from several factors. The most significant and demanding challenge when shopping without external assistance is the precise location of products on the shelf. Particularly when there are changes in the assortment and product placement, those affected have considerable difficulties in finding the desired product. Uneven lighting also complicates orientation. Weighing vegetables or fruit is nearly impossible since the scales cannot be operated without sufficient visual capacity. Another obstacle is finding the checkout counter. In the supermarket, no information is provided for blind and visually impaired people on how to reach it. These obstacles and other limitations are intended to be overcome with the VIRAS shopping cart in order to provide an improved shopping experience for those affected.

Shopping with VIRAS

Barrier-free shopping will be possible with the VIRAS shopping cart. The autonomous system is equipped with a stereo camera and integrated speech recognition and speech output functions. The shopping cart is motor-driven and operates self-driving. This will enable the user to be guided to the desired products independently in the future. To ensure safety, the shopping cart should avoid people and obstacles in the supermarket. The stereo camera is not only used for 3D mapping of the supermarket but also identifies the products and the user's hand, providing him with acoustic instructions to reach the target product. Product recognition is based on AI object recognition. This innovative application eliminates the need to scan the product's barcode, making it possible to reliably detect them regardless of their orientation. If a shelf is empty, the user can be notified.

A shopping process for blind and visually impaired people could then be as follows:

Localization of the shopping cart in the supermarket through acoustic signals.
Verbal input of the shopping list.
Navigation to the products while avoiding obstacles or people.
Halting the cart upon reaching the product.
Product recognition through AI object recognition.
Acoustic naming of the product's position on the shelf.
User hand recognition.
Acoustic guidance on the position of the user's hand relative to the product.
Acoustic confirmation of the product name when the user has selected the correct product.
Navigation to the next product or the checkout.

Visions

The VIRAS project aims for a series of exciting developments. Furthermore, the user group is intended to be expanded. In addition to blind and visually impaired people, the shopping cart should support people with limited mobility. Even individuals without restrictions can use the VIRAS shopping cart as a navigation aid to quickly and efficiently locate products. In addition, functions for inventory control should be implemented. This includes the ability to check product availability in real-time, and promotional offers can also be provided through the shopping cart.