Augmented Restaurant Menu Translation for Tourists
This project aims to enhance the dining experience for foreign travelers. Designed to address the tension between curiosity about new foods and fear of the unknown, this application aims to make foreign cuisine more accessible and enjoyable, encouraging diners to explore with confidence through a seamless interaction enabled by AR.
Problem statement.
For many travelers, the excitement of exploring new cuisines is often spoiled by confusion and ambiguity with menu translations. Traditional translation tools, while helpful, miss relevant information to convey the cultural significance and culinary context of local dishes. This can lead to hesitation, reducing the willingness to engage with unfamiliar food. It can also lead to less than ideal user experiences, with users relying on several tools in search of all the relevant information. This project addresses this challenge.
EXPLORATORY STUDY
We simulated the ordering experience and conducted semi-structured interviews with 10 international participants in a French restaurant.
Our goal was to understand the challenges and strategies employed by non-French-speaking users when faced with the task of ordering food in an unfamiliar linguistic and cultural landscape. Our analysis showed a tension between the excitement of trying new foods and the fear of making an unsatisfactory choice, worsened by the cultural distance between the user’s background and the restaurant’s cuisine. Participants expressed frustration with the limitations of online tools, which often provided out-of-context or insufficient information, forcing them to resort to a combination of tools (i.e., search engines, translation). Our study underscored the importance of culturally augmented translation—translations that do more than convert text from one language to another, but also provide dietary information and cultural context, including the dish’s history and customary consumption practices.
Fig.: Overview of existing process.
OUR RESEARCH QUESTIONS
How could we empower tourists to confidently select a dish abroad despite cultural and language barriers?
How could we make the access to the relevant information seamless?
THE CONCEPT
Our system is focused on enhancing menu translations by providing additional cultural insights and by ensuring users can seamlessly access this information.
By utilising Augmented Reality (AR), our app enriches traditional menu translations with additional information including basic and extended information elements, making menus more accessible and engaging for foreign users. We chose AR to maintain the tangible benefits of paper menus, such as ease of browsing and facilitating communication, while seamlessly integrating expanded content. As the user points their smartphone camera at a menu, the app recognizes text through optical character recognition (OCR) technology and matches it with our database entries (that can be extracted from trusted sources).
BASIC INFORMATION ELEMENTS
- Translation of the menu ttem text, concise description and representative images of the dish.
EXTENDED INFORMATION ELEMENTS
- Dish Categories: Indicates whether a dish is considered traditional, exotic, or similar to the user’s home cuisine, helping diners navigate their culinary preferences and comfort levels. It describes the rationale for this categorisation.
- Culture-Specific Information from User Comments: Incorporates insights extracted from user comments about a dish.
1. Scan paper menu.
2. Explore AR Overlay.
3. Point to item to see preview card.
4. Swipe up for more details.
USER TESTING
We evaluated a prototype of our application with 26 international participants.
Feedback highlighted the app’s effectiveness in reducing uncertainty around menu choices and enhancing interest in local cuisine, increasing curiosity and willingness to try new dishes, crediting the cultural context offered by the app. The UI design and interaction worked well in general, as it enabled participants to seamlessly explore the menu. Participants also mentioned areas for improvement, e.g., translations sometimes lacked depth, missing out on the nuances of dishes. There’s was also a strong interest in expanding the cultural content to include stories and the social context of cuisines. Feedback also highlighted the need for improving the app’s augmented reality performance, particularly addressing slow response times and recognition issues in various lighting conditions.
LESSONS LEARNED
Innovation arises from interdisciplinary collaboration.
The work on this project has revealed the impact that design and technology can have on cultural exploration. One of the key lessons learned is the importance of interdisciplinary collaboration in addressing complex challenges at the intersection of technology, culture, and user experience. The project highlighted the value of integrating sociological insights with UX design principles and cutting-edge NLP and retrieval technology to create a solution that is not only innovative but also grounded to users’ desires for authentic cultural experiences.
LEARN MORE.
Restaurant Menu Understanding: Illustrating the Need for Culturally Augmented Translation
Gallo, D., Willamowski, J., Wisatekaew, Y., Bruyat, A., Grasso, M.A. 2020. International Conference on Human-Computer Interaction with Mobile Devices and Services (Mobile HCI). Short paper →