Mobile apps are becoming more popular in society whether professional, entertainment or simply communication fields since they can provide a collection of useful features in daily activities. One of the essential requirements for an application to be successful in the market is the Graphical User Interface which encompasses most of the user interaction. Thus, the GUI test is an important practice to control the issues that affect the interaction and user experience. However, those tests face some challenges like high-cost maintenance in UI changes. This paper presents a tool that can extract information from mobile GUI interfaces. The information extracted includes the GUI components type, layout hierarchy, and belonging scenario, as well as component position in the screen, component image, and pixel map. Such a pixel map provides the component class of each image pixel to enable semantic segmentation training schemes. Such a dataset may support the research on machine applications associated, for instance, with the identification of changes employing the user interface.
Paper
Full text
Pic2Translator: A tool to extract mobile GUI components for machine learning and tasks
Semantic Scholar · Computer Science · 2021
Abstract
Mobile apps are becoming more popular in society whether professional, entertainment or simply communication fields since they can provide a collection of useful features in daily activities. One of the essential requirements for an application to be successful in the market is the Graphical User Interface which encompasses most of the user interaction. Thus, the GUI test is an important practice to control the issues that affect the interaction and user experience. However, those tests face some challenges like high-cost maintenance in UI changes. This paper presents a tool that can extract information from mobile GUI interfaces. The information extracted includes the GUI components type, layout hierarchy, and belonging scenario, as well as component position in the screen, component image, and pixel map. Such a pixel map provides the component class of each image pixel to enable semantic segmentation training schemes. Such a dataset may support the research on machine applications associated, for instance, with the identification of changes employing the user interface.