Lightweight Neural App Control

We would like to thank the reviewer for their review and address their concerns below. 1. Accuracy Improvement: While we acknowledge that the improvement in accuracy of LiMAC compared to our fine-tuned Florence does not fully solve the problem, we still believe it represents a meaningful and valuable step forward (and in most cases with large improvements in computational efficiency/speed). We have carefully evaluated LiMAC and provided a thorough analysis of its strengths and weaknesses, which we hope highlights the contribution of our work in advancing the field. Regarding the “more parameters”, while the overall framework does have more parameters, we believe that statement does not paint the full picture, as these parameters do not simply compose a larger network (i.e., not all parameters are called at the same time). For example, if one was to consider action type selection as its own separate task, our 520M AcT model outperformed larger models. Moreover, the parameters of the VLM are only used when the VLM is called, which only occurs for open-app or input-text actions, making up 13.9% of the total actions for AndroidControl and 10.6% for AitW respectively. 2. Online Evaluation: We definitely agree that online evaluation is a valuable addition and currently missing from our work, and is an aspect that we aim to address in the future. We believe that fully autonomous, online, and generalisable app agent control, is far from solved [1,2]. Nevertheless, the focus of our study differs from that of DigiRL [3]. In DigiRL, the authors train separate models for different goal categories (General and Webshop) and evaluate on a relatively small subset of goals (the first 96 from the train and test set). In contrast, our work utilises the AndroidControl dataset, which contains more than 13K distinct goals, as well as our subset of AitW, with just under 9K goals. Generalising fine-tuned models in online settings remains an open problem, as highlighted in [1,2] which demonstrate that fine-tuned models perform poorly on a large set of online tasks. 3. Novelty: *We have added an acknowledgement of [4] and [5] in Sec 3.1, discussing that they have also explored embedding individual UI elements.* We would argue that the architecture as a whole is novel even if some of the individual components have been explored before. Indeed, Li et al. [4] (which we thank the reviewer for bringing to our attention) and Rawles et al (AitW) [5] have explored embedding individual UI elements, however we still believe we look deeper into the matter (e.g., using fine-tuned CLIP for the embedding accompanied with ablation studies, using concatenations of embeddings of CLIP and BERT, etc). Furthermore, as the reviewer noted, we use the hidden states of the transformer corresponding to these embeddings for a contrastive learning objective (which to the best of our knowledge has not been explored before). The gated architecture is obviously not a novel idea, and it is pretty natural to the task, but we still believe reporting its capabilities in this context is valuable. We hope this clarifies our approach and contributions, and we thank the reviewer again for their constructive suggestions. We also appreciate the comments on the on-device implementation. [1] Chen et al. SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation [2] Zhang et al. LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation [3] Bai et al. DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning. [4] Li et. al. Mapping Natural Language Instructions to Mobile UI Action Sequences [5] Rawles et al Android in the Wild: A Large-Scale Dataset for Android Device Control

We thank the reviewer for their valuable feedback. *We now include a paragraph with clarifications on the memory footprint and the use of the term “Lightweight” in Sec 3, paragraph 2.* We acknowledge the concern about memory usage, as our approach does require an additional model to be loaded. While we did not claim a smaller memory footprint, we have now clarified this in Section 3 of the paper. That being said, there is also some nuance to this discussion, as memory operates differently from compute: its cost is small as long as the device has enough capacity. Most modern devices can accommodate transformer-based models up to ~7B parameters [1, 2], though models at the higher end of this range often require aggressive quantization (which impacts performance). Our small VLM + 500M model should be within that limit, even allowing overhead for the OS and open apps. Our use of "lightweight" refers instead to the fact that we i) kept the overall parameter size manageable and likely to fit on a typical modern device (no need for unloading models), leaving some space for the operating system and other apps, ii) are more efficient in our computations (e.g., no auto-regressive generation in AcT), iii) are using less parameters on average for computation (which should translate to less required floating point operations), and iv) simply having faster inference speed. We have now clarified this in Sec 3. We also thank the reviewer for mentioning ActionBert, as it’s indeed relevant and could be explored in future work. [1] Li et al. Large Language Model Performance Benchmarking on Mobile Platforms: A Thorough Evaluation. [2] Laskaridis et al. MELTing point: Mobile Evaluation of Language Transformers.

Thank you for addressing this point. I am raising my score to a 8. I think this paper is well written, spells out its contributions clearly, provides careful analysis with ablations, and makes a new contribution towards actually deploying models on-device. Note: > and T3A is used for text generation In the AitW dataset There is a missing period after generation.

We would like to thank the reviewer for their review and address their concerns below. 1. Positional Encodings for Nesting: The use of positional embeddings allows us to capture the relative positioning of UI elements, helping AcT understand which elements are nearby or grouped together. Without positional embeddings, the order of elements on the screen would be less meaningful. In our approach, nesting information is implicitly encoded in the UI icons themselves, as these icons often have overlapping pixels. By combining this spatial information with the positional embeddings, AcT is able to effectively distinguish between nested and non-nested elements. We attempted to incorporate depth embeddings into the nested structure, combining these with positional embeddings to precisely represent the nesting of UI elements. However, this integration did not yield any noticeable improvement in performance. We hope this clarifies the role of positional embeddings in handling nested UI elements. 2. Distinct Action Types: *We have updated the paper to reflect this more clearly, and have added a link to the relevant appendix in Section 3.3.* The distinct action types are outlined in Appendix A of the paper, specifically Table 5. Though there are only 10 distinct action types in AndroidControl, and 8 in AitW, there are a combined 11 distinct actions. The distinct actions are: (1) open-app, (2) click, (3) long-press, (4) input-text, (5-8) scroll (right/left/up/down), (9) navigate-home, (10) navigate-back, and (11) wait. open-app, wait, and long-press do not feature in AitW, while navigate-home does not feature in AndroidControl. 3. Florence Fine-Tuned or Not? Yes, it is fine-tuned. The Florence2 baseline refers to the same Florence2 model used within our LiMAC framework. Practically, we fine-tune Florence2 only once for each dataset, and we use it both as part of LiMAC as well as to compare against it. 4. End-to-End Accuracy: *We have improved the wording of this explanation in our paper, section 4.2.* Overall accuracy is dependent on both the action type and the action specifications. When predicting a “wait” action, if the action type is predicted correctly, this immediately yields a positive result for that timestep. However, when predicting actions with additional parameters, such as “input-text”, even if the action type is predicted correctly, the overall result may be negative if the VLM fails to fill the “text” input field correctly. Using an example to try and illustrate this, take an episode with 10 steps, where the correct action is “wait” for 5 steps and “input-text” for the other 5. Predicting “wait” for all 10 steps will yield an overall accuracy of 0.5. However, predicting “input-text” for all 10 steps will yield an accuracy of 0.5 * (probability of the VLM to give the correct text input). If this VLM accuracy is 80%, the overall accuracy will be 0.5 * 0.8 = 0.4. Minor Issues: Thank you for pointing these out! These have now been fixed. 5. Running on Smartphone: We have not yet explored deploying the model on smartphones. Instead, we focus on training and comparing agents in simulated environments, with an understanding that limited computational capacity is a real and key constraint in mobile devices. By leveraging open-source datasets, we aim to improve performance and efficiency in these settings. The smaller size of AcT and its faster response times, however, highlight its promising potential for smartphone applications in the future.

We would like to thank the reviewer for their review and address their concerns below. Weaknesses: 1. Limited Datasets and Tasks: We appreciate the reviewer’s concern regarding the limited size of datasets in our study. While we agree that a larger dataset would provide valuable insights, our work focuses on open-source data, and gathering additional data presents significant (and out of scope) challenges. Specifically, collecting data through emulator interaction is difficult, because Android emulators typically do not offer reward functions to assess task success. AndroidWorld [1] is the only exception we are aware of, though it is limited to just 116 tasks. We are actively working to address this limitation, in future work, by solving the challenges associated with Android emulators, which will allow for a wider and more diverse set of tasks. This limitation is mentioned in the conclusion of the paper. 2. More Analysis of Failure Modes: *We thank the reviewer for suggesting a deeper analysis of failure modes is a valuable suggestion. We have added such an analysis in the Appendix D3 (Figure 4).* Questions: 1. Handling of Dynamic Elements and Changing Interfaces: Regarding the handling of dynamic elements and shifting interfaces, retraining would be required when transitioning to completely different operating systems, such as from Android to iOS. However, within the same OS, we expect the accuracy drop across different versions to be minimal, having observed only slight performance variations between train and test sets, and with AitW for example containing a range of Android OS versions and phone models. 2. Impact of VLM Size: We believe that training larger Vision-Language Models (VLMs) could potentially improve performance. However, as suggested in the AndroidControl paper [2], achieving 95% accuracy may require up to 60 million high quality episodes, using the PaLM-S model, which may contain over 100B parameters. While scaling up models is crucial, it is also important to recognise the significant resources required to reach such levels of performance (and that is why believe novel research directions that do not focus simply on scale are valuable). 3. Error handling and Recovery: *We now briefly discuss and acknowledge this limitation in Sec 6.* In the current scope of our work, we focus primarily on the core functionalities and performance evaluation of LiMAC. While comprehensive mechanisms for error recovery are essential, they were not the primary focus. We recognise the importance of robust error handling and recovery processes, and we intend to address these in future work. We also agree that safeguarding personal data is a concern, and while the current version of LiMAC does not specifically address this, we have included this as a direction of future research in the conclusion. [1] Rawles et al. AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. [2] Li et al. On the Effects of Data Scale on Computer Control Agents.

We would like to thank the reviewer for their review and reply to their concerns below. 1. Android Focus: We focus on Android because it offers two open-source datasets and has a significant share of the global market as a mobile phone OS. These factors make our findings highly relevant and broadly applicable. We also want to emphasise that our methodology is flexible and can be adapted to other operating systems, such as iOS, once similar datasets become available. 2. UI Element Scalability: *We have included a study in Appendix D4 (Figure 5) that examines how the number of UI elements affects the ratio of successful to unsuccessful action predictions.* In the evaluated datasets, the number of UI elements per observation varies from 1 to 290, with most observations having fewer than 100 UI elements. While it's possible for an observation to exceed this range, potentially leading to out-of-distribution issues, we believe that the current span adequately encompasses the bulk of practical scenarios encountered in app control. This ensures that our model is capable of managing a diverse range of tasks. 3. Fixed Action Space: *We have updated our description of the action space in Appendix A* While the action space is fixed, it is the same as the AitW and AndroidControl datasets (where it is also fixed) and aligns with prior work on app agents and emulators, such as SeeAct [1], AndroidWorld [2], AppAgent [3], MobileAgent [4], DigiRL [5]. These actions are grounded by how a human interacts with the phone (e.g., click, or swipe). This fixed action space covers the vast majority of actions available on Android devices, and we believe it provides a robust representation of real-world app control behaviour. 4. Simulated Data vs. Real-World Devices: We acknowledge the concern that the approach may face challenges in real-world settings. However, this is a broader issue in the field, not unique to our approach and requires significant amounts of work. By leveraging open-source datasets, we aim to improve performance and efficiency in the simulated setting as a foundation. The smaller size of AcT and its faster response times highlight its promising potential for on-device smartphone applications in the future. [1] Zheng et al. Gpt-4V(ision) is a Generalist Web Agent, if Grounded. [2] Rawles et al. Androidworld: A Dynamic Benchmarking Environment for Autonomous Agents. [3] Yang et al. Appagent: Multimodal Agents as Smartphone Users. [4] Wang et al. Mobile-agent: Autonomous Multi-modal Mobile Device Agent with Visual Perception. [5] Bai et al. DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning.