Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility

  title={Interaction Proxies for Runtime Repair and Enhancement of Mobile Application Accessibility},
  author={Xiaoyi Zhang and Anne Spencer Ross and Anat Caspi and James Fogarty and Jacob O. Wobbrock},
  journal={Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems},
We introduce interaction proxies as a strategy for runtime repair and enhancement of the accessibility of mobile applications. Conceptually, interaction proxies are inserted between an application's original interface and the manifest interface that a person uses to perceive and manipulate the application. This strategy allows third-party developers and researchers to modify an interaction without an application's source code, without rooting the phone, without otherwise modifying an… 

Robust Annotation of Mobile Application Interfaces in Methods for Accessibility Repair and Enhancement

This research develops methods for robust annotation of mobile app interface elements based in screen identifiers, element identifiers, and screen equivalence heuristics for use in runtime interface modification.

Improving Smartphone Accessibility with Personalizable Static Overlays

This work presents an approach that superimposes a virtual overlay to all other interfaces ensuring interface consistency by re-structuring how content is accessed in every screen, and split the screen, dedicating half to a configurable set of static options mimicking always available physical buttons regardless of context.

Accessible or Not? An Empirical Investigation of Android App Accessibility

An automated app page exploration tool, named Xbot, is proposed to facilitate app accessibility testing and automatically collect accessibility issues by leveraging the instrumentation technique and static program analysis, and an empirical study from the perspective of accessibility issues themselves is carried out to investigate novel characteristics of accessibility Issues.

Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels

This paper trained a robust, fast, memory-efficient, on-device model to detect UI elements and built Screen Recognition to generate accessibility metadata to augment iOS VoiceOver, demonstrating that this approach improves the accessibility of existing mobile apps, enabling even previously inaccessible apps to be used.

Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinements

This work proposes a system, Reflow, which automatically applies small, personalized UI adaptations, called refinements —to mobile app screens to improve touch efficiency, and demonstrates that Reflow’s refinements useful UI adaptations to improvetouch interactions.

Automated Repair of Size-Based Inaccessibility Issues in Mobile Applications

This paper's empirical evaluation showed that the approach was able to successfully resolve 99% of the reported Size-Based Inaccessibility issues and received a high approval rating in a user study of the appearance of the repaired user interfaces.

X-Ray: Screenshot Accessibility via Embedded Metadata

X-Ray, a system that captures and embeds the semantics of the underlying content into images, suggests a method for embedding accessibility metadata into otherwise inaccessible formats, enabling them to retain the more accessible representations that are present at capture time.

StateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible

The technical evaluation shows that StateLens can accurately reconstruct interfaces from stationary, hand-held, and web videos, and, a user study of the complete system demonstrates that State Lens successfully enables blind users to access otherwise inaccessible dynamic touchscreens.

Code and Contribution in Interactive Systems Research

This position paper considers the case where code is closely linked to contribution, and explores cases where the link is less direct, and examines several forms of interactive systems contribution based in a differentiation of what a system accomplishes, versus (2) how it accomplishes that.

RePlay: Touchscreen Interaction Substitution Method for Accessible Gaming

RePlay overcomes key limitations of existing approaches, allowing users to quickly access all app interface elements, even those not visible to the OS (a common situation especially with games) through personalized mapping of interface elements to alternative inputs, such as external switches or non-verbal vocal sounds, hence adapting to users with diverse abilities.



Cracking the cocoa nut: user interface programming at runtime

Scotty, a prototype implementation for Mac OS X Cocoa that enables developers to modify existing applications at runtime, is described and a collection of interaction and functional transformations on existing off-the-shelf applications are demonstrated.

A11y Attacks: Exploiting Accessibility in Operating Systems

This paper presents the first security evaluation of accessibility support for four of the most popular computing platforms: Microsoft Windows, Ubuntu Linux, iOS, and Android, and identifies twelve attacks that can bypass state-of-the-art defense mechanisms deployed on these OSs.

User interface toolkit mechanisms for securing interface elements

This work describes a security-aware user interface toolkit architecture that secures interface elements while providing developers with the flexibility and expressivity traditionally desired in a user interfaces toolkit.

Pixel-based methods for widget state and style in a runtime implementation of sliding widgets

Three new sets of methods are presented: methods for pixel-based modeling of widgets in multiple states, methods for managing the combinatorial complexity that arises in creating a multitude of runtime enhancements, and methods for styling runtime enhancements to preserve consistency with the design of an existing interface.

Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure

This work presents Prefab, a system for implementing advanced behaviors through the reverse engineering of the pixels in graphical interfaces,formed by how user interface toolkits paint interfaces, which features a separation of the modeling of widget layout from the recognition of widget appearance.

Associating the visual representation of user interfaces with their internal structures and metadata

This paper presents a hybrid framework, PAX, which associates the visual representation of user interfaces and their internal hierarchical metadata, i.e. the pixels, with the content, role, and value of GUI widgets.

A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces

This work implements Grossman and Balakrishnan's Bubble Cursor, the fastest general pointing facilitation technique in the literature, and obtains the necessary knowledge of interface targets using a combination of pixel-level analysis and social annotation.

Past, present, and future of user interface software tools

This article considers cases of both success and failure in past user interface tools, and extracts a set of themes which can serve as lessons for future work.

Accessmonkey: a collaborative scripting framework for web users and developers

The Accessmonkey framework is introduced, a common scripting framework that web users, web developers and web researchers can use to collaboratively improve accessibility and advances the idea that Javascript and dynamic web content can be used to improve inaccessible content instead of being a cause of it.

CAN: composable accessibility infrastructure via data-driven crowdsourcing

A crowdsourcing infrastructure that collects web accessibility issues and their fixes, dynamically composes solutions on-the-fly, and delivers the crowd-sourced content as teaching materials for web developers to learn how to build accessible components in a cost-efficient manner.