Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure

@article{Dixon2010PrefabIA,
  title={Prefab: implementing advanced behaviors using pixel-based reverse engineering of interface structure},
  author={Morgan Dixon and James Fogarty},
  journal={Proceedings of the SIGCHI Conference on Human Factors in Computing Systems},
  year={2010}
}
  • M. Dixon, J. Fogarty
  • Published 10 April 2010
  • Computer Science
  • Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Current chasms between applications implemented with different user interface toolkits make it difficult to implement and explore potentially important interaction techniques in new and existing applications, limiting the progress and impact of human-computer interaction research. We examine an approach based in the single most common characteristic of all graphical user interface toolkits, that they ultimately paint pixels to a display. We present Prefab, a system for implementing advanced… 

Figures from this paper

Content and hierarchy in pixel-based methods for reverse engineering interface structure
TLDR
This work presents new methods for hierarchical models of complex widgets, real-time interpretation of interface content, and real- time interpretation of content and hierarchy throughout an entire interface.
Pixel-based reverse engineering of graphical interfaces
  • M. Dixon
  • Computer Science
    UIST '13 Adjunct
  • 2013
TLDR
This dissertation proposes pixel-based interpretation to enable modification of interfaces without their source code and independent of their underlying toolkit implementation to transcending the rigidity and fragmentation of modern interfaces.
Pixel-based methods for widget state and style in a runtime implementation of sliding widgets
TLDR
Three new sets of methods are presented: methods for pixel-based modeling of widgets in multiple states, methods for managing the combinatorial complexity that arises in creating a multitude of runtime enhancements, and methods for styling runtime enhancements to preserve consistency with the design of an existing interface.
Using graphical representation of user interfaces as visual references
TLDR
This thesis investigates using graphical representations of user interfaces as direct visual references to support various kinds of applications and proposes to develop a visual history system that enables users to search and browse what they have seen on their computer screens.
A general-purpose target-aware pointing enhancement using pixel-level analysis of graphical interfaces
TLDR
This work implements Grossman and Balakrishnan's Bubble Cursor, the fastest general pointing facilitation technique in the literature, and obtains the necessary knowledge of interface targets using a combination of pixel-level analysis and social annotation.
Reverse Engineering Mobile Application User Interfaces with REMAUI (T)
  • T. Nguyen, C. Csallner
  • Computer Science, Art
    2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE)
  • 2015
TLDR
The first technique to automatically Reverse Engineer Mobile Application User Interfaces (REMAUI) is introduced, which identifies user interface elements such as images, texts, containers, and lists, via computer vision and optical character recognition (OCR) techniques.
Prefab: What if Every GUI were Open-Source?
TLDR
This position statement examines a vision in which anybody can modify any GUI of any application, similar to a scenario where every GUI of every application is open-source.
ShowMeHow: Translating User Interface Instructions Between Similar Applications
TLDR
UI translation interfaces are introduced which enable users to locate commands in one application using the interface language of another application, and to repurpose tutorials between similar applications.
Prefab layers and prefab annotations: extensible pixel-based interpretation of graphical interfaces
TLDR
This work introduces a new toolkit for pixel based enhancements, focused on two areas of support: Prefab Layers and Prefab Annotations, which help developers overcome subtle but critical dependencies between code and data.
P2A: A Tool for Converting Pixels to Animated Mobile Application User Interfaces
  • S. Natarajan, C. Csallner
  • Computer Science, Art
    2018 IEEE/ACM 5th International Conference on Mobile Software Engineering and Systems (MOBILESoft)
  • 2018
TLDR
The P2A tool adopts computer vision techniques for developing animated mobile applications that infers from mobile application screen designs the user interface portion of an application's source code and other assets that are ready to be compiled and executed on a mobile phone.
...
1
2
3
4
5
...

References

SHOWING 1-10 OF 32 REFERENCES
Phosphor: explaining transitions in the user interface using afterglow effects
TLDR
A framework of transition designs for widgets, icons, and objects in drawing programs that supports both types of users, including those who already understood the transition and those who may have been distracted.
Systematic output modification in a 2D user interface toolkit
TLDR
This paper uses a set of “hooks” to undo output transformations when input is to be dispatched and addresses issues of input by examining a number of extensions to the toolkit input subsystem to accommodate transformed graphical output.
User interface façades: towards fully adaptable user interfaces
TLDR
This paper presents User Interface Façades, a system that provides users with simple ways to adapt, reconfigure, and re-combine existing graphical interfaces, through the use of direct manipulation techniques.
Supporting dynamic downloadable appearances in an extensible user interface toolkit
TLDR
New user interface toolkit techniques designed to support dynamic, even downloadable, appearance* changes for graphical user interfaces are described.
Implementing interface attachments based on surface representations
TLDR
An architecture for supporting interface attachments - small interactive programs which are designed to augment the functionality of other applications, which supports the creation of aset of uniform services that can be applied across a more diverseset of applications than traditional approaches.
Past, present, and future of user interface software tools
TLDR
This article considers cases of both success and failure in past user interface tools, and extracts a set of themes which can serve as lessons for future work.
Side views: persistent, on-demand previews for open-ended tasks
TLDR
Results from field studies of expert users that motivated this work are summarized, then the design of Side Views is discussed in detail, showing how Side Views' design affords their use as tools for clarifying, comparing, and contrasting commands; generating alternative visualizations; experimenting without modifying the original data; and as tools that support the serendipitous discovery of viable alternatives.
Programming by example: visual generalization in programming by example
TLDR
Here, a different, and radical, approach is explored—using the visual properties of the interaction elements themselves, including size, shape, color, and appearance—to describe user intentions.
Visual Generalization in Programming by Example
WinCuts: manipulating arbitrary window regions for more effective use of screen space
TLDR
A novel interaction technique that allows users to replicate arbitrary regions of existing windows into independent windows called WinCuts, where each WinCut is a live view of a region of the source window with which users can interact.
...
1
2
3
4
...