UI Toolkits !

HCID 520 User Interface Software & Technology

http://www.cryptonomicon.com/beginning.html

Xerox Alto 1973

Command Line (UNIX shell, DOS prompt) Interaction driven by system User prompted for input when needed Text-based input and output !Event-Driven Interfaces (GUIs) Interaction driven by user System waits for user action and responds Pointing & text input, graphical output More complex architecture & development

Evolution of User Interfaces

Component Model

Encapsulate interactive components (widgets) Component library (button, slider, container) Interface built as a hierarchy of components !Widgets drawn by underlying graphics library !Input event generation and dispatch Historically mouse & keyboard, now touch, … !Bounds management & damage/redraw Model geometry, redraw updated regions only

Component Model

Java Swing Widgets

User Interface ComponentsLabel TextArea

ButtonsEach component is a clipped 2D canvas with its own coordinate system.

User Interface Components

Each component (widget) is an object with: !A bounding box !A paint method for drawing itself Drawn in the component’s coordinate system !Callbacks to process input events

public void paint(Graphics g) { g.fillRect(…); // interior g.drawString(…); // label g.drawRect(…); // outline }

Containment HierarchyWindow

Panel

TextArea Panel

Button Button

Label

Component LayoutWindow

Panel

TextArea Panel

Button Button

Label

Border Layout (direct placement)

NORTH

CENTER

SOUTH

strutssprings

“Struts and Springs” (simple

constraint-based layout)

Each container is responsible for positioning its contents.

Event-Driven Architecture

User input is modeled as events that must be processed by the system. !Hardware Events Mouse: move, button-down, button-up Keyboard: key-down, key-up !Inferred Events Mouse: click, double-click, enter, exit, drag Keyboard: key-press Window: move, resize

Events

Events encapsulate the information needed for components to react to input or changes. !Event Type (mouse moved, key down, etc) Event Target (the input component) Timestamp (when did event occur) Modifiers (Ctrl, Shift, Alt, etc) Event Content: Mouse: x,y coordinates, button pressed, # clicks Keyboard: which key was pressed

Anatomy of an Event

Event Dispatch Loop

Event Queue List of input events

Event Loop (runs in dedicated thread)

Remove next event from queue Determine event type Find proper component(s) Invoke callbacks on components Repeat, or wait until event arrives

Component Invoked callback method Update application state Request repaint, if needed

Mouse moved (t0,x,y)

Window

Panel

TextArea Panel

Button Button

Label

Event Queue Mouse moved (t0,x,y) Mouse pressed (t1,x,y,1) Mouse dragged (t2,x,y,1) Key typed (t3, ‘F1’) … (queue and dispatch incoming events in a dedicated thread)

/* callback for TextArea */ public void mouseMoved(e) { // process mouse moved event }

Event Dispatch

Go to http://mhcid.washington.edu !Open the JavaScript console, enter: ! var h = document.querySelector("#header"); h.addEventListener("click", function(event) { console.log(event); });!Click the header image at the top of the page. !Inspect the events in the JavaScript console.

Demo: Browser DOM Events

Event Capture Process event either as it moves down or up the component hierarchy? Default is up. Why? !

Event Minutiae

Window

Panel

TextArea Panel

Button Button

Label

Event Queue Mouse moved (t0,x,y) Mouse pressed (t1,x,y,1) Mouse dragged (t2,x,y,1) Key typed (t3, ‘F1’) … (queue and dispatch incoming events in a dedicated thread)

/* callback for TextArea */ public void mouseMoved(e) { // process mouse moved event }

Event Capture (Down)

Window

Panel

TextArea Panel

Button Button

Label

Event Queue Mouse moved (t0,x,y) Mouse pressed (t1,x,y,1) Mouse dragged (t2,x,y,1) Key typed (t3, ‘F1’) … (queue and dispatch incoming events in a dedicated thread)

Event Bubbling (Up)

Event Capture Process event either as it moves down or up the component hierarchy? Default is up. Why? !Throttling & Debouncing You don’t always want to process every event, both for performance and experience reasons. !Examples: Flurry of mouse drags > process last event Rapid enter/exit > pause to process exit

Event Minutiae

Model View Controller

Architecture for interactive applications Introduced by Smalltalk developers at PARC Partitions app to be scalable & maintainable

Model View Controller (MVC)

View

Controller

Model

Information the app is trying to manipulate Circuits: logic gates and connecting wires Drawing: geometry and color of shapes

Model

View

Controller

Model

Implements a visual display of the model Apps may have multiple views for one model Example: shape & numerical views of drawing

View

View

Controller

Model

Implements a visual display of the model !When model changes, each view must be notified so that it can update itself.

View

View

Controller

Model

Processes all input events from the user Query view to interpret events (e.g., selection) Call methods of model to make changes Model updates & notifies views

Controller

View

Controller

Model

Example Drawing Application

Blue circles: 3 Red squares: 2

The Controller handles input…

Blue circles: 3 Red squares: 2

… updates the Model …

Blue circles: 3 Red squares: 2

Click!

… and Views update in response.

Blue circles: 4 Red squares: 2

Controller must contact view to interpret what some user events mean. Example: Mouse(x, y) -> selected object !“Pattern of behavior in response to user events (controller issues) is independent of visual geometry (view issues).”

Relationship of View & Controller

View and controller are tightly intertwined Lots of communication between the two Occur in pairs: each view needs a controller Some architectures combine into a single class

Combining View & Controller

View

ControllerModel

Why MVC?

Putting MVC into one class does not scale Model may have more than one view Each view different, update on model change !Separation aids maintenance & extensibility Easy to change view or add a new view later Models can be extended; old views still work Flexibility of input handling when using separate controllers (e.g., mouse vs. touch)

Why MVC?

Adding Views Later

Blue circles: 3 Red squares: 2

Erase and Redraw Using background color to erase fails Draw shape in new position -> loses ordering !Move in model and then redraw everything Change position of shapes in model Model keeps shapes in a desired order Tell all views to redraw themselves in order Slow for large / complex drawings

Changing the Display

Move a red square…

Blue circles: 3 Red squares: 2

Erase at old position, draw at new

Blue circles: 3 Red squares: 2

Register “damage”, redraw region

Blue circles: 3 Red squares: 2

View informs window system of areas that need to be updated (i.e., damaged) !Does not redraw these areas yet… !Window system batches updates & clips them to visible portions of window !On next event cycle, window system invokes repaint, passes region that need updates !Critical for many 2D graphics systems, typically not used in 3D interfaces. Why?

Damage / Redraw

Cascading Style Sheets

Traditional GUI tools often make assumptions: Rectangular (non-overlapping?) components Difficult to style graphic elements !Many modern frameworks are more flexible, using a scenegraph to model the display. Ex: the browser document object model (DOM). !Decouple content from appearance using separate stylesheets to specify design.

Technical Limitations

A language for styling an HTML document W3C standard, first introduced in 1996 !Goal: separate content from presentation Specify layout, spacing, colors, fonts, … Share formatting across pages on a site Enable various formats for a single page !Why “cascading”? Multiple style definitions may apply to an element, ordered (“cascaded”) by priority.

Cascading Style Sheets

!p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333;}

CSS Example

!p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333;}

CSS ExampleSelector style p (paragraph) elements

!p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333;}

CSS ExampleSelector

Property

!p { margin-bottom: 1em; font-family: Helvetica, Arial; font-size: 16px; line-height: 24px; color: #333333;}

CSS ExampleSelector

Property Value … or #333, or rgb(51,51,51)

Tag (select all elements of a certain type) a, p, h1, ul, li, etc !

CSS Selectors

Tag (select all elements of a certain type) a, p, h1, ul, li, etc !Id (select by unique identifier, prefix #) <div id="logo"> —> #logo !

CSS Selectors

Tag (select all elements of a certain type) a, p, h1, ul, li, etc !Id (select by unique identifier, prefix #) <div id="logo"> —> #logo !Class (select by shared identifier, prefix .) <div class="c1 c2"> —> .c1 !

CSS Selectors

Tag (select all elements of a certain type) a, p, h1, ul, li, etc !Id (select by unique identifier, prefix #) <div id="logo"> —> #logo !Class (select by shared identifier, prefix .) <div class="c1 c2"> —> .c1 !These selectors can be combined: div.headlines

CSS Selectors

For multiple selections, use a comma div.headlines, div.news { … }Selects both div.headlines and div.news elements !

CSS Selector Combinations

For multiple selections, use a comma div.headlines, div.news { … }Selects both div.headlines and div.news elements !For nested selections, use a space div.headlines ul li { … }Selects all li within a ul within a div.headlines !

CSS Selector Combinations

For multiple selections, use a comma div.headlines, div.news { … }Selects both div.headlines and div.news elements !For nested selections, use a space div.headlines ul li { … }Selects all li within a ul within a div.headlines !For direct descendants, use ‘>’ div.headlines > ul li { … }Now the ul must be the direct child of div.headlines

CSS Selector Combinations

CSS Box Model

block<div>!!inline<span>!inline-block!!none

CSS display property

<body> ! <div id=“left”> Sidebar column content (assume max 400px tall) </div> ! <div id=“right”> Main column content (variable height) </div> ! <div id=“footer”> Footer column content (assume 50px tall) </div>

</body>

Specify a 2-Column Layout

<body> —> select with body <div id=“left”> —> select with #left Sidebar column content (assume max 400px tall) </div> ! <div id=“right”> —> select with #right Main column content (variable height) </div> ! <div id=“footer”> —> select with #footer Footer column content (assume 50px tall) </div>

</body>

Specify a 2-Column Layout

Layout with no styling

Layout with absolute positioning, 250 pixel sidebar

What are some shortcomings of this layout? Why did we include body { position: relative }?

Layout with absolute positioning, without position:relative

position: relative sets the coordinate space for absolutely positioned elements. In this example the position [0,0] is relative to the browser window, not the body tag.

Layout with float: left and margin-left

What are some shortcomings of this layout? Why did we include #footer { clear: both }?

Layout with float: left and margin-left, without clear: both

clear: both tells the browser layout engine to clear any floats (both left and right) before placing the element. !Next, can we make the sidebar resize automatically in response to the window width?

Float layout using percentages

What if we want to ensure the width of the left sidebar never falls below a minimum size?

Here’s one attempt at setting a minimum width. Why is this unsatisfactory?

Float layout using percentages and minimum width

Float layout using percentages

Let’s go back to our previous design… How can we limit the maximum width of the page?

Float layout using percentages and body max-width

Use the max-width property on the body element.

Adding automatic margins

margin-left: auto; margin-right: auto centers the body.

margin (-left, etc) font padding (-left, etc) font-family border (-left, etc) font-size border-radius font-weight overflow (-x/-y) font-variant background-color line-height background-image text-transform position text-align float vertical-align (max-/min-) width, height display

Useful CSS Properties

CSS preprocessors (LESS / SASS) Add support for variables (e.g., $baseColor) Perform computation, unit conversion Enable better reuse, less repetition Now commonly used in web applications !HTML/CSS frameworks (e.g., Bootstrap) Provide better defaults “out of the box” More dynamic components (popup menus) Responsive design; mobile friendly

CSS Power Tools

UI Tools Discussion

Brief UI tool history, establish vocabulary in wide use today, and chart future trajectories. !At any point did they “miss the mark”? !

Myers, Hudson & Pausch

Brief UI tool history, establish vocabulary in wide use today, and chart future trajectories. !At any point did they “miss the mark”? !Web: subsuming many desktop applications !Design vs. Dev: In practice often done by teams with varied skills (interaction design, graphic design, implementation) !3D: Mostly limited to games? BUT fabrication now likely to drive new 3D design tools…

Myers, Hudson & Pausch

Not just for desktop GUI widgets & events Mobile applications (iOS, Android) Information visualization (d3.js, Protovis, Prefuse) Tangible user interfaces (Papier-Maché) Gestural user interfaces (Kinect API) Speech user interfaces (MS Speech API) Proxemic user interfaces (Proximity Toolkit) !Many exhibit similar tensions around expressiveness, efficiency and ease-of-use

Toolkits Abound…

Simplify & accelerate UI development Lower the threshold of specification difficulty Raise the ceiling of what can be expressed But hard to do both! !Establish path of least resistance Makes common / successful designs easy Makes unusual / ineffective designs harder Can limit & shape what we conceive / design !Enforce consistency of design Maintain familiar widgets and interactions

Toolkit Benefits & Shortcomings

Timeliness & Future Predictions Threshold/Ceiling, Expressivity vs. Usability Varied Display Sizes (from Walls to Watches) Personalization User Attention & Cognitive Load End User Programming Constraints / Model-Driven / Automatic Design Security

Discussion Question Topics

The authors of the main reading mention the possibility of tools that not only enable design systems that have high usability, but that actual promote or help ensure that they are. How might this be taken further in the future to help prevent designs that cause users frustration or discomfort (or error)? - Avery !The author suggests that there should be a balance between creating UIs that are accessible while also allowing for innovation to accommodate for ubiquitous computing. Shouldn't UI design and technology revolve around how a task is performed as opposed to creating technologies to fit a certain form factor or input/output interaction? - Garrick