advanced techniques: graphics | pebble developer retreat 2014
DESCRIPTION
You can find the video recording here: https://www.youtube.com/watch?v=lYoHh19RNy4 Heiko Behrens and Matthew Hungerford present advanced programming techniques for Pebble. This presentation focused on graphics techniques including run-time dithering, offline dithering, pixel manipulations, and frame-buffer drawing. This talk featured the amiga boing ball dithering demo. Day 1 - Video 3BTRANSCRIPT
GRAPHICS
MATT & HEIKO – DEVELOPER EXPERIENCE ENGINEERSON TWITTER AS @MTHUNGERFORD & @HBEHRENS
THE DISPLAY
144x168px @ 175 DPI
black & white only
25+ FPS
A Pixel’s Journey Dithering
Accessing the Frame Buffer
CONTENT
A PIXEL’S JOURNEY
LAYER STACK
Layers are added to the layer_stack from back to front starting from the window layer. !
Layers are then rendered from back to front into the application frame buffer.
Window
THE COMPOSITOR
Example of a non-fullscreen application (144x152):
Application frame buffer
Compositor frame buffer
Finaldisplay
GRAPHICS APIS
graphics_draw_text, graphics_draw_bitmap_in_rectLayer-equivalents working directly on GContext
gpath_draw_filled, gpath_draw_outlineDrawings vector graphics
graphics_draw_pixel, graphics_draw_line graphics_draw_circle, graphics_fill_circle, graphics_draw_rect, graphics_draw_round_rect, graphics_fill_rect
Drawings raster graphics
app_timer_register, app_timer_reschedule, app_timer_cancel Animation, PropertyAnimation
Animate graphics
static void custom_update_proc(Layer *layer, GContext *ctx);
Q&A
source: https://www.youtube.com/watch?v=Ej2ELEK-C3Q Where To?
GRAPHICS APIS
graphics_draw_text, graphics_draw_bitmap_in_rectLayer-equivalents working directly on GContext
gpath_draw_filled, gpath_draw_outlineDrawings vector graphics
graphics_draw_pixel, graphics_draw_line graphics_draw_circle, graphics_fill_circle, graphics_draw_rect, graphics_draw_round_rect, graphics_fill_rect
Drawings
app_timer_register, app_timer_reschedule, app_timer_cancel Animation, PropertyAnimation
Animate graphics
raster graphics
static void custom_update_proc(Layer *layer, GContext *ctx);
graphics_capture_frame_buffer, graphics_release_frame_buffer
Doing dithering at runtime
Drawings raster graphicsadvanced
DITHERING
Dither is an intentionally applied form of noise used to randomize quantization error, preventing large-scale patterns such as color banding in images.http://en.wikipedia.org/wiki/Dither
256 shades of gray
16 shades of gray + nearest color
16 shades of gray + random noise
16 shades of gray + ordered dithering
black & white + nearest color
black & white + noise
black & white + ordered dithering
16 shades of gray + ordered dithering
BLACK AND WHITE
16 shades of gray + random noise16 shades of gray + nearest color
256 shades of gray
16 SHADES OF GRAY
original ordered dithering Floyd-Steinbergdithering
nearest color
for each y from top to bottom for each x from left to right oldpixel := pixel[x][y] newpixel := find_closest_palette_color(oldpixel) pixel[x][y] := newpixel quant_error := oldpixel - newpixel pixel[x+1][y ] := pixel[x+1][y ] + quant_error * 7/16 pixel[x-1][y+1] := pixel[x-1][y+1] + quant_error * 3/16 pixel[x ][y+1] := pixel[x ][y+1] + quant_error * 5/16 pixel[x+1][y+1] := pixel[x+1][y+1] + quant_error * 1/16
FLOYD-STEINBERG DITHERING
foreach y foreach x oldpixel := pixel[x][y] + threshold_map_4x4[x mod 4][y mod 4] newpixel := find_closest_palette_color(oldpixel) pixel[x][y] := newpixel
threshold_map_4x4 =
ORDERED DITHERING (BAYER MATRIX)
source: Joel Yliluoma’s dithering page - http://bisqwit.iki.fi/story/howto/dither/jy/ original from PSX game “Chrono Cross”
source: Joel Yliluoma’s dithering page - http://bisqwit.iki.fi/story/howto/dither/jy/ nearest neighbor to 12 colors
source: Joel Yliluoma’s dithering page - http://bisqwit.iki.fi/story/howto/dither/jy/ Floyd-Steinberg to 12 colors
OFFLINE DITHERING
BETTER BITMAPS (GIMP/PHOTOSHOP)
Resize to desired size (<144x168) using sinc filter Convert image to grayscale Modify brightness/contrast to sharpen image Posterize to 3 colors (if dithering to gray_50) or 2 colors for black and white. Undo-Redo brightness/contrast & posterize till desired effect Convert image to black and white with ordered (positional) dithering
Better Bitmaps (GIMP/Photoshop)Open Original Image in the GIMP/Photoshop
Better Bitmaps (GIMP/Photoshop)Resize: Image -> Scale Image
Better Bitmaps (GIMP/Photoshop)Resize to desired size (<144x168) using sinc filter
Better Bitmaps (GIMP/Photoshop)Grayscale: Image -> Mode -> Grayscale
Better Bitmaps (GIMP/Photoshop)Brightness-Contrast: Colors -> Brightness-Contrast
Better Bitmaps (GIMP/Photoshop)Slide contrast to right to remove colors and balance with brightness
Better Bitmaps (GIMP/Photoshop)Posterize: Colors -> Posterize
Better Bitmaps (GIMP/Photoshop)Posterize: Select 3 colors (2 for black & white only). 3rd color is Gray.
Better Bitmaps (GIMP/Photoshop)Convert to 1-bit (b&w): Image -> Mode -> Indexed
Better Bitmaps (GIMP/Photoshop)Select b&w, choose dithering Positioned (Ordered) for dithered gray
Better Bitmaps (GIMP/Photoshop)Final Product: Robot with dithered gray_50
BETTER BITMAPS (IMAGEMAGICK)
convert orig.png -adaptive-resize '144x168>' -fill '#FFFFFF00' -opaque none -type Grayscale -colorspace Gray -black-threshold 30% -white-threshold 70% -ordered-dither 2x1 -colors 2 -depth 1 -define png:compression-level=9 -define png:compression-strategy=0 -define png:exclude-chunk=all new.png
DITHERED ANIMATIONS (APNG)
ImageMagick scripting !
Ordered dithering (gray_50) !
(a)PNG compression for 200 frames !
Resource loading and drawing at high FPS (10 FPS in this example)
RUNTIME DITHERING
50% GRAY
PRE-DITHERED
FLOYD STEINBERG
ORDERED DITHERING
PRE-DITHERED
AT RUNTIME
PRE-DITHERED
AT RUNTIME
PRE-DITHERED
AT RUNTIME
PRE-DITHERED
AT RUNTIME
PRE-DITHERED
AT RUNTIME
50% GRAY
PRE-DITHERED
FLOYD STEINBERG
ORDERED DITHERING
source: Joel Yliluoma’s dithering page - http://bisqwit.iki.fi/story/howto/dither/jy/ reduction to 18 colors
IMPLEMENTDITHERING
AT RUNTIME
foreach y foreach x oldpixel := pixel[x][y] + threshold_map_4x4[x mod 4][y mod 4] newpixel := find_closest_palette_color(oldpixel) pixel[x][y] := newpixel
threshold_map_4x4 =
ORDERED DITHERING (BAYER MATRIX)
foreach y foreach x newpixel := pixel[x][y] > threshold_map_8x8[x mod 8][y mod 8] ? white : black; pixel[x][y] := newpixel
ORDERED DITHERING BLACK & WHITE
foreach y foreach x newpixel := pixel[x][y] > threshold_map_8x8[x mod 8][y mod 8] ? white : black; pixel[x][y] := newpixel
ORDERED DITHERING BLACK & WHITE
IMPLEMENTED IN C// Bayer matrix for ordered dithering static const uint8_t ditherMatrix[8][8] = { { 0*4, 32*4, 8*4, 40*4, 2*4, 34*4, 10*4, 42*4}, {48*4, 16*4, 56*4, 24*4, 50*4, 18*4, 58*4, 26*4}, {12*4, 44*4, 4*4, 36*4, 14*4, 46*4, 6*4, 38*4}, {60*4, 28*4, 52*4, 20*4, 62*4, 30*4, 54*4, 22*4}, { 3*4, 35*4, 11*4, 43*4, 1*4, 33*4, 9*4, 41*4}, {51*4, 19*4, 59*4, 27*4, 49*4, 17*4, 57*4, 25*4}, {15*4, 47*4, 7*4, 39*4, 13*4, 45*4, 5*4, 37*4}, {63*4, 31*4, 55*4, 23*4, 61*4, 29*4, 53*4, 21*4} }; !static GColor color_for_gray(int16_t x, int16_t y, uint8_t gray) { return gray > ditherMatrix[y % 8][x % 8] ? GColorWhite : GColorBlack; }
static GColor color_for_gray(int16_t x, int16_t y, uint8_t gray); !static void update_proc_draw_pixel(Layer *layer, GContext *ctx) {
!!!!!!!}
const int16_t h = window_size.h; for (int16_t y = 0; y < h; y++) {
!!!! }
uint8_t row_gray = (uint8_t)((gray_top * (h - y) + y * gray_bottom) / h);
DOING LIVE DITHERINGgray_top
gray_bottom
AVOIDING FLOATS frac = 0.0 gray_top * (1 - 0.0) + 0.0 * gray_bottom
frac = 0.25 gray_top * (1 - 0.25) + 0.25 * gray_bottom
frac = 0.75 gray_top * (1 - 0.75) + 0.75 * gray_bottom
frac = 1.0 gray_top * (1 - 1) + 1.0 * gray_bottom
float frac = (float)y / h; // 0.0 … 1.0 row_gray = gray_top * (1 - frac) + frac * gray_bottom
AVOIDING FLOATS
// 1 * h = h // frac * h = y
uint8_t row_gray = (uint8_t)((gray_top * (h - y) + y * gray_bottom) / h);
float frac = (float)y / h; // 0.0 … 1.0 row_gray = gray_top * (1 - frac) + frac * gray_bottom
static GColor color_for_gray(int16_t x, int16_t y, uint8_t gray); !static void update_proc_draw_pixel(Layer *layer, GContext *ctx) {
!!!!!!!}
const int16_t h = window_size.h; for (int16_t y = 0; y < h; y++) {
!!!! }
uint8_t row_gray = (uint8_t)((gray_top * (h - y) + y * gray_bottom) / h);
DOING LIVE DITHERINGgray_top
gray_bottom
DOING LIVE DITHERINGgray_top
gray_bottom
for (int16_t x = 0; x < window_size.w; x++) { graphics_context_set_stroke_color(ctx, color_for_gray(x, y, row_gray)); graphics_draw_pixel(ctx, GPoint(x, y)); }
uint8_t row_gray = (uint8_t)((gray_top * (h - y) + y * gray_bottom) / h);
const int16_t h = window_size.h; for (int16_t y = 0; y < h; y++) {
!!!! }
static GColor color_for_gray(int16_t x, int16_t y, uint8_t gray); !static void update_proc_draw_pixel(Layer *layer, GContext *ctx) {
!!!!!!!}
source: https://www.youtube.com/watch?v=V5O8gj0SQZ0
DOING FAST(!) LIVE DITHERING
* graphics_capture_frame_buffer(GContext* ctx)
bool graphics_release_frame_buffer(GContext* ctx, GBitmap* buffer)
GBitmap
Gives you a GBitmap to represent the current frame buffer
…and locks out any other graphic function until you release it
GBITMAP RECAP
0
4
8
12
16
29px
4 bytes per row
5px
unused
unused
unused
unused
unused
Fragment of 8 pixels in a row, storedin 8 bits or 1 byte (with value 0x5C).0 1 0 1 1 1 0 0
Fragment of 8 pixels in a row, storedin 8 bits or 1 byte (with value 0x5C).0 1 0 1 1 1 0 0
typedef struct { void *addr; uint16_t row_size_bytes; // ... GRect bounds; } GBitmap;
GBitmap
uint8_t *byte_offset ; byte_offset += x / 8;
= (uint8_t*)bmp->addr
byte_offset *= y * bmp->row_size_bytes;
DOING FAST(!) LIVE DITHERING
static void update_proc_frame_buffer(Layer *layer, GContext *ctx) { GBitmap *fb = graphics_capture_frame_buffer(ctx); !!!!!!!!!!!!! graphics_release_frame_buffer(ctx, fb); }
uint8_t *row = (uint8_t *)fb->addr; row += fb->bounds.origin.y * fb->row_size_bytes; !!!!!!!!!!
const int16_t h = fb->bounds.size.h; for (int16_t y = fb->bounds.origin.y; y < h; y++) { uint8_t row_gray_value = (uint8_t)((gray_top * (h - y) + y * gray_bottom) / h); !!!! row += fb->row_size_bytes; }
uint8_t row_gray_dither_pattern = (y, row_gray_value); dither_pattern_8
RENDER 8 PIXELS AT ONCE
static uint8_t (int16_t y, uint8_t gray) { return color_for_gray(0, y, gray) << 7 | color_for_gray(1, y, gray) << 6 | color_for_gray(2, y, gray) << 5 | color_for_gray(3, y, gray) << 4 | color_for_gray(4, y, gray) << 3 | color_for_gray(5, y, gray) << 2 | color_for_gray(6, y, gray) << 1 | color_for_gray(7, y, gray) << 0;}
dither_pattern_8
DOING FAST(!) LIVE DITHERING
static void update_proc_frame_buffer(Layer *layer, GContext *ctx) { GBitmap *fb = graphics_capture_frame_buffer(ctx); !!!!!!!!!!!!! graphics_release_frame_buffer(ctx, fb); }
uint8_t *row = (uint8_t *)fb->addr; row += fb->bounds.origin.y * fb->row_size_bytes; !!!!!!!!!!
const int16_t h = fb->bounds.size.h; for (int16_t y = fb->bounds.origin.y; y < h; y++) { uint8_t row_gray_value = (uint8_t)((gray_top * (h - y) + y * gray_bottom) / h); !!!! row += fb->row_size_bytes; }
uint8_t row_gray_dither_pattern = (y, row_gray_value); dither_pattern_8 memset(row, row_gray_dither_pattern, fb->row_size_bytes);
source: https://www.youtube.com/watch?v=V5O8gj0SQZ0
BOING BALL
Q&A
source: https://www.youtube.com/watch?v=Z2QMG9UwXI0