Shape Manipulation using Pixel Rewrites

George W. FurnasYan Qu

School of Information andDepartment of Computer Science and Engineering

University of MichiganAnn Arbor, MI 48109

AbstractIn the interest of developing techniques for manipulatingshapes that do not have easy algebraic generators, weexplore pixel-rewriting rules that work directly on the rasterrepresentations. This paper illustrates several simple pixellevel rules and how they can be put together to achieve avariety of complex kinds of functionality including movingarbitrary blobs, shortening curves, and area preservingdeformations. A new pixel-rule match optimization is alsopresented.

Keywords: graphical rewrites, pixel rewrites, shapemanipulation, graphical computation, visual computation

1 INTRODUCTIONAlgebraic models typically generate the forms used incomputer graphics. Leveraging centuries of intellectualwork, we can use the power of analytic geometry tomanipulate formulas for splines and polygons, and therebymanipulate the corresponding shapes. There is a weakness,however, to this approach: it is easiest to work with formsthat have simple algebraic generators. Working with atriangle is easy, but a shape that differs from the triangle byonly a few pixels is disproportionately difficult. Manyforms in the real world, from Monet’s “Water Lilies” to thesurface of the moon, are easily captured in bitmaps, yetwould require difficult algebraic modeling. The work hereexamines how computation done directly on the pixels mayhold more possibilities than previously realized for workingwith shapes, even if those shapes have no easy algebraiccharacterization.

Pixels have been used in computation in severalsubfields related to graphics. Paint programs, for example,

use the raster array as the principal data structure, andvarious operators change pixels on it, as the user’s mousemoves brushes or pencils to put color on the canvas. Acurve drawn at the pixel level (e.g., Figure 1(a)), however,is very hard to manipulate. If it were a rasterization of aspline representation, the user could move the controlpoints to reshape the curve. However, if the curve is justpixels, editing the curve comes down to tediously erasingand then redrawing pixels. The underlying algebraicrepresentation for the spline has been more accessible toversatile computational manipulation than are raw pixels.The work here explores the possibility of redressing thatimbalance.

Where familiar vector graphics are implemented withanalytic geometric representations in computer languagesthat use strings of symbols (like C++), the work here uses acomputational model that focuses on manipulating pixels –a Pixel Rewrite System (PRS). It is essentially a rule-basedproduction system from A.I.[3], but the rules matchspecified pixel patterns on a “field” and then rewrite them.If there are multiple matches, a conflict resolution schemepicks a winner, e.g., based on explicit rule priorities andrecency of match. The rule corresponding to the winningmatch performs its rewrite, changing the pixel field. Theevolution of the field under the iterated rewrites results in akind of pixel-based visual computation.

As a simple example, suppose that having drawn thepixel curve of Figure 1(a), the user decided it was too “longand loose”, and needed “tightening up.” This would not betrivial, even with a spline representation, and in a rasterversion one would typically just erase and redraw the wholecurve. As we will explain later in the paper, such a global

(a) (b) (c) (d) (e) (f)

Figure 1: Successive shortenings of an arbitrary pixel curve, accomplished by pixel rewrites. (Details are given in section 3.5).

shortening is not difficult to accomplish, however, right inthe pixels with a pixel rewrite system.

This paper explores how pixel rewrites can provide avariety of shape-rich computations. After an overview ofrelated work, we show some basic pixel rewritingalgorithms. These algorithms are then used in morecomplicated examples. We also present a new patternmatching technique used to speed up the implementation.

2 RELATED WORKThe work here uses pixel-rewrite rules to explore thevariety of interesting manipulations of shape possible whencomputing directly in the pixels. While the focus and manyof the details here are novel, the work is situated in thecontext of several bodies of work. There is only space tomention some highlights.

First, there have been several specific efforts exploringgraphical rewrites. Perhaps the longest standingexplorations of grid-based rewrites worked with ArrayGrammars [12][15], where symbols in a two (or higher)dimensional array are rewritten as new sub-configurationsof other symbols. Most of the original motivation andsubsequent use of such systems has been in image analysisand in shape generation (as opposed to manipulation), andhas used the array analogs of Context Free (or occasionallyContext Sensitive) grammars, for reasons of parsability.The pixel rewrite systems of this paper are essentially themore general, unrestricted (so called “isotonic”) arraygrammars, augmented by priority-based conflict resolution.We are interested in going beyond CF and CS rewrites toobtain the full power of Turing Equivalent graphicalcomputation, as we not just create, but work in variousways with shape on the canvas.

Graphical rewrites are also used in other grammar-basedapproaches in graphics, including L-systems[9] and ShapeGrammars[18]. Unlike the work here and array grammars,these efforts often use vector representations, instead of 2Dcanvas of cells. Like array grammars, however, theemphasis has been mostly on Context Free grammars, tospecify classes of shapes and generate instances for shapecreation (e.g., trees for L-Systems, architectural forms forShape Grammars). Again, we pursue richer, TuringEquivalent computation so that we can more arbitrarilyreshape, rearrange, and otherwise manipulate existingshapes.

Kurlander and Bier [7] [8] explored powerful graphicalsearch and replace capabilities for graphical editors. Theyallowed users, for example, to change all yellowrectangular boxes with bold text to green ovals with italics.Their primary interest was on traditional drawing typevector objects and their graphical search and replacecapability was provided at the level of a user’s single unit

task. In this paper, we exclusively use rasterrepresentations, and whole sets of rules run in complicatedsequences to perform graphical computations resulting incomplex graphical manipulations.

The graphical grid rewrites of the Agentsheets [11],Chemtrains[1] and Coco/Kidsim [17] systems supported theteaching of programming to children, particularly forsimulations – a domain that was both engaging for kids andquite amenable to graphical rewrites. The work developedvaluable interface ideas for easy creation of graphicalrewrites, as well as explorations of conflict resolution andthe modularization and control of rule sets. Some of theseideas are incorporated in the work presented here. However,the principal focus in this paper is on shape manipulation,not simulation, and on grids of simple pixels, not the morecomplex cell objects these systems explored.

The graphical rewrite system we use is a directdescendant of Furnas’s BITPICT system[4], a pixelrewriting system proposed as a possible model for “purelygraphical” reasoning. More recently Furnas et al.[5] havelooked at how pixel rewrites can make use of intermediateconstructions to do computation. We build on that workhere, using some of the fundamental algorithms developedthere, and using a multiple-layer capability to facilitateintermediate constructions, but with a focus on visualcomputation for shape analysis and manipulation.

Yamamoto’s VISULAN system [20] extended Furnas’spixel-rewrite programming language to include typedeclarations, modularization, and “extended” rules thatmatch and rewrite logical conjuncts of patterns. Thesesystem-level features look extremely powerful, and shouldbe valuable in supporting application functionality like theshape rich manipulations we explore here.

Second, beyond graphical rewrites per se, is a largercircle of literature on grid-based computing in general.Perhaps most important here are Cellular Automata (e.g.,[2][19]), which like a PRS, have a grid representation ofstates which change over time as a result of simple “rules”,often giving rise to interesting emergent functionality.Cellular Automata “rules,” however, are implemented asFinite State Machines that change the state of each cellbased on its current state and that of its neighbors. Our rulesrewrite whole local configurations of cells at once, makingit easier to work with patterns of varying granularity. Thishas made it easier to program desired, rather than just“emergent,” functionality.

A third relevant area of research involves other sorts ofexplicit, pixel-level computations. Computer vision usespixel operations extensively in the early stages ofpreprocessing and parsing. Filters highlight edges byenhancing brightness gradients in the pixel array. (See

standard texts, e.g., [10][13]) In morphological analysis[14], noisy edges are healed by first dilation (expanding thelocal pixel neighborhood to fill small gaps) and thenerosion (thinning back to the original feature size). Later,parsing of shapes begins by tracing, pixel by pixel, thetopologically connected components in the pixel array.Image processing is also central to photo manipulationapplications such as photoshop, using pixel operations likeblurring, sharpening, and color substitution. Several ofthese image processing operations can be implemented in aPRS, and are useful in our shape manipulations. There are,however several interesting differences. Most of the imageprocessing operations are filters where each pixel isrewritten as a specific function of its neighbors. Ourrewrites are much less restrictive, and highly patternsensitive in their local application. Another difference isthat in image processing, pixel values are treated as intervalor ratio scale, and numerical operations like multiplicationand addition are used extensively. In our current work,pixel values are only nominal scale – exact match and valuereplacement are the only operators defined. This deliberateconstraint encourages the use of pixel value for discretestate information, leading to the rather different kind ofcomputation than when a pixel is used to store somecontinuous, image-relevant value. Ultimately, some hybrid,combining what can be learned from these differentapproaches will be most useful. Finally, unlike in imageprocessing, our motivation is not to process images so theycan be vectorized and recognized, as in computer vision,nor to enhance or create special effects on images. We wantto work with raster representations more generically:moving, rearranging, deforming, and reshaping. We use amuch broader set of operations in our rewrites to achievethis wider variety of goals.

Finally, the field of Digital Geometry [7] analyzes thespecial topological and geometric problems of discrete gridworlds. Because of our focus on shape in pixels, manyconcepts from Digital Geometry, like connectedness,containment, “continuity,” deformation, and distance havebeen relevant to our algorithms for shape manipulation.

In summary, there has been much fruitful work ongraphical rewrites, grid computation and pixel manipulationin many fields. The work in this paper draws upon it, but isdistinguished most by its elaborate use of sets of pixel-levelrewrite rules to enable general and varied manipulations ofshapes as represented directly in the pixels.

3 EXAMPLE ALGORITHMSIn this section we introduce several pixel rewrite algorithmsfor working with shape. We begin with an introduction tosome of the most basic rules and their capabilities, to give asense of the building blocks upon which all the otheralgorithms are based. Then we review two fundamentalalgorithms we have introduced elsewhere, and show someexample shape-manipulation algorithms that can be builtusing them. We then introduce several new shapemanipulation algorithms.

Note that the more complicated algorithms use from ahalf-dozen to several dozen rules, and there is not spacehere to show and explain them in detail. The initialoverview of simple rules together with the sequence ofpanels in the figures for the more complex algorithms mustsuffice. The sets of rules fire in fairly rich sequencesdictated by the interaction of the state of the pixel canvas,and the match patterns and priorities of the rules – alldevised as part of the pixel programming task. While notaltogether trivial, it has seemed quite comparable to othersorts of programming for engineering students to master.

(a) Flood (d) BoundingBox

(e) NibbleTip

(f) Thin

(g) 8- Connected

(c) FeatureMark - EdgeMark

(b) MovePixel (to right)

Key:

Allow Reflections

Allow Rot & Refl

Allow Rotations

Translations only

Figure 2: Seven single-rule systems. More complex rulesets are built on basic operations like these.

3.1 Some basic rulesFigure 2 shows several single-rule systems and theirconsequences. Some of the underlying concepts are familiarfrom image processing and other graphics contexts. Indeed,these simple rules could be much more efficientlyimplemented in systems dedicated to those uses. The pointhere, however, is that they are easily implemented in a PRSand become building blocks of more complex algorithmswell beyond the scope of these other systems.

One of the most useful and basic algorithms is the Floodwhich, in (a), recolors a gray neighbor of a yellow pixel tobe also yellow, eventually flooding with yellow allneighboring space that is connected by gray to the originalseed. Flooding is used, for example, to find all the pixelsconnected by the flooded-over-color to the initial seed.Another simple rule is PixelMove (b) which simply movesa single pixel over, in this case only to the right. Anothercategory of simple rule is FeatureMark, here the specialcase, EdgeMark. In (c), it marks with blue all left and rightexterior edges where turquoise meets gray.

Simple shape changing operations are a bit morecomplicated. The rule in (d) locally fills in corner pixels,ultimately taking a shape to its bounding box. (e) nibblesaway tips (pixels with only one 4-adjacent neighbor),ultimately pruning back all arborizations. (f) eats awayotherwise unattached corner pixels of 4x4 masses, thinningblobs to a skeleton. (g) takes two diagonally adjacent pixelsand removes any redundant 4-adjacent intermediary,ultimately cleaning up any 4-connected curve to make itpurely 8-connected.

These and similar simple rules can be used in manyways, as building blocks of more complicated algorithms.Conflict resolution schemes become increasingly importantas rule sets get more complicated. The primary approach is

to give rules different priorities, such that low priority rulesonly get a chance to fire if there are none of higher prioritythat match. A secondary approach, used to break priorityties, is to give precedence to matches that have beenawaiting attention the longest. This is what allows theflooding, for example, to spread in a breadth first way,instead of a random way – a feature that we will use shortly.

Our system allows multiple 2-D layers. The field cancontain many layers, and rules can match and rewritepatterns spanning multiple layers. Although we do not showit here, layers can be used as a first-class third dimensionallowing voxel rewriting (see also [21] ) which we can usefor 3D shape manipulation. Layers are also heavily usedfor the intermediate graphical calculations explored in [5](e.g., one might just mark edges in a “scratch” copy layer).For clarity in figures here, however, we have used a singlelayer whenever possible.

3.2 Move blobThe standard way to move an arbitrarily shaped pixel blobwould be to somehow define its boundary in a bit-mask,then using the bit-mask, simply bitblt the blob to a newlocation, perhaps via some temporary array. Given theefficiency of this existing technique (and its commonhardware acceleration) why bother with the more tedious

(a) (b) (c)

(d) (e) (f)

Figure 3: Blob Move. (a) Blob with red pull-dot. (b) Blobflooded from mark. (c) Inner back edge and outer frontedged marked. (d) Flood of “marking finished” signal. (e)Back edge deleted, front edge added. (f) After cleanup.Blob now moved over 1 pixel. (See Furnas, et al [5] fordetails.)

(a) (b) (c)

(d) (e) (f)

(g) (h) (i)

Figure 4: Blob move variations. (a) A small blob at thebeginning of an arbitrary pixel curve winding through afield of other blobs. The curve’s pixels turn successivelyinto pull-dots, ultimately dragging the blob along the curve.(b) When it nears other blobs, a push-dot is interposed, and(c),those blobs move out of the way. (d)-(f) Eventuallyleaving a cleared path (g) around the original curve of a sizedictated by the sweeping blob. (h)(i) If the back edge of theoriginal blob is never deleted, a ribbon can be left behind.

methods to be shown here?

There are at least two answers. First, moving blobs ofarbitrary size and shape is a basic capability forarrangement and manipulation. If we cannot do it withpixel rewrites, it would bode poorly for the suitability ofPRSs for shape manipulation. Furnas et al [5] showed howit can in fact be done with only local rewrites, using someof the simple rules above. A variant is shown in Figure 3. Astarting configuration containing several green blobs, oneof which has a red “puller” dot next to it. The first task is toidentify the full extent of the blob to be moved. This isaccomplished by first marking the pixel of the blob that isadjacent to the pull-dot (feature marking), and then usingthis mark as a seed for a flood that colors the blob a new“selected” color. More feature marking is then used to markthe exterior forward edge, and the interior back edge. Thefront edge is extended as marked, and the back edgedeleted, resulting in a blob of the original shape moved overone pixel. (Special rules handle single pixel wide features,where front and back edges collide. See [5] for details.)

The second reason to implement blob-move in pixelrewrites is that, as we have often found, once we implementsome familiar capability in pixel rewrites, we find it has adifferent neighborhood of variant algorithms than thestandard (in this case, bitblt) algorithm. For example, thelocal and contingent nature of the pixel rewrite rules meansthat the blob move can respond in richer ways to its localcontext.

For example, in Figure 4a, a blob is at the beginning ofan arbitrary white pixel path. If we write a rule whichalways turns the white pixel adjacent to the blob red, thenthe blob will be pulled, pixel by pixel, along that arbitrarypath. (The rule must be low priority, so it does not fireagain until the blob-move is complete.) We could alsowrite rules that insert a “push-dot” next to other blobs themoving blob gets close to. Using a trivial “push-dot”

variant of Figure 3 that pushes instead of pulls a blob, theadjacent blobs will be automatically pushed out of the way(as will their neighbors in turn as necessary.) Alternatively,encountered blobs can be added onto, or be deformed by,the moving one. In another variant, if we turn off the erase-trailing-edge rules, the result is a “ribbon” created bydragging the shape along the path – again all done in thepixels (Figure 4(h)(i)). The curve-following, the collisiondetection, recursive pushing and ribbon-creation are allsimple variants in pixel rewrite space.

The simplest form of the repeated blob moving in Figure4 was not particularly efficient. Because of all the floodingin the simple move algorithm, moving the blob one pixelover takes time proportional to its area. If one wants tomove the blob repeatedly (as in path following cases), onecan use simple rules to create a kind of exoskeleton thatincludes and connects all edges of the blob. All the work(the move signals, the front-edge-growth and back-edge-deletion) can then just work with this exoskeleton. Theskeletonization (Figure 5) is done in a second layer: theoriginal blob copied, its edges marked, the rest deleted. Tolink the exterior edges with the edges of any possibleinternal lakes, a special flooding process is used: if theflood meets itself from different directions, it has gonearound a lake. Those meeting points can be used to define aconnecting “bridge” to the outer edges. The result is a fullyconnected skeletal green blob, which is then moved likeany other blob (as in Figure 3), except that when its edgesare extended or deleted, those of the original blob in thetop layer are altered along as well. The result is analgorithm where second and subsequent moves only taketime proportional to the perimeter of the blob. This thusconstitutes a notable example of a pixel algorithmconstructing a persistent, arbitrarily-shaped, pixel-level datastructure (the exoskeleton), which it then uses later forefficiency purposes.

(a) (b’) (c’) (d’) (e’) (f’) (g’)

(h’) (i) (j’) (k’) (l)

(a')

(i’) (l’)

Figure 5: Move Blob by Exoskeleton. (a) &(a’) Exo-skeletonization is accomplished using a second layer (light bluebackground). (b’) The blob is copied to the second layer and (c’) internal edges are marked. (d’)(e’) A dark blue flood from arandom seed location propagates. (f’)(g’) It draws connector pixels where it meets itself. (h’) The blue flood is then deletedand the rest returned to green, leaving a connected exo-skeleton of the original. The overview (i)&(i’) shows a red pull dot onthe first layer, which is copied (j’) to the second layer where a standard blob-move is performed (begun in (k’)). When frontedges are added and rear edges deleted on the skeleton, they are also done on the corresponding parts of the blob above onLayer 1, and (l)&(l’) the whole blob and skeleton are moved together.

3.3 Shortest PathIn [5] we presented an algorithm to find the shortest pathsbetween sets of arbitrarily shaped sources and sinks, aroundarbitrary obstacles. The algorithm can be used directly forrich shape creation to draw short curves automaticallybetween arbitrary shapes, avoiding arbitrary obstacles. Wehave, however, found it very useful in several other shapemanipulation algorithms.

The basic algorithm is reviewed in Figure 6 (see [5] fordetails). A breadth-first flooding originates at the sinks,accomplished by a cycle of three edge marking rules. The

result is a growing “distance” field with iso-distancecontours in different colors. Three colors are necessary andsufficient to allow a hypothetical walker standing on anycontour to know which way is down hill. As the contoursgrow, when they touch source blobs, the point of contact ismarked (it is the closest pixel to the sink), the whole blob isflood-marked as “reached”, and contour growing continues.When the field growth is complete, lower priority rules startfrom the marked point on each source and walk downhill,following the cycle of colors, leaving a trail to the sink.

3.4 Shorten curveThe shortest path algorithm is part of the set of rules used toshorten the curve in Figure 1. As shown in Figure 7, asmall halo is grown around the curve and the result iscopied to another layer, creating a three pixel wide channel.There, a source is marked at one end of the original curve,

(a) (b)

(c) (d) (c) (d) (c) (d)

Figure 6: Shortest path. (a) Sources [red], Sinks [blue] andObstacles [brown]. (b) A Mod3 distance field grown layerby layer around sinks. (c) “walking” down to Sinks. (g)Shortest paths are complete. (See [5] for details).

Layer 2 Layer 2

Layer 2

(a) (b’) (c’)

(d’) (e) (f)

Figure 7: Shorten Curve. (a) Original curve, with halo. (b’)copied to Layer 2 to serve as channel. (c’) A Mod3 distancefield along channel. (d’) Shortest path (pastel colors)through the channel. (e) Resulting shorter curve copied backto Layer1. (f) Final curve after cleanup.

Layer2

(a) (b) (c) (c’) (d’) (e’) (e)

(f) (f’) (g) (g’) (h’) (h) (i) Figure 8: Area-Preserving Deformation. (a) A brown rectangular “deformable” blob with two black obstacles. (b) Red pusherpixels added (e.g., by user) before computation. Computation begins: (c) Touched blob-pixels turned glowing green. (c’)Obstacles, pushers and touched pixels are copied to Layer2. (d’)The first touched pixel (circled in the figure) sends out adistance field. (e’)When the field first floods to a spot not occupied by blob or obstacles, (e) the touched pixel walks down thefield to the free spot, moving there… Somewhat later (f), more pixels have moved as (f’) each pixel in turn sends out a fieldsearching for the nearest free spot… Eventually (g) the last touched pixel sends out its field, (h’) finds a free spot, and (h)moves there. A few rules clean up, leaving the blob deformed, but with its total area preserved, and each displaced pixelrelocated to the nearest free location.

and a sink at the other. The shortest path algorithm thenfinds a new path between them within the channel. Theresult is the shortest possible curve which has the sameendpoints and is nowhere more than one pixel away from

the original curve. (See [16] for an analysis of this notionof curve approximation.)

3.5 Area Preserving DeformationA second example using the shortest path algorithmprovides a kind of area-preserving, minimal shapedeformation. In Figure 8, a blob of brown pixels has awhole set of pixels on its left touched (e.g., by a user) by ared “pusher” boundary to signify that all adjacent blobpixels must be moved. Distance fields grown successivelyfrom each of the touched pixels find the nearest freelocation for that pixel to be moved to. After each touchedpixel is moved, the result is a blob whose area is the sameas the original, but whose designated pixels have moved aslittle as possible. Note that such an area-preservingtransformation would be exceedingly difficult toaccomplish if the shapes were represented algebraically,e.g., by their bounding spline.

3.6 Radar Controlled MovementArranging shapes to be nicely spaced apart is a commonactivity for graphic artists. A simple version is seen as the“distribute” capability in Draw programs. Pixel rewrites canbe used to position arbitrary shapes automatically withinarbitrarily shaped regions. Figure 9 shows an algorithm fora simple case, using a kind of radar to get points to repeleach other. One point turns red and sends out a breadth-first flood that grows until it touches something (an obstacleor another point). That collision is marked and used to seeda reverse, “echo” flood, which runs at higher priority. Whenthe echo returns to the original point, that point is movedaway from the echo by one pixel. The various points taketurns sending out this “radar” and moving away from each

Rules

( a )

(b)

(c) (d) (e) (f) (g)

(h) (i) (j) (k) (l) (m)

Figure 10: Line smoothing. (a) Monotonic 8-connected line to be smoothed. (b) Rules for “gas particles” moving up anddown and extending ends of “pistons.” (c) Simple 2-segment line. (d) Gas paricles born (rules not shown). (e) Top pistonextended. (f) Equalization in progress. (g) Equilibrium. (h)-(l) Process working on original curve of (a). (m) At equilibrium,with comparison lines.

(a) (b)

(c) (d)

(e) (f)

(g) (h)

Figure 9: Radar Move. (a) Five dots in a region. (b) Onesends out radar. (c) The radar-touch seeds reflection. (d) Thereflection floods back and (e) touches original source. (f) Itmoves away. (g) After cleanup. Dots take turns movingapart, and later (h) are well spaced in the region.

other. Radar can be combined with blob-move to getarbitrarily shaped blobs to spread themselves out over anarbitrarily shaped area.

3.7 Smooth monotonic lineAnother algorithm for pixel-level shape manipulation takesa monotonic curve that is jaggy and smoothes it; that is, itdistributes the jaggies more evenly. The basic idea is basedon a thermodynamic metaphor – gas pressure equalizationin a series of linked pistons. Each vertical segment of thecurve in Figure 10 is endowed with a “gas particle” thatbounces up and down vertically in its segment. When theparticle hits the top of its segment, it pushes on it, extendingit upward one pixel, at the expense of its upward neighbor’ssegment. Symmetrically, it extends its bottom downwardwhen it bounces there. The particles in all the segments aregoverned by the same rules, of equal priority, with the neteffect of giving all the particles the same “temperature”, sothe particles move up and down at essentially the samelinear speed. As a result, the short segments get their endsknocked on more often, and hence tend to grow as the longones shrink. Over time the segments stochastically evenout. The process may be stopped at any time, after anydesired amount of smoothing.

3.8 Eliminating Zones of CurvesThis final example manipulates zones of intersecting pixelcurves. Zones are primitive regions, ones that have nofurther subdivisions. Imagine someone working with a GISwho has drawn curves around certain overlapping regionsof interest, as shown in Figure 11(a). They then decide they

do not want certain of the resulting zones to be included,and mark them for deletion with the dark marks indicated.Figure 11 shows how pixel rewrites contract the greencurves so that they no longer include the indicated regions,and are properly inside or outside the other boundaries.Note how it uses the flood, feature-mark, thin, and 8-connrules of Figure 2. (The extra pixel space separating thecurves in the final figure was a deliberate aesthetic designedinto the algorithm.)

4 OPTIMIZATIONIn addition to exploring shape manipulation algorithmspossible with pixel rewrites, we have also been working onthe underlying execution engine, and have devised variousoptimizations. One of them is particularly interesting froma spatial computation standpoint, since it speeds up thepattern matching process by integrating the spatialstructures of patterns and sequential aspects of rule firings.

A brute force version of the underlying rule executionengine would have each rule scan the entire field at thebeginning of each rule execution cycle, looking formatches. A “changed rectangle” strategy is an obviousspeedup: scan the whole field once to create an initialmatch list, thereafter just keep track of what happens nearthe rectangle where a rewrite takes place and update thematch list accordingly. A more subtle mechanism, based onwhat we call Sequential Rule Dependency Computation,pre-computes the implications of the firing of one rule uponthe subsequent match status of other rules.

. (a) (b) (c) (d) (e)

(f) (g) (h) (i) (j)

Figure 11. Eliminating overlap zones of arbitrarily-shaped intersecting curves. (a) Three intersecting curves with (b) twozones marked with dots for elimination. (c) Marked zones flooded from original seedmark. (d) Foreign boundary marked.(e) New boundary drawn. (f) Original flood converted to boundary color, becoming essentially a very fat boundary. (g)Conversion of area to fat boundary complete. (h) Fat boundary thinned. (i) Thinning complete. (j) After cleanup.

Momentarily ignoring geometric transformations liketranslation and rotation, a pixel pattern (to be matched, orrewritten) simply specifies some pixels at some fixedposition and their respective values. Formally, it is a pairA=⟨S,M⟩ where S is a set of pixels (the support) and M is amapping from S into pixel values (e.g., grayscale or RGB).For two patterns, A and B, we write Compatible(A,B) iff A andB agree where they overlap, i.e., for every pixel s in SA∩SB,MA(s)=MB(s). (Any two patterns are trivially compatible ifthey do not overlap.) To accomodate geometrictransformations, we define Transform(A,delta) to be thepattern A transformed (e.g., translated, rotated) per someparameter vector delta (e.g., an offset vector). Finally we letCompatibleDeltas(A,B) be the set of all deltas such thatCompatible(A,Transform(B,delta)).

The optimization here is based on precomputing how thefiring of any rule Alpha can possibly affect the match statusof some other rule Beta, so we can narrow our searchaccordingly. Let the rules have Left- and Right-Hand-SidesAlphaL , BetaL and AlphaR , BetaR respectively. If rule Alphajust fired, then AlphaL must have matched in the previousfield. This alone restricts where BetaL could have matched –only at offsets compatible with AlphaL’s. For example, inFigure 12, if AlphaL matched at the location indicated, BetaL

could not have matched in the location shown in the centralpanel of (b): At the indicated offset, the lower middle pixelof BetaL overlays the upper right pixel of AlphaL, but theformer is supposed to be dark, while the latter is light – theyare incompatible at that transformation. The set ofCompatibleDeltas(AlphaL,BetaL ) are thus exactly thetransformations of BetaL that could have coexisted withAlphaL in the previous field (mapped out in Figure 12c). Soif the matchlist had any occurrences of BetaL in it, theymust be elements of this set. Similarly, after Alpha fires,writing AlphaR in place of AlphaL, the only possible waysB e t a L could now match is with deltas in the setCompatibleDeltas(AlphaR,BetaL) (Figure 12e). Thus the onlypossible changes that firing Alpha can make to theappearance of Beta in the match list are members of thedifference set:

CompatibleDeltas(AlphaL,BetaL) - CompatibleDeltas(AlphaR,BetaL).

If we consider this a "signed" set difference, all and onlypositive locations (blue +’s in 12f) must be searched in thefield for possible addition to the match list. All and onlynegative elements (red x’s in 12f) must be checked in thematch list – if they are there, they are deleted withoutsearching the field (Figure 12f).

In practice, tremendous speed-ups result from thisoptimization. This is particularly true if rule sets use manycolors, essentially because patterns have more ways tomismatch. For example suppose some rule, Alpha, has only

red and green pixels in it, and a second rule, Beta, has onlyblue and yellow. This precomputation will determine that,since the two rules use completely different colors, there areno non-trivial compatible deltas, and hence there can be nochanges in the set of Beta’s matches when Alpha fires – andno search is needed at all.

5 DISCUSSIONIn this paper we have shown a considerable variety ofalgorithms that enrich our ability to work with pixilatedrepresentations of shapes, including several variations onmoving blobs, shortening and smoothing curves, radarcontrolled movement, and automatic elimination of regions.The critical point is that these techniques work just as wellon shapes that have no easy algebraic generators as thosethat do. The blobs moved, the curves shortened, theintersecting contours redrawn could have been of any shapethat can be represented directly in the pixels. Thisversatility was possible because the computation isinherently “visual”, working directly on the pixelrepresentation by using a pixel-rewrite mechanism, insteadof manipulating underlying algebraic representations.Furthermore, representing shape directly in the pixels yieldseasy computational access to certain properties, likecollisions, complex distance fields, topologicalconnectivity, areas and lengths.

This work is in its early stages. We would not, at thispoint, want to make strong claims that pixel rewrites are thebest way to do shape manipulation. We do however,

Beta: Alpha:

Deltas (offsets) to check:

Change Rectangle Method

Deltas (offsets) to check:

Unification- Based Rule Dependency

Method

(a)

(f) = (e) – (c) (g)

YES no YES

Compatible(Alpha L,Transform(BetaL,delta))? (b) (c) Compatible Deltas BEFORE

Compatible(Alpha R,Transform(BetaL,delta))? (d)

YES no YES

AFTER

(e) Compatible Deltas

BEFORE

AFTER

DIFF

Figure 12: Sequential Rule Dependency Computation. (a)Two rules, Alpha and Beta. (b).Testing BetaL against AlphaL.(c) Only places BetaL could have matched before Alphafired. (d) Testing BetaL against AlphaR. (e) Only places BetaLcould have matched after Alpha fired. To update the matchlist at runtime need only check differences (f), typicallymuch smaller set than (g)

believe that their versatility has been underestimated, andthat their ability to work with arbitrary shapes, and to makecomputationally accessible certain properties of thoseshapes, suggest that they deserves more attention.

Future work would include developing more algorithms,both for shape manipulation and shape analysis.Challenges arising from the familiar constraints of the gridnature of the raster representation must also be addressed.(E.g., it took quite a while to discover a pixel rewritealgorithm that would draw a round Euclidean circle in thisrectilinear grid world.) We are also interested in multi-scalerewrites, using multiple layers with different resolutions. Torelieve the flatness of the pixel representations, future workcan use layers to mark and annotate semantic aggregates ofpixels, e.g., successive pixel layers coding successivelevels of part hierarchies of objects, all spatially structuredan manipulable right in the pixels. It would also beinteresting to explore how any such algorithms might beused interactively.

The result of this continuing line of research should be apowerful variety of ways to compute and interact with pixelrepresentations for the manipulation and analysis of shapes,and a better understanding of when pixel vs. algebraicmethods are most appropriate.

ACKNOWLEDGEMENTS

We would like to thank Intel and NSF (IIS-9877170) fortheir generous support of this work.

REFERENCES

[1] Bell, B. & Lewis, C. (1993) ChemTrains: A Languagefor Creating Behaving Pictures. In 1993 IEEEWorkshop on Visual Languages, 1993, 188-195.

[2] Codd, E. F., Cellular automata, New York: AcademicPress, 1968.

[3] Davis, R. & King, J. An overview of productionsystems. Rep. STAN-CS-75-524, Computer ScienceDept., Stanford Univ., Stanford, CA, 1975.

[4] Furnas, G.W. (1991) New Graphical ReasoningModels for Understanding Graphical Interfaces. Proc.of CHI '91 Conf. on Hum. Factors in Comp. Sys. 1991,71-78.

[5] Furnas, George, Qu, Yan, Shrivastava, Sanjeev, andPeters, Gregory, The use of intermediate graphicalconstructions in problem solving with dynamic, pixel-level diagrams. Proceedings of Diagrams 2000,published as: Michael Anderson, Peter Cheng, VolkerHarslev (Eds.) Theory and Application of Diagrams,Lecture Notes in A. I. #1889,Springer Verlag.

[6] Gardner M. (1971) Cellular Automata, Self-Reproduction, Garden of Eden and the Game of Life.Scientific American 224: (2) 112.

[7] Klette, R., Rosenfeld, A., & Sloboda, F (Eds.) (1998)Advances in Digital and Computational Geometry,New York: Springer-Verlag.

[8] Kurlander, D & Bier, Eric A. (1988) Graphical searchand replace, Computer Graphics, 22(4), 113-120.

[9] Lindenmayer, A. (1968) Mathematical models forcellular interaction in development, Parts I and II: J.Theor. Biol., 18, 280-315.

[10] Parker, J. R. (1997) Algorithms for image processingand computer vision New York: Wiley Computer Pub

[11] Repenning, A. & Fahlen, L.E. (1993) Agentsheets: ATool for Building Domain-Oriented VisualProgramming Environments Proc. of ACMINTERCHI'93 Conf. on Human Factors in Comp. Sys.,142-143.

[12] Rosenfeld, A. (1976.) Array and Web Grammars: AnOverview. In Lindenmayer, A., Rozenberg, G. (Eds.)Automata Languages, Development. North-HollandPublishing Company. 517-529.

[13] Russ, John C. (1998) The Image ProcessingHandbook 3rd Ed, Boca Raton, FL: CRC Press.

[14] Serra, Jean (1982) Image Analysis and MathematicalMorphology, New York: Academic Press.

[15] Siromoney, G., Siromoney, R, Krithivasan, K. (1973.)Picture Languages with Array Rewriting Rules.Information and Control, 22. Academic Press, 447-470.

[16] Sloboda, F., Zatko, B, and Stoer, J (1998) Onapproximation of planar one-dimensional continua. InKlette, R., Rosenfeld, A., & Sloboda, F (Eds.) (1998)Advances in Digital and Computational Geometry,New York: Springer-Verlag.

[17] Smith, D.C., Cypher, A. & Spohrer, J. (1994) KidSim:Programming Agents Without a ProgrammingLanguage. Comm of the ACM, 37(7), 54-67

[18] Stiny, G, (1980) Introduction to shape and shapegrammars, Environment And Planning B-Planning &Design, 7 (3): 343-351.

[19] Toffoli, T. & Margolus, N. (1987) Cellular AutomataMachines: A New Environment for Modeling,Cambridge, MA: MIT Press.

[20] Yamamoto, Kakuya (1996). Visulan: A VisualProgramming Language for Self-Changing Bitmap.Proc. of International Conference on VisualInformation Systems, Victoria Univ. of Tech.cooperation with IEEE (Melbourne, Australia), 88-96.

[21] Yamamoto, Kakuya. 3D-Visulan: A 3D ProgrammingLanguage for 3D Applications. Proc. of PacificWorkshop on Distributed Multimedia Systems, TheHong Kong Univ. of Science and Technology (HongKong), pp.199-206, 1996