W.Buxton, S. Patel, W.Reeves, andR. Baecker

Structured Sound Synthesis ProjectComputer Systems Research GroupUniversity of TorontoToronto, Ontario, Canada M5S lAI

Introduction

In conversation, we typically constrain our com-ments to fall within the scope of the topic at hand.Scope can be used to define the sphere of action ofany activity. When conductors request that players"play a little more staccato in the lower brass," theyare imposing a certain scope (lower brass) on a spe-cific action (play). Scope is an important concept inmany contexts in which we desire to specify theprecise range of some command. In this article wewill consider the expression of scope in interactivescore editors (Buxton et al. 1979).

Most existing score editors restrict the degree towhich they permit users to impose scope upon op-erators, or cQmmands (Smith 1972; Reeves et al.1978; Wallraff 1978). For the most part, operators(such as "play" or "delete") can only be directed to-ward single notes or entire scores. This is un-satisfactory because ideally any operator should beable to be directed to any arbitrary grouping ofnotes the composer thinks of in the mind's ear.There are, however, considerable problems in im-plementing such a facility, most notably developingthe semantics and syntax appropriate for such spec-ification in a musical context. Addressing these is-sues is a current topic of research with the Struc-tured Synthesis Sound Project (SSSP). Although thisresearch is still in progress, we believe that theinterim results presented here help to give somestructure to the problem and will be of use to thosecurrently writing or designing score-editingsoftware.

S. Patel is currently with Human Computing Resources,Toronto, Canada. W. Reeves is now with Lucasfilm Corp.,Novato, California. R. Baecker is with both the University ofToronto and Human Computing Resources.

Computer Music Journal, Vol. 5, No.3, Fall 1981,0148-9267/81/000050-07 $05.00/0@ 1981 Massachusetts Institute of Technology.

so

Scope in InteractiveScore Editors

The Semantics of Scope

The objective in scope specification is to identifywithout ambiguity the notes that serve as the oper-and of a command. If this is to be done fluently, thegrouping criteria must match the way that thenotes may be grouped in the composer's mind.Such criteria form the basis for an unambiguous de-scription of the notes. A first step in our researchwas to attempt to gain some understanding of thesecriteria through techniques of observation and in-terrogation. As a result, we have come to viewgrouping criteria as falling into five basiccategories:

Simple grouping criterion - scope is either thewhole score or a single note (e.g., "play thethird note").

Block-of-time grouping criterion - scope is aset of notes contained in a particular timeinterval (e.g., "delete the notes of the thirdbar").

Local-attributes grouping criterion - notes areencompassed in the scope on the basis ofself-contained attributes such as pitch or du-ration (e.g., "raise the volume of all notes be-low C3").

Contextual-attributes grouping criterion - anote's context (relationship to other notesand their attributes) determines whether thenote is included in the scope of an opera-tor (e.g., specifications such as "all notesfollowed by a leap of an octave upwards" or"sounding simultaneously with three ormore other notes").

Named-structural-entities grouping criterion -scope is specified by naming the musicalstructure to be affected (e.g., "motif A").

We see each successive category as growing in se-mantic power. Together, these categories form afoundation for the semantics of scope specification.

Computer Music TournaI

Simple Scope

When scope can only be expressed as a single noteor an entire score, we refer to it as simple scope. Ifa program allows both single-note and whole-scorescope to be expressed, the minimum basis for ascore editor is provided. With regard to the underly-ing representation of the score data, simple scopepresents no real problems since the operand of acommand is either the entire data set or a singlerecord. At this level, scope lends itself well to ei-ther a graphics or alphanumeric mode of specifica-tion. In graphics mode, for example, the entire scorecan be encompassed by activating the light-buttoncorresponding to the desired operator. An operatorcan modify a single note by pointing at the desirednote. (See the article by Buxton et al. [1979] formore details on such graphic techniques.) Alpha-numeric-based interaction, on the other hand, canbe modeled on line-oriented text editors. In suchcases, scope can be specified as the line currentlybeing edited or as the entire score file.

Scope by Block of Time

While simple scope can provide the basis for an ele-mentary score editor, many compositionally impor-tant concepts are commonly encountered thatcannot be handled by such an editor. For example,we might want to audition the last section of a longscore or a sequence of notes in the middle, or wemight want to transpose a particular chord. In eachcase, we want to address the set of notes fallingwithin a particular block of time. The situation iscommonplace in which such blocks are expressedwith regard to note position by number (as in ourfirst example), bars, rehearsal marks, seconds, orbeats. If such concepts can be expressed in a conge-nial way, the editor's power is substantially in-creased, as is its usefulness to the composer. Forexample, notes of isolated IIchunksll of the scorecan be heard in context, chords can be treated assingle entities, and entire sections can be saved orcopied in a single gesture for the purpose of repeats.

In making scope specification available by block

of time, the designer must pay close attention todelimiters and their effect on the underlying datastructures. Bar lines or rehearsal marks cannot beused unless these concepts are kept in the database. If time is represented by fractions of a beatwith respect to a metronome marking (as in theSSSP system described by Buxton et al. [1978]),specification by units of real time (such as seconds)may be awkward to implement.

The mode of interaction also has an effect on thetechniques of delimitation available in a system.Graphics-based systems that permit scrollingthrough and zooming in and out of the score lendthemselves well to use of the current viewport asthe scope delimiter. The specification of blocks oftime by the use of markers on a time line is an-other technique better suited to a graphics-basedapproach. Alphanumeric systems are often moreappropriate when numerical values are used as de-limiters, such as when precise timings in secondsare required.

An ideal environment would provide a number ofalternative methods for delimiting scope. For thepurposes of the current discussion, however, wewill present only one sample approach taken fromthe SSSP alphanumeric score editor seed (Buxton1981a). seed is modeled on the line-oriented texteditor ed (Kernighan 1974). ed is the most com-monly used text editor at our facility. While from apurely musical perspective it is not the best modelto use, seed was based on ed to permit text-editingskills to be applied to music and vice versa. How-ever, we chose not to edit scores as text files.Rather, SSSP music editors operate directly on themusical data structures. This has several advan-tages: (1) scores need not be compiled into a lower-level representation to be performed, which resultsin fewer files and operations and permits real-timesynthesis for purposes of verification; (2) the editorcan be interpretative because it has knowledgeabout the various operations and can therefore pro-vide error checking and other forms of help; (3)scores are edited in the common language of theSSSP system and therefore any score can be editedby any score editor whether or not it is graphics-based.

W Buxton et al. 51

In seed note numbers rather than line numbersare the basis for navigating through the score. Thusnote number is the basis for scope delimitation.

The following examples illustrate sample imple-mentation of scope by block of time. First, the spec-ification of scope at this semantic level encom-passes the previous level, simple scope. The firstexample,

3d

means "delete the third note," while

*p

means "print the entire score" (by convention, *means "all notes"). Double delimiters are used toexpand upon simple scope. The example

3,81

means "let me listen (1)to notes three througheight," while the statement

lO,$w fred

means "write (w) or save notes ten through to theend (specified by the special symbol $) as a newscore called fred."

The specification of scope at this level has metwith a very positive response from composers. Cer-tain frustrations have arisen, however, which dem-onstrate that extraction of a block of time cannotenable us to handle all cases encountered in the ed-iting of scores. Therefore the technique is insuffi-cient, regardless of the units used for boundarydelimitation.

Scope by LocalAttributes

The problem with specifying scope using the tech-niques discussed thus far is that all notes in the in-dicated block of time are encompassed. Thusconcepts such as "all quarter notes in the third bar"or "all brass playing mt below middle C in this sec-

52

tion" cannot be accommodated. While time-blockspecification helps the composer focus in on onewhole area of the score, it does not permit the iden-tification of the specific notes in question. Suchidentification can be made if a note has some user-specified characteristic included in the scope. Forthe purposes of this discussion, let us say that the"legal" characteristics are the local attributes ofthe notes themselves. Besides block of time, weshould be able to take a note's pitch, duration, loud-ness, instrument, and other properties into consid-eration before we include it in the scope.

The power of scope specification at this level de-pends on the composer's ability to express ar-bitrarily complex relations among these attributes.This implies that logical relations (e.g., equivalence,greater than, not equal to, etc.) and conjunctions(e.g., and, or, and exclusive or) must be expressible.

With implemented score-editing systems, thecrucial limitation is that only attribute fields thatform part of the data base can be considered as cri-teria for inclusion in the current scope. It is notpossible to specify "all notes played by trumpets,"for example, if no record is kept of orchestration.(The only exceptions to this are attributes that areimplicit, such as note number or time. With thisprovision, it becomes clear that simple and block-of-time levels of scope specification are encom-passed by the local-attributes leveL) While it is easyfor a musician to group notes mentally according tosome relationship, it becomes more difficult whennotes must be unambiguously identified to a com-puter. To get a feeling for some of the problems in-volved, let us look at some specific examples.Again, these are taken from the alphanumeric scoreeditor seed.

{freq < C4}p

means "print all notes having a frequency less thanC4." (In seed, the relational expression appears be-tween braces and before the operator. In this andother examples, key words such as "frequency" in-dicate particular note attributes. All key words canbe abbreviated in order to reduce typing.)

{freq = C4, G3}orch flute

Computer Music Journal

will cause all notes having a pitch of C4 or G3 to beorchestrated with the "flute." An example of amore complex relation would be

{dur <= 1/4& obi != flute}d

which would delete all notes not orchestrated by"flute" that had a duration less than or equal to aquarter note.

{# >= 3 & # <= 12 & vol <lOO}setvol +20

can also be written as

3,12{vol < 1O0}sv +20,

which means "increment by 20 units the volume ofall notes from the 3rd through the 12th whose vol-ume is less than 100."

There are several pQints worth noting about theseexamples. First, it would be difficult to expresssuch logical relations using graphical techniques.Although the concepts can be expressed easily innatural language, the only way to express them to acomputer is with typed alphanumerics. The syntaxused to express the relational concepts in the exam-pIes is rather arcane. From a human-factors point ofview, some trade-off must be made between sim-ilarity to natural language and succinctness. (Somefeeling for this conflict can be seen in the last ex-ample-compare the natural-language interpreta-tion with the second version in sced notation.)Clearly, much research remains to be done. Wehope that work such as that of Card, Moran, andNewell (1980) and Ledgard and coworkers (1980)will help pave the way. These researchers raise is-sues that must be dealt with in future score editors.The paper by Card, Moran, and Newell is a studyquantifying the relationship between the verbosityof a message and the efficiency of the human-computer dialogue. The paper by the second groupinvestigates the influence of natural-language-basedconstructs on the efficiency of text-editing tasks.

Second, while the simple and block-of-time lev-els of scope could be carried out using general-purpose text editors (if the score is representedas a text file), this is not the case with the local-

-

attributes level. Here it is clear that the application-specific nature of the attributes (and the operators,e.g., "orch," or "orchestrate a note by attaching aninstrument to it") makes the expression of complexlogical relations impractical. We are no longer deal-ing with concepts that can be expressed simply astypographical operators. To enable scope specifica-tion at this level, it appears that a special-purposeeditor that understands musical attributes must beprovided.

Scope by ContextualAttributes

The previous section showed how a score editor'spower can be increased by having a note's inclusionin the scope depend on the note's conformation tocertain composer-defined characteristics. Often,however, more complex relations than those seenthus far are required. Simple relations based onpitch or duration are not enough. Rather, scope in-clusion is sometimes best specified in terms of thenotes' contextual attributes, or of the relationshipsamong different notes and their attributes. Wemight want to specify, for example, a type of chord:"all notes that sound in combination with three ormore others," or a motif: "all notes orchestratedwith brass playing a dotted eighth, followed by asixteenth," or an intervallic sequence: "all notespreceded by a step downward of a minor second andfollowed by an upward leap greater than a perfectfifth."

The ability to express such constraints is clearlyof musical value, and musicians have no difficultyin grouping notes according to such criteria ver-bally or mentally. When it comes to identifying (un-ambiguously) such groups to a computer, however,problems arise. In contrast to the situation inwhich scope is defined by local attributes, the num-ber of criteria for grouping in context is infinite. Tounderstand context, the editor must have a farhigher level of musical "knowledge" (1) to under-stand the various criteria and (2) to perform theconsequent pattern recognition on the data basethat will isolate the notes in question. Even if theappropriate concepts could be understood by theeditor, the specification language used by the com-

W Buxton et al. 53

poser would probably be too cumbersome to be use-ful. We have seen, for example, how specification oflocal attributes can approach the threshold of prac-tical complexity.

As formulated, specification of scope at the con-textual level would seem impractical without agreat deal of research. We can either undertake thisresearch or reformulate the problem. For the pur-poses at hand, we have chosen to reformulate theproblem.

Musicians have no problems dealing with theconcepts under consideration. An alternative ap-proach, therefore, would be to make best use of therequisite knowledge of the composer. The task isthen to provide an environment in which the com-poser can use this knowledge to identify manuallythe desired notes in an efficient way. The originalproblem involved the computer collecting the notesthat fit a description supplied by the composer. Weare now sidestepping the description problem andhaving the composer identify the notes. A descrip-tive approach is being replaced by a demonstrativeone.

The success of this demonstrative approach de-pends on two conditions. First, the notation musthighlight the relevant features of the musical data.Second, a straightforward means of interactionmust be provided. As a result of both of these con-ditions, we have chosen to take a graphics-based ap-proach in our experiments at the contextual level.With graphics, we have the notational flexibility tohighlight most of the different features and rela-tionships involved. We also have a far greater rangeof options as means of interaction (alphanumericsis one such option).

We have implemented this technique in the pro-gram scriva (Buxton 1981b). In scriva, an intuitivegesture-circling-can be combined with spatiallydistributed data as a means of isolating desiredgroups. More than one circle can be drawn on thescreen (Buxton et al. 1979). Doughnutlike circleswithin a circle can be drawn, with the effect that allnotes within the larger circle, except those con-tained in the inner one, are included in the currentscope. There can be as many "holes in the dough-nut" as desired and a circle within the hole is againa circle of inclusion.

54

Often the notes we want to group into the con-textual-attributes level of scope are distributedthroughout the score, making circling impractical.In such cases, the desired notes can be "collected"if we point at them one by one with the graphicscursor (Buxton et al. 1979).

With the techniques seen in the examples we canmake some headway in the specification of scopeaccording to context. The techniques are still ratherprimitive, however, and much work remains to bedone.

Scope by NamedStructural Entities

At the contextual-attributes level of scope, we en-countered problems owing to our inability to de-scribe adequately the notes constituting the scopeof an operator. As a result, we resorted to a demon-strative approach. Another alternative is to identifythe notes in question by name. In cases wherethese notes constitute a musically significant struc-ture in the score, this is common practice amongmusicians. Simple examples would be "motif A,""the second theme of the first movement," or "theostinato in the second section." Names such as"the second theme" are too vague for automaticisolation of the intended notes by a computer pro-gram. This problem can often be circumvented if (1)we are more rigorous as to what constitutes a nameand (2) if, when composed, musically significantstructures are named by the composer.

What does this mean musically? Our approach toscope at the named-structural-entity level is basedon the assumption that the score is more than acollection of notes. It is assumed that the score ismade up of sections, motifs, and parts, each ofwhich has an explicit name. Our approach is alsobased on the assumption that the composer has as-sembled the score so that there are relationshipsamong these structures. Finally, our approach as-sumes that the internal representation reflects thisstructural "recipe." If these three conditions aremet, there is no reason why any of these structuralentities cannot be identified directly by name. Atthe contextual-attributes level, we failed to perform

Computer Music Journal

Fig. 1. Snapshot of an im-age generated by a three-dimensional score displayand editor developed atthe SSSP.

~SYntactic Breadth

the analysis that would extract the referenced partof the score. At the named-structural-entities level,the equivalent to such an analysis is given: it is theexplicit structure as pieced together by the com-poser. This structure is reflected in the datastructures.

Rendering such an approach practical has oneother property. Whereas up to this point scope hasbeen based on purely physical or acoustical proper-ties of the data (time, pitch, timbre, etc.), we nowhave a means of scope specification with a com-positional foundation!

There are, however, practical limitations to thenamed-entity approach. Fundamental among theseis the need for an appropriate model for the under-lying structure of scores. A list structure is too sim-ple musically and a general relational network toocomplex to handle (due to space/time trade-offsthat would result in poor performance). We havechosen a hierarchical "tree" structure (Buxton et al.1978). While this clearly has musical limitations, itis the most general structure we feel to be man-ageable at this moment.

The technique is impractical except for larger-scale structures. It would not be reasonable to ex-pect the composer to name every single entity. Yetit is exactly these larger structures that are themost poorly handled by the demonstrative tech-

--

8

nique. Smaller structures that are not well suited tothe naming technique are generally easily handledby the demonstrative approach.

Our first experiments with structuring data bynamed entities had to do with notational tech-niques. How could the external representation ofthe data reflect the underlying structure so as to aidthe composer in exploiting the information? Theexample shown in Fig. 1 is an illustration of thiswork. Illustrated is an example score called SUBataken from the program treed (Kwan 1978) (con-trived for the purposes of demonstrating the sys-tem). The feet at the bottom of the displayrepresent note events whose durations are propor-tional to the length of the line that is the foot,along the time axis. The triangles labeled SUBb,SUBc, and so on represent large syntactic catego-ries, that is, subscores. These subs cores encapsulatetwo or more notes. Thus SUBa is the root of thetree, and SUBf is the lowest nonterminal. Thenumbers 1, 2, . . . 8 next to the terminal branchesof the tree are labels used in editing. Since this is asnapshot of a dynamic program display that canbe rotated in various ways, certain features thatemerge in movement cannot easily be seen in thisstatic image. For example, frequency information isembedded in this representation that comes outmore clearly in the dynamic image. A clear struc-ture is provided (in this example, at least) that facil-itates the expression of concepts such as "playSUBb," or "transpose SUBd."

We do not yet have usable score editors that workon scores structured in this way. The number ofproblems encountered make progress very slow.One key challenge has been to use this complexlevel of internal representation while retaining ourability to play the scores in real time without anycompilation or preprocessing. Our recent efforts tomeet this challenge have focused on the perfor-mance of such structures. These efforts resulted inthe conduct system (Buxton et al. 1980). One of thekey concepts reflected in the conduct system is theability to address and transform named structuralentities in real time. conduct is a convincing exam-ple of the power and practicality of scope specifica-tion at this level. We are now trying to bring thispower into the domain of our score-editing tools.

W Buxton et al. 55

Conclusion

In this article we have described how the notion ofscope can be developed within the context of inter-active score editors. Our objective is to extend themeans by which composers can express scope ac-cording to musically relevant constraints. There arethree different syntactic approaches to scope defini-tion: the descriptive approach, the demonstrativeapproach, and the naming approach. Differing com-binations of syntactic approach and semantic levelsometimes make conflicting demands on both theunderlying internal representation and the mode ofthe musician-machine dialogue. With the descrip-tive approach, greater semantic power can beachieved by use of alphanumerics than by use ofgraphics-based techniques. On the other hand,graphics appear to be the most appropriate tech-nique for the demonstrative approach. Both graph-ics and alphanumerics can be used in addressingnap1ed syntactic entities.

Acknowledgments

The work reported in this paper has benefited fromdiscussions, comments, and other types of inputfrom many people associated with the SSSP. In par-ticular, we would like to acknowledge the contribu-tions of James Montgomery, Martin Lamb, WesleyLowe, William Matthews, Otto Laske, Leslie Gon-dor, K. C. Smith, Susan Frykberg, and Curtis Roads.The research has been funded by the Social Sci-ences and Humanities Research Council of Canada,whose support we gratefully acknowledge.

S6

References

Buxton, W. 1981 a. "A Tutorial Introduction to sced." InMusic Software User's Manual, 2nd ed., ed. W. Buxton.Toronto: Computer Systems Research Group, Univer-sity of Toronto.

Buxton, W. 1981b. "Tutorial Introduction to scriva." InMusic Software User's Manual, 2nd ed., ed. W. Buxton.Toronto: Computer Systems Research Group, Univer-sity of Toronto.

Buxton, W. et al. 1978. "The Use of Hierarchy and In-stance in a Data Structure for Computer Music." Com-puter Music Journal 2(4): 10-20.

Buxton, W. et al. 1980. "A Microcomputer-Based Con-ducting System." Computer Music Journal 4(1): 8-21.

Buxton, W. et al. 1979. "The Evolution of the SSSP ScoreEditing Tools." Computer Music Journal 3(4): 14-26.

Card, S. K., T. P. Moran, and A. Newell. 1980. "The Key-stroke-Level Model for User Performance Time withInteractive Systems." Communications of the Associa-tion for Computing Machinery 23(7): 396-410.

Kernighan, B. W. 1974. "A Tutorial Introduction to theUNIX Text Editor." Murray Hill, New Jersey: Bell Lab-oratories Technical Memorandum 74-1273-17.

Kwan, A. 1978. "Treed Project Report." Unpublishedmanuscript. Toronto: c.S.R.G., University of Toronto.

Ledgard, H. et al. 1980. "The Natural Language of Interac-tive Systems." Communications of the Association forComputing Machinery 23( 10): 556-563.

Reeves, W. et al. 1978. "Ludwig: An Example of Interac-tive Computer Graphics in a Score Editor." In Proceed-ings of the 1978 International Computer MusicConference, vol. 2, ed. C. Roads. Evanston, Illinois:Northwestern University Press, pp. 392-409.

Smith, 1. 1972. "SCORE-A Musician's Approach toComputer Music." Journal of the Audio EngineeringSociety 20( 1): 7-14.

Walraff, D. 1978. "Nedit-A Graphical Editor for MusicalScores." In Proceedings of the 1978 International Com-puter Music Conference, vol. 2, ed. C. Roads. Evanston,Illinois: Northwestern University Press, pp. 410-450.

Computer Music Journal