A deep look at contemporary operating systems finds dataflow principles of value to computer networks large and small.

Operating SystemsPrinciples for

Data Flow NetworksPeter J. Denning

Purdue University

L 1970 the term computer network conjured upimages of computing centers in different citiesinterconnected with telephone lines. By 1975 theprospect of inexpensive memory and processingshowed that the next generation of operatingsystems at individual computing centers would beconcerned much less with memory allocation andprocessor scheduling; thus the term computer net-work also began to conjure up images of computingcenters internally organized as networks of special-purpose processors. By 1980 the technology oflarge-scale integration will permit sophisticatedcomputers to be configured as large arrays of verysimple processing elements. The term computernetwork will then conjure up images of gangedmicrocomputers working concurrently on manyportions of a single computation.The important components of any network of

computers are nodes and data links. The nodes areprocessing elements (service centers), each with itsown private (dedicated) memory and processor units.The data links are communications channels thatpermit messages to be sent among the nodes. Datalinks usually have a queue at the receiver to storemessages until the receiver is ready to processthem. Because each node has its own resources, theonly possibility of interaction among nodes isthrough the data links.A network in which the nodes are activated by

the arrival of messages (specifying tasks) is calleda data flow network. A node is initially in an "idle"state. As soon as it receives a set of messagessufficient to specify a task, it becomes "active"and begins work on that task. It may accept yetfurther tasks while active. It will transmit messages

as it completes tasks, and it will return to the idlestate when there are no more tasks.The concept of a data flow network is at least as

old as electronic computing. It is the basis of the"neuron net" model of finite state automata studiedby John von Neumann and others in the 1940'sand 1950's, and of the "Petri Net" model of con-current systems, introduced by Carl Adam Petriin the early 1960's. It is the subject of the "programschema" model of computation studied by theore-ticians in the late 1960's. It appeared in the formof ready-acknowledge signalling among hardwaremodules in the Illiac II computer at the Universityof Illinois in the late 1950's and in the Macromod-ular Computer Project at Washington Universityin St. Louis in the early. 1960's. It has been used inoperating systems in' the mechanisms by whichconcurrent programs exchange messages.Today, computer architects are searching for a

solid conceptual basis for designing and control-ling networks of computers. This quest resemblesthe search a decade ago for a conceptual basis ofoperating systems design. The purpose of thisarticle is to cull from the field of operating systemsa few concepts-those of data flow-which mayboth simplify networks of computers and helpprove that they meet their specifications. Thispaper sketches out our experience with the softwareaspects of data flow, in a framework for structur-ing one's thinking about computer networks. (Toemphasize the overall structure, the concepts arestated in simple terms-with references to theliterature in which the nasty details appear. Thisis done at the risk of giving the impression that allthe problems are understood fully and solved.)

0018-9200/78/0700-0086$00.75 (3 1978 IEEE COMPUTER86

The next section reviews the concepts underlyingtraditional operating systems and the reasons thatsome of these concepts do not apply in data flownetworks. The concepts that do apply work indifferent ways depending on the scale of the net-work. Additional sections elaborate how data flowconcepts can be used in networks large, medium,and small: dispersed networks connecting manycomputing centers, local networks of special-pur-pose processors at a given computing center, andmodular computers for data flow programs.

Comparison with traditionaloperating systems

The traditional framework for an operating systemis a single computing center having a limitedamount of equipment with which to serve a largenumber of customers. Figure 1 illustrates that thejobs (programs) P1 P2,. . ., P, submitted by nusers are presented concurrently to the computingfacility for processing. Each job consists of asequence of tasks, a task being an interval (burst)of processing on some device in the system.Typically, a job alternates between tasks that usea central processing unit and tasks that perform aninput/output operation. For example, Task 1.1might represent a 100-msec CPU burst, Task 1.2a 50-msec disk swap, Task 1.3 a 150-msec CPUburst, and so on.

It is easy to conceive of jobs in which some tasksexecute in parallel-for example, the job couldcontinue to use the CPU while performing I/O.Unfortunately, present-day programming languagesdo not allow the user to express the possibilitythat one task can be concurrent with another;operating systems are typically able to uncoveronly about 5 percent CPU I/O overlap in jobs. Anew kind of language-the dataflow language(discussed later)-does permit the programmer tospecify parallel tasks.The responsibilities of the operating system in

the computing center fall in two broad classes-allocating the hardware resources such as proces-sors, I/O devices, and main memory, and imple-menting software resources such as files, processes,and message channels.,

Hardware resource allocation is required in atraditional computing center because the amountof equipment actually needed to process all thetasks of all the jobs far exceeds what is available.Moreover, the equipment itself is very expensive;no user could afford to pay for the entire machineduring the period required to process his entirejob. The operating system maintains a series ofqueues for various types of resources and attemptsto schedule all the pending tasks so that all thedevices are simultaneously busy most of the time.This tends to maximize the number of tasks persecond that are completed and thereby to distributethe costs of running the equipment among as manydifferent jobs as possible.In a traditional operating system, none of the

devices contains enough computing power to doany of the scheduling. The scheduling routines,which are activated by interrupt signals from thedevices, are run preemptively on the CPU. Thehigh level of preemptions of regular CPU tasks canconstitute a considerable overhead.

Virtual objects. Software resource implementationis needed to provide a more useful and more reliableprogramming environment for the user. Softwareresources are often called virtual objects by operat-ing systems specialists-a term signifying thatwhat appears to the user to be a simple objectmay in fact be simulated by a complicated under-lying mechanism. A file is an example of a virtualobject. It appears to the programmer simply as aset of records which he may update or read oncommand from his program. However, the operat-ing system must format the records and storethem in various media, it must set up buffers andprovide subroutines to control the transmission ofrecords between main memory and secondarymemory, and it must protect files from unauthorizedaccess or accidental damage. Other examples ofvirtual objects are

processes-programs in parallel execution;virtual memory-the set of addresses that can begenerated by a process that can exceed avail-able main memory;

semaphores-channels through which processescan send signals;

Figure 1. A computing center processing a set of concurrent job streams.

July 1978 87

message queues-channels through which proces-ses can exchange messages; and

catalogs-directories to the files.By implementing useful virtual objects, an oper-

ating system can save the programmer from aconsiderable amount of programming which hewould otherwise have to do himself. By hiding theinternal details of the objects and of the task ofcontrolling the devices used to implement them,the operating system can protect its parts frommisuse; it can then be more reliable for all users.A few experimental programming languages

allow the user to declare his own virtual objects.Many operating system designers believe thatsuch languages-and operating systems that sup-port them efficiently-will be essential for the verylarge data processing problems looming in the nextdecade.2-5

Decentralization. Although it might seem thathardware resource allocation is a simpler respon-sibility than implementing software resources, thereis evidence to the contrary. The SOLO operatingsystem, which dispenses with all forms of dynamicstorage allocation, has been completely expressedwith 1300 lines of Concurrent Pascal.2 Traditionaloperating systems are typically a thousand timeslarger. (However, it is unfair to conclude thatdynamic storage allocation is the major source ofoperating system complexity: typical operatingsystems implement many more kinds of softwareresources than SOLO.)Through today's cheap processor and memory

technology, adequate supplies of hardware re-sources can distribute the processing capabilityamong the nodes of the computing center, reducethe amount of scheduling, and eliminate mostCPU interruptions. As a result, operating systemscan be significantly simplified.Figure 2 illustrates this concept, suggesting that

jobs can now move among many nodes within acomputing center. Each node performs one task(using its own private set of hardware resources)and dispatches a request for a job's next taskdirectly to another node capable of performing thattask. Because there is less dependence on running

Figure 2. Processing concurrent job streams through many nodesof a modular computer center.

scheduling routines on the CPU, this type of systemtends to decentralize its control and to acquirethe attributes of a data flow network.It is a mistake to suppose that operating systems

would disappear if there were sufficient hardwareto eliminate the hardware resource allocation re-sponsibilities. The remaining responsibilities-forsoftware resources-would increase because thelarger processing capacity would attract largerprograms-which in turn make more use of virtualobjects to achieve modular design.

It is a mistake to suppose that operatingsystems would disappear if there were

sufficient hardware . . . hardware cannotimplement virtual objects.

Control flow vs. data flow. The realities of tradi-tional computing centers-a small amount of hard-ware resources relative to the demand for comput-ing, together with the requirement that mostscheduling decisions be made by programs run onthe CPU-have caused centralization of control tobe an important principle in the design of operatingsystems.For most of us, programs are sequences of instruc-

tions run in a computing system having one (or avery few) CPU(s). Therefore, control flow is also animportant principle, not only of programming butalso of operating systems. It is characterized by afascination with instruction pointers. Much of theprogramming literature focuses attention on how"control structures" guide an instruction pointerin an orderly way through the program text. Pro-gress in a program is measured by the number ofinstruction-pointer moves. An operation is activatedby moving an instruction pointer to the entrypoint of a subroutine. Many explanations of con-currency treat process creation as the spawning ofnew instruction pointers, semaphore signalling as amethod of preventing an instruction pointer frompassing a certain position until a signal arrives,and mutual exclusion as a restriction on the numberof instruction pointers allowed to be within acritical section at the same time.By contrast, computer networks are based on the

principles of decentralization and data flow. Becauseeach node has its own set of hardware resources,nodes can be in different physical locations-thereis no central resource allocator. Because each servernode is activated by the arrival of tasks, progressis measured by task flows, not by the nurnber ofchanges in the positions of instruction pointers. Inthis tontext, a given level of concurrency is met byproviding more parallel paths through the network,not by interrupt-driven central control mechanisms.Given types of virtual objects are implementedby the facilities of nodes dedicated to theirmanagement.Even as there are some fundamental conceptual

differences between networks of computers and

COMPUTER88

traditional operating systems, there are importantsimilarities. Many operating systems include com-ponents that operate on the principle of data flow.The RC4000 multiprogramming system is an

early operating system based on data flow.6 Itsnucleus allows users to regard the basic machineas a network of processes connected by first-in-first-out (FIFO) message links. If there is one send-er and one receiver for each link, the computationperformed by a network of processes is deter-minate.7 Because the operating system automati-cally stamps each message with the identity of thesender, it is possible to have secure data transmis-sions in such a network.8-10 The concept of messagelink is implemented as the "queue" data structurein Concurrent Pascal2 and as the "pipe" in BellLabs' UNIX timesharing system."' The principle oforganizing the operating system about a data flownetwork of processes has been extended and refinedin the DEMOS operating system for the Cray-icomputer. 12

Since 1971, queueing network models have beenused with great success in computer performanceevaluation. A queueing network is an example ofa data flow network connecting independent servicecenters. Many of the analytic tools can be applieddirectly to networks of computers in order to pre-dict delays, throughputs, and queue-length distri-butions, and to determine the capacity of the net-work for each class of jobs.'3 Many analytic toolshave also been developed for the capacity andresponsiveness of the communication part of geo-graphically dispersed networks.'4To summarize, traditional operating systems

have been dominated by concepts of centralizationand control flow, whereas networks of computersare dominated by decentralization and data flow.Those aspects of operating systems design dealingwith centralized resource allocation are useful, atmost, inside the nodes of a network of computers.However, the aspects dealing with networks ofservers (or processes) are useful for the entirenetwork of computers. These points will be expand-led in the next sections for specific classes ofnetworks.

Five desirable properties

Users expect computer systems to behave accord-ing to minimum standards, irrespective of whetherthese systems are based on central or distributedcomputing, or on control or data flow. The fivemost important properties are job determinacy,long-term storage, reliability, security, and pre-dictable response.Job determinacy is the requirement that the

system preserve the input/output function specifiedby each program. A job's output should dependonly on the input but not on the relative delayseither at the nodes or on the data links. In otherwords,- a job should be reproducible in that it givesthe same result for the given input no matter what

the internal configuration of nodes and data links,and no matter how many other jobs are present inthe system.Long term storage is the memory system that

allows each user to store files of records indefinite-ly. It allows him to access these files as neededand to share them with other, selected users. Theuser may need access to his files from variousnodes of the network and an inexpensive methodof moving copies from one node to another.

Reliability means that the probability that (partof) a job is lost through a node or data link failure,or through a deadlock, should be small. If a job isdamaged by a failure in the network, an errormessage should be returned to the user.

Programming languages which allowthe user to declare virtual objects, such

as files, will be essential in thenext decade.

Security means that it is (nearly) impossible forany private data to fall into unauthorized hands.This means not only that each node must secureany information under its care, but also that intel-ligible copies of messages in data links cannoteasily be made and that spurious messages cannoteasily be injected onto the data links.Predictable response means that it should be

possible to estimate the important parameters ofjobs and nodes, and to use these parameters tocalculate, quickly and accurately, performancemeasures for time periods of interest. Typicalmeasures include the response time for a given jobor for classes of jobs, and the task flows by jobclass on various data links. Sometimes, especiallyin real-time control systems, it must be possible tocomplete a job within a given deadline.These properties are ambitious but within easy

reach of today's technology. Some can be met byapplying old results in the new context, others byrecent innovations.Because hardware resource. allocation is not an

important issue of computer networks, the designercan concentrate directly on satisfying the fivedesirable properties in the three classes of net-works-dispersed networks of computing centers,local networks of special-purpose processors, andmachines that execute data flow programs. Theseproperties can be met by implementing specialnodes to manage each kind of virtual object, byproviding adequate capacity on the data links andat the nodes, and by constraining the manner inwhich the data links are used.

Dispersed networks

Figure 3 shows a common form of network inwhich a collection of independent computing cen-ters, usually at different locations, is connected bya communications net. Each computing center is

July 1978 89

Figure 3. A dispersed network of computers.

connected to the net by an interface computer,which translates between the standard formatsand conventions of the net and the local formatsand conventions of the center. The net itself mayinclude telephone links, microwave links, and satel-lite links. This is a simple form of data flow net-work-the communications net supplies data linksbetween any two computing centers, and a com-puting center is activated by the arrivals of mes-sages specifying jobs.To avoid long holding times on any communica-

tions link, the net is "packet switched." A messageis broken into a sequence of equal-size packets fortransmission on the net. A packet contains identi-fiers for the source and destination, a portion ofthe message, and a sequence number. The packetsof a message may follow different paths, arrivingout of order at the receiver; but the receiver canreassemble the message in proper order by usingthe sequence numbers.The well-known Arpanet was the prototype of

this class of networks.'4 The research was brokeninto three parts-the design of a reliable communi-cations net with bounded delays, the design of themessage formats and conventions,* and the designof the interface computers. The recent researchfor this type of network has concentrated on thecommunications net-finding the cheapest intercityroutes for the links, maintaining reliability throughredundant links, maintaining capacity as satelliteradio links are brought on line (it is easy to jam the

*For reasons now lost, these formats and conventions have cometo be known as "protocols." However, the dictionary does notsupport this use of the word.

medium with too many broadcasts), and maintain-ing secure data communications.In the Arpanet the objective was to connect

existing computing centers with minimal modifica-tions in the (traditional) operating system at eachcenter. A simple approach was adopted. A user atone center can log in, via the net, at anothercenter and thereby gain access to the other center'sfacilities. In effect, the user submits his transac-tions to a remote site. It is very difficult to movedata files among sites because file formats of onecenter are not observed at another. In short, it isfar cheaper to log in at a distant site than it is tomove needed data to the local site for processing.The five desirable properties are conceptually

straightforward to implement in a dispersed net-work:

1. Job determinacy. In a dispersed processingsystem, a job's determinacy is implemented bystandard methods in the operating system at acomputing center.7'5 Sequence numbers assurethat the packets constituting a message can beput back in proper order on reception.2. Long term storage. Each computing centerhas its own file system. A user at a given nodemay log in at another node and thereby gainaccess to any of his files stored in the local filesystem. However, it is inconvenient and inexpen-sive to move a file to another site for processing.

3. Reliability. This is a function of the errorrecovery procedures at the computing centerthat processes the job'5 and also of the redundan-cy built into the communications net. Multinodedeadlocks cannot occur since each job uses onlyone node.4. Security. Messages must be encrypted forsending on the net.'7 It is interesting that theconcept of signing a message with a secret en-cryption key 18-19 is similar to the concept ofstamping an interprocess message with theidentity of the sender.9'0 Individual computingcenters must secure each job received from thenet; this can be done using standard approachesto operating system security.120'2' It is importantto note that communications security and inter-nal computing center security are distinct prob-lems-encryption cannot solve both.5. Predictable response. I developed analytictools exist for calculating thro hputs and delaysin the net'4 and in the center at processes ajob 13.22.23

There is increasing interest in using dispersednetworks as distributed data management systems,which would allow a user at one site to pose queriesand updates for records throughout the network.This objective attaches much higher importance tothe long-term storage facilities of the network asa whole than does the Arpanet. It cannot be metwithout imposing standard record and file formatsthroughout the network-hence the various nodes

COMPUTER90

must have a similar design. Data security is ofeven greater importance in such a network, for theentire contents of files and records can be trans-mitted in the open medium; for this reason, it isnot yet clear that distributed data managementsystems are a good idea. In any event, encryptionmechanisms will be a standard part of the interfacecomputers.' 18"9

Networks of special-purpose processors

Within a given.computing center, the variousdevices make up a local network of special-purposeprocessors. Figure 4 illustrates the network for atimesharing system. The boxes represent networknodes for various types of service. Each nodecontains a queue, and is normally active whentasks are present in its queue. The data linksare paths by which jobs "move" from one deviceto another. Each job is a chain of tasks (requests)for devices.

In such a system, each job is stored in a centralmemory system. Corresponding to each job is adescriptor, which is a small block of words in mainmemory telling where the job's program and dataare stored, which task is next to execute, and whatdevice the job is currently using. The data linksactually carry simple packets that are enqueuedat the receiving devices; each packet points to a jobdescriptor. To start a task for a job, a device refersto the descriptor for the details of the task; whenthe task is done, the device updates the descriptorto indicate completion, and it sends a message tothe device required for the next task of the job.Because of the central memory, no job is actually

sent on the data links. The data transmission costsare negligible and the same data formats applyautomatically throughout the system. These net-works, therefore, have a different character fromdispersed networks-but they may be componentsof dispersed networks.This type of network is usually called a queueing

network. A rich set of analytic tools for such net-works exists. These tools allow the analyst topredict throughputs, utilizations, delays, andqueue-length distributions at the various devicesfor various classes of jobs.1322 They allow findingbottlenecks under given assumptions about thefrequencies with which jobs generate tasks and thespeeds of the devices.1323

If the resources of a computing center can beorganized into a queueing network, the five desir-able properties are conceptually easy to meet:

1. Job determinacy. Each job is a linear chain oftasks. As long as the tasks are completed one ata time in the specified order, there is no dangerthat the results can depend on the relativespeeds of the devices.

2. Long term storage. The central memory sys-tem shared by the jobs is a suitable repositoryfor files and records. The design of such systemsis well understood.

Figure 4. A queueing network model of a timesharing system.

3. Reliability. Standard methods of error detec-tion and recovery can prevent job loss.16 Dead-lock can be a problem if a node can be stopped(blocked) waiting for an action at another node(see below). However, if no blocking is possible,there can be no deadlocks.4. Security. The usual methods of access controlcan be applied to prevent any task of a job froman unauthorized access to some object in thesystem.8'9'215. Predictable response. As noted above, thereare many easy-to-use analytic tools for answeringquestions about performance and system capacity.

Traditional operating systems do not exhibitall these convenient properties, since the require-ment to reallocate dynamically some "passive"resources, such as memory, among the nodes ruinsthe assumption that the devices be independent.For example, a device processing a given taskmight stop to wait for buffer space to be releasedby a second task at another device. A deadlockcan result if the second task can also wait for anaction of the stopped first task. Our analytic toolsdo not handle networks in which one device canblock progress of another. Special deadlock pre-vention procedures must be added to a system toavoid deadlock among active tasks competing forlimited passive hardware resources.7'1"Cheaper technology may remove some, but not

all, of the threat of interdevice blocking and dead-lock. We will be able to dedicate to each deviceall the passive hardware resource it needs. How-ever, contention for virtual objects-especiallyshared files and records-will still remain; it cannotbe avoided by any amount of memory. To preventthis danger of deadlock, the internal data com-munications among the tasks of jobs must be putin some hierarchical structure.7"'Cheaper technology will also allow more ambitious

systems to be configured as queueing networks.For example, a network of processes, such assupported in the RC4000 and the DEMOS systems,could be built as a queueing network wherein each

July 1978 91

"device" would be a sophisticated service center.One can envisage service centers for communi-cating with users, compiling programs in variouslanguages, managing files, and initiating newjobs-every important service of a traditionaloperating system could be realized as a node insuch a network. Automatic reconfiguration of pro-cessing and memory resources, and of data links,would enable the system to adapt to the computa-tional demands placed on it.Queueing network. models are based on the

assumption that each job forms a linear chain oftasks. A less restrictive model of a job, based ona partially ordered set of tasks, allows for concur-rent execution of some tasks. (In Figure 5, forexample, tasks 3, 4, and 5 could be executed inparallel. Task 6 cannot be started until 3, 4, and 5are finished.) How this affects the network dependson the additional assumptions made about themodel. One can view the diagram of Figure 5 asa "dataflow program," from whence it becomes.a dataflow network in which the tasks are opera-tors that "fire" when all operands have arrived oninput links (this concept is taken up in the nextsection). Alternatively, one can view the diagramas a precedence chart specifying the order in whichprocedures implementing the tasks are called.(This is a control-flow view.) In this case, the tasksread inputs from, and store results into, a centralmemory; determinacy can be guaranteed only ifpotentially concurrent tasks have no output loca-tion in common with another task's input oroutput locations.*715

Figure 5. A partially ordered set of tasks.

*This condition of noninterference can be proved to be equivalentto assuming that the sequence of values written in everymemory cell is unique-i.e.. independent of the timings of thetasks." This means that the set of tasks is equivalent to adataflow network whose links correspond to the originalmemory cells and implement FIFO queueing. In this sensenothing is gained conceptually by the control-flow view of apartially ordered set of noninterfering tasks.

We conclude that the queueing network is auseful scheme for organizing the function processorsof a given computing center. It will become morecommonplace in future systems, where cheap tech-nology will permit each device to have its own,private set of hardware resources.

Dataflow programs

The two previous sections have considered, first,a large-scale network comprising different computingcenters and, second, a medium-scale network com-prising the processing devices of a given computingcenter. This section takes up the possibility of asmall-scale network-a dataflow program.A dataflow program implements a function from

input streams of data packets to output streamsof data packets. It is represented as a dataflownetwork in which FIFO data links connect operatornodes. As usual, a node activates as soon as oneoperand is present on each input data link; itconsumes each input packet and produces packetscontaining results for its output links. Figure 6shows a dataflow program that evaluates theexpression R=A*B + C*D; the next three resultsthat will appear on the output link are 5, 5, and8. Note that this program is simply the precedencetree of the expression, with queues shown on thelinks. The queueing permits pipelining-the abilityof the network to be at various stages of evaluatingdifferent expressions at. the same time. Pipeliningenhances the network's ability to process manycomputations concurrently.The ultimate purpose of a dataflow program is to

give the programmer a natural way of expressingthe concurrency inherent in an algorithm, therebyallowing high bandwidth parallel execution. Thisform of expression can be a much more effectiveway of representing concurrency than conceptssuch as "fork" and "join" in control-flow programs.Figure 6 is an instance of the job model encoun-

tered in Figure 5-a partially ordered set of tasks.While this may be an adequate description of jobs,it is inadequate for most programs, which alsorequire operations for selecting cases and foriteration.Figure 7 shows a scheme for expressing case

selection. A packet of the incoming data streamis routed leftward (to subnetwork N,) if the datacontained in the packet satisfies condition C; other-wise, it is routed rightward (to subnetwork N2).The split operator also generates "control packets,"each containing "T" or "F" to indicate whetherthe corresponding data packet was sent to N, orto N2. The merge operator uses the incomingstream of control packets to route data from N, orN2 to the output-e.g., if the next control packetcontains "T," the merge operator waits for a datapacket from N, whereupon it will consume thecontrol packet and pass the data packet to theoutput. The split/merge mechanism is needed tocause outputs to emerge in the same order as the

COMPUTER92

Figure 6. A dataflow program.

corresponding inputs. Without this constraint, thedataflow program would be indeterminate: itsresults could depend on the relative processingspeeds of its-components (e.g., N, and N2).Figure 8 shows how the split/merge scheme can

be used for iteration. Here the control link isinitialized with a "T" packet. This iteration isdeterminate, but it suffers from the limitationthat N2 can be used by only one task at a time; nointernal pipelining is possible.Dataflow languages are being developed at

MIT,242" at IBM T. J. Watson Research Center,26at the University of California at Irvine,27 and atIRIA Laboratory in France.26 A compiler of sucha language can be envisaged as a translator intoa primitive dataflow network which is executedon an interpreter.Figure 9 is a conceptual view of an interpreter

for a dataflow program. The operators and linksare encoded and stored in a memory of activestorage elements. Whenever an operator nodebecomes enabled because operands are present onall its input links, its host storage element sendsa packet to the extractor network. A packet con-tains an operation code, the operands, and theaddresses of links that are to receive the results.The extractor routes each packet to a functionunit for the required kind of operation; the resultsare fed back through an insertion network andstored in the output data links of the operator.There are several published proposals for asyn-chronous machines that implement the interpreterthrough wide bandwidth parallel execution ofdataflow programs.24.25.28,29Because we have little operating experience with

dataflow programs and machines, the followingremarks about the five desirable properties aremore speculative than for other types of networks:

1. Job determinacy. This is assured automaticallyby connecting operators through FIFO data linksand by properly merging data streams previouslysplit. 25.27.28

2. Long term storage. The cache memory tech-nology can be used to back up the active memory(Figure 9) with a passive memory for long-termstorage. The research problem is representingstructured data for storage in a high-bandwidthmedium accessible to many concurrent oper-ations.25

3. Reliability. Freedom from deadlock can inprinciple be assured by configuring the network

Figure 7. Selection in a dataflow program.

July 1978 93

Figure 8. Iteration in a dataflow program.

Figure 9. A dataf low interpreter.

so that it has properties similar to those of "liveand safe" Petri Nets.30 Protection against failureof function units and memory in data flowmachines can be achieved through redundanthardware; in one proposal, triple redundancycan be achieved at about 2.8 times the cost of asingle machine, and double redundancy at 1.7times the cost.3'4. Security. Access controls can verify thatdataflow programs refer only to authorizedinformation. The source and destination addressesin packets increase security. 810 The explicitnessof dataflow paths may allow better enforcementof information flow policies than is possible forsequential programs.32

5. Predictable response. Instruction processingrates and maximum delays can be calculatedeasily.33 However, the methods of queueing net-work analysis must be extended to answer per-formance questions for given programs on dataflow machines. This is an open research area.Dataflow languages, machines, and programs are

in a highly experimental stage. Many problems

remain to be solved, such as pipelined iteration,recursion, processing structured data, security, andpredicting response. It is a fascinating subject.

Conclusion

A network of computers is characterized by acollection of independent processing nodes con-nected by data links that enqueue unreceivedmessages. Because the nodes are activated by thearrivals of messages, these systems are examplesof dataflow programs.The traditional context of an operating system

is a single computing center in which a smallnumber of resources must be multiplexed among alarge number of customers. This context has pro-duced op'erating systems built around a centralresource controller. These systems are often domin-ated by the concept of control flow in their structure.Despite their orientation around centralized

resource control and control flow, many operatingsystems employ some concepts of dataflow, espe-cially in their methods of exchanging messagesamong processes. The concepts and theory of theseaspects of operating systems may be helpful indesigning networks of computers in which the fivedesirable properties-job determinacy, long termstorage, reliability, security, and predictableresponse-can be implemented in a straightforwardway.The application of these concepts depends on the

scale of the network. Table 1 summarizes the fiveproperties against scale. On the large scale, a dis-persed processing network deals only with jobsthat require processing at a single node. Traditionaloperating system principles can be used to realizethe objectives at the individual nodes and moderncommunications technology, in the communicationnet. At present, the high cost of data communica-tions impedes the use of these networks as dis-tributed data management systems.At the medium scale, a queueing network deals

with jobs which are sequences of tasks for indi-vidual devices. Cheaper technology is about tomake feasible the practice of dedicating to eachdevice all the hardware resources it needs. Byallowing simple messages on the data links torefer to complex tasks, a central memory keepsinternal data communication costs negligible inthese networks. Operating systems, which can beregarded as networks of processes rendering ser-vice to users, can be implemented as queueingnetworks.At the small scale, a dataflow program is an

expression of a dataflow network among the oper-ators (tasks) of a job. The current technology ismaking feasible the implementation of dataflowmachines. These machines can use the many dataflow concepts developed over two decades. Theycan achieve much higher concurrency than ispossible in traditional machines.

COMPUTER94

Table 1. Comparison of network classes.

DispersedNetwork

LocalNetwork

DataflowProgram

Job Determinate computing Determinate devices Determinate operatorsDeterminacy centers Jobs = task-chains FIFO data link queues

Sequence-numbered packets Proper split/mergeoperations

Long Term Storage File systems at computing Central memory shared Central memorycenters by all devices Structured data and

File system virtual objects stillan open problem

Reliability Redundant hardware and Redundant hardware and Redundant hardware ansoftware at each software open problemcomputing center Deadlock avoidance by Deadlock avoidance byRedundant links nonblocking devices analogy with liveMultinode deadlocks impossible and by hierarchical and sate Petri Nets

process signallingSecurity Access controls at the Access controls in the Access controls

computing centers central memory, assisted Packets signed withData encryption in the net by hardware source/destination

Signing packets with iden- Possible informationtifiers of senders flow certification

Performance Net delays analyzable Queueing network analysis Open problemPrediction Node delays analyzable.

This article has suggested a framework forstructuring one's thinking about dataflow net-works-for letting considerations of software affectthe design before the hardware is built. In thisbrief space it has been impossible to analyze allthe ideas fully. The simplified discussions maygive the impression that we know all the answers-we do not.

We may be more successful if we buildreal systems to be more like models

which embody the desired propertiesand which are tractable.

However, the simplified discussions are verydefinitely meant to suggest that we may well bemore successful by building real systems to be morelike models which embody the desired properties-and which are tractable. This established principleof engineering has been slow to gain acceptanceamong computer engineers. But, at last, we beginto find programmers who are willing to use struc-tured programming to obtain more understandableprograms whose correctness is easier to establish.We begin to find operating systems designerswho are willing to restrict process interactionswith semaphores or to put processes in deadlock-avoiding hierarchies. We begin to find performanceevaluators who are willing to impose controls onschedulers in order to make the real system con-form to the queueing network model, therebymaking its performance questions more easily andaccurately answerable.

If this trend continues, mirabile dictu, we mightcome to understand what our computers aredoing! -

Acknowledgments

This work was supported in part by NationalScience Foundation Grant GJ-41289 at PurdueUniversity.

I appreciate the help of Jack Dennis and JohnShore in clarifying some of the concepts reportedhere, and of Douglas Comer and Herbert Schwet-man in reading an early version of the manuscript.I am most deeply grateful to the guest editors,Steven and Svetlana Kartashev, without whosepainstaking review and patient suggestions thispaper would not be here.

References

1. P. J. Denning, J. C. Browne, and J. Peterson, "TheImpact of Operating Systems Research on SoftwareTechnology," in Impact of Research on SoftwareTechnology (P. Wegner, Ed.), MIT Press, Cambridge,Massachusetts, 1978.

2. P. Brinch Hansen, The Architecture of ConcurrentPrograms, Prentice-Hall, Englewood Cliffs, NewJersey, 1977.

3. E. Cohen and D. Jefferson, "Protection in the HydraOperating System," Proc. Fifth ACM Symposiumon Operating Svstems Principles, November 1975,pp. 141-160.

July 1978 95

4. A. K. Jones, "The Narrowing Gap Between Lan-guage Systems and Operating Systems," Proc.IFIP Congress, North-Holland Publishing Co., NewYork, pp. 869-873.

5. T. A. Linden, "Operating Systems Structures ToSupport Security and Reliable Software," ComputingSurveys, Vol. 8, No. 4, December 1976, pp. 409-445.

6. P. Brinch Hansen, "The Nucleus of a Multipro-gramming System," CACM, Vol. 13, No. 4, April1970, pp. 238-241, 250.

7. P. Brinch Hansen, Operating Systems Principles,Prentice-Hall, Englewood Cliffs, New Jersey, 1973.

8. P. J. Denning, "Third Generation Computer Sys-tems," Computing Surveys, Vol. 3, No. 4, December1971, pp. 175-216.

9. G. S. Graham and P. J. Denning, "Protection:Principles and Practice," AFIPS Conf Proc., Vol. 40,1972 SJCC, pp. 417-429.

10. B. W. Lampson, "Protection," Proc. Fifth PrincetonConf on Inform. Sciences and Systems, PrincetonUniversity, Princeton, New Jersey, March 1971,pp. 437-443. Reprinted in ACM SIGOPS OperatingSystems Review 8, 1, January 1974, pp. 192-196.

11. D. M. Ritchie and K. Thompson, "The UNIX Time-sharing System," CACM, Vol. 17, No. 7, July1974, pp. 365-375.

12. F. Baskett, J. H. Howard, and J. T. Montague,"Task Communication in DEMOS," Proc. Sixth ACMSymposium on Operating Systems Principles, No-vember 1977, pp. 23-31.

13. P. J. Denning and J. P. Buzen, "The OperationalAnalysis of Queueing Network Models," ComputingSurveys, Vol. 10, No. 3, September 1978, to appear.

14. L. Kleinrock, Queueing Systems Vol. 2: ComputerApplications, John Wiley and Sons, Somerset, NewJersey, 1976, 549 pp.

15. E. G. Coffman, Jr. and P. J. Denning, OperatingSystems Theory, Prentice-Hall, Englewood Cliffs,New Jersey, 1973.

16. B. Randell, P. Lee, and P. Treleaven, "ReliabilityIssues in Computing Systems," Computing Surveys,Vol. 10, No. 2, June 1978.

17. G. J. Popek and C. S. Kline, "Design Issues forSecure Computer Networks," Technical Report,Computer Science Dept., UCLA, Los Angeles, Cali-fornia, January 1978.

18. W. Diffie and M. Hellman, "New Directions inCryptography," IEEE Trans. on Inform. Theory,Vol. IT-22, No. 6, November 1976, pp. 644-654.

19. R. L. Rivest, A. Shamir, and L. Aldeman, "A Methodfor Obtaining Digital Signatures and Public KeyCryptosystems," CACM, Vol. 21, No. 2, Febru-ary 1978, pp. 120-126.

20. P. J. Denning, "Fault Tolerant Operating Systems,"Computing Surveys, Vol. 8, No. 4, December 1976,pp. 359-389.

21. J. H. Saltzer and M. D. Schroeder, "The Protectionof Information in Computer Systems," Proc. IEEE,Vol. 63, No. 9, September 1975.

22. E. Gelenbe and R. R. Muntz, "Probability Modelsof Computer Systems I: Exact Results," ActaInformatica, Vol. 7, No. 1, 1976, pp. 35-60.

23. R. R. Muntz, "Analytic Modeling of InteractiveSystems," Proc. IEEE, Vol. 63, No. 6, June 1975,pp. 946-953.

24. J. B. Dennis and R. P. Misuanas, "A ComputerArchitecure for Highly Parallel Signal Processing,"Proc. ACMAnnual Conf., 1974, pp. 402-409.

25. J. Rumbaugh, "A Data Flow Multiprocessor,"IEEE Trans. Computers, Vol. C-26, No. 2, February1977, pp. 138-146.

26. P. R. Kosinski, "A Dataflow Language for OperatingSystems Programming," Proc. ACM SIGOPS/SIG-PLAN Interface Meeting, SIGPLAN Notices, Vol. 9,No. 4, April 1975, pp. 50-59.

27. Arvind, K. P. Gostelow, and W. Plouffe, "Indeter-minacy, Monitors, and Dataflow," Proc. Sixth ACMSymposium on Operating Systems Principles, Novem-ber 1977, pp. 159-169.

28. A. L. Davis, "The Architecture and Systems Meth-odology of DDM1: A Recursively Structured DataDriven Machine," Proc. Fifth Annual ACM/IEEESymposium on Computer Architecture, April 1978,pp. 210-215.

29. G. Kahn and D. B. McQueen, "Coroutines and Net-works of Parallel Processes," Ibid, pp. 993-998.

30. A. Plas, "LAU System Architecture: A ParallelData Driven Processor Based on Single Assign-ment," Proc. Int'l Conf on Parallel Processing,1976, pp. 293-302.

31. J. L. Peterson, "Petri Nets," Computing Surveys,Vol. 9, No. 3, September 1977, pp. 223-252.

32. D. P. Misunas, "Error Detection and Recovery ina Data Flow Processor," Proc. Int'l Conf on ParallelProcessing, 1976, pp. 117-122.

33. D. E. Denning and P. J. Denning, "Certification ofPrograms for Secure Information Flow," CACM,Vol. 20, No. 7, July 1977, pp. 504-513.

34. D. P. Misunas, "Performance Analysis of a DataFlow Processor," Proc. ACM/IEEE Int'l Conf onParallel Processing, 1976, pp. 100-105.

Peter J. Denning is perhaps bestknown for his invention of the workingset concept for memory management.A professor of computer sciences atPurdue Unpiversity and a prolific writer,he has published over 75 papers andarticles in operating and programmingsystems. His book with E.G. Coffman,Jr., Operating Systems Theory, waspublished in 1973; his book with Jack

Dennis and Joseph Qualitz, Machines, Languages, andComputation, appeared in 1978; and a third book (withJeffrey Buzen), on computer performance evaluation isin preparation.His research interests range widely: theory of computa-

tion, operating system theory and analysis performanceevaluation, protection and security, and software engi-neering. He has held many editorial positions: editor-in-chief of Computing Surveys, editor of the Elsevier/ North-Holland Series on Operating and Programming Systems,and associate editor of technical journals.Denning received the SM and PhD degrees in electrical

engineering from MIT. He is a member of ACM, asenior member of IEEE, and a member of the IEEEComputer Society. He is vice-president of ACM untilJuly 1980.

COMPUTER96