© copyright ibm corporation 2004. ... · developerworks > rational > ... and the pressure to...


Search help

IBM home | Products & services | Support & downloads | My account

developerWorks > Rational >

Issue contents

Editor's notes—August 2004

Compliance with federal and state mandates has been affecting software development practice in many industries over the past decade, and the pressure to bring software systems into spec has only heightened in the post 9/11 era. Nowhere is this pressure being felt more greatly than in the "regulated" industries—banking, defense, transportation, pharmaceuticals and other life-science businesses—which are seeking higher degrees of integration, predictability, and overall quality across their business systems. This month, we interviewed three experts in the life sciences arena, whose insights into the new mandates for computing systems apply equally well across the regulated industry spectrum.

And you'll find much more in the table of contents below!

Happy iterations, Mike Perrow Editor-in-Chief

Features Coping with compliance: Meeting regulatory mandates with Good Systems Practices by Jack Wilber This roundtable focuses on how new compliance issues are affecting companies that build and maintain software systems for regulated environments. It looks at how these companies are improving their development processes and product quality by adopting Good Systems Practices (GSP) that also ensure compliance.

Business modeling practices: Using the IBM Rational Unified Process, IBM WebSphere Business Integration Modeler, and IBM Rational Rose/XDE by Maria Ericsson This article discusses various contexts for business modeling efforts and provides an overview of the technologies IBM offers to support business transition through three phases of e-business adoption: Function, Integration, and On Demand computing.

Teams and projects Scaling down large projects to meet the agile “sweet spot” by Philippe Kruchten This article describes techniques for applying agile software development methods to a large project that normally would be considered beyond the scope of agility. The author explains the communications capability that a software architecture team can offer coding teams focused on architectural chunks, even when those teams are disconnected by geography, culture, and specialization.

Strategic components for identifying and justifying a program by Michael F. Hanford Mike Hanford continues the program management discussion he initiated in the May 2004 issue, looking at the decisions involved in creating a program after you apply a strategic process to identify business needs. He discusses steps and deliverables to justify proceeding with the program, and how to plan make the

issue contents

archives

subscribe

submit an article

contact us

Entire issue in .pdf

Download the entire issue in .pdf (2.9 MB)

Page 1 of 2The Rational Edge: Issue contents

8/13/2004

© Copyright IBM Corporation 2004.

Getting from use cases to code Part II: Use Case Design by Gary Evans The second of a two-part series on transforming the requirements captured in use cases into implementable representations and code, this article presents the steps involved in Use Case Design activity within the Rational Unified Process, or RUP, where technology-dependent decisions are made.

Theory and practice

Measuring up by Gary Pollice Software developers are notoriously bad at planning projects because they can’t estimate schedules or cost reliably, and they fail to evaluate what happened at the end of a project. This article discusses how software developers can measure their work and arrive at more accurate estimates for planning purposes. Read about IBM’s innovative, research-based process for aligning different organizations in the latest issue of Think! When two or more business entities come together, one of the biggest challenges they face is determining how they will work together. With this new methodology from IBM Research, they can work through cultural issues, agree on business practices, and provide the detail and clarity necessary to put these practices into action.

transition to mobilizing the effort.

Book excerpt: What Is Thought?–“The Mind Is a Computer Program” (Chapter 2) by Eric Baum This book argues that the human mind’s complexity is the outcome of evolution, which has built thought processes that act unlike the standard algorithms of computer science. To understand the mind, we need to understand these thought processes and the evolutionary process that produced them in computational terms.

Rational reader Book review: Enterprise Patterns and MDA by Jim Arlow and Ila Neustadt Reviewed by Ben Lieberman Business archetypes and pervasive universal concepts occur consistently in multiple business domains, and have a deep history within most business domains. Based on this premise, the authors provide a catalog of business domain archetype patterns to use as part of a model driven architecture (MDA).


Page 2 of 2The Rational Edge: Issue contents

8/13/2004


Use + - ( ) " " Search help

IBM home | Products & services | Support & downloads | My account developerWorks > Rational

Coping with compliance: Meeting regulatory mandates with Good Systems PracticesJack Wilber Independent Consultant 4 Aug 2004

from The Rational Edge: This roundtable focuses on how new compliance issues are affecting companies that build and maintain software systems for regulated environments. It looks at how these companies are improving their development processes and product quality by adopting Good Systems Practices (GSP) that also ensure compliance.

New demands for compliance are having a significant impact on companies that build, customize, or maintain software systems for regulated industries. First, more and more computer systems must comply with government regulations. Businesses that never had to consider compliance issues before now must follow mandates such as HIPAA (Health Insurance Portability and Accountability Act), the Sarbanes-Oxley Act, the FDA's 21 CFR 11 (Part 11 of Title 21 of the Food and Drug Administration's Code of Federal Regulations),1 or other regulatory requirements. Even companies that have been developing compliant systems for years are finding that more of their internal systems now require compliance—a trend that is accelerating as businesses seek to integrate systems and databases organization-wide. Today, a typical pharmaceutical company's systems may need to comply with HIPAA as well as regulations from the FDA, the EPA (Environmental Protection Agency), and the SEC (Securities and Exchange Commission).

A second major trend is that companies are beginning to rethink the approaches they have long used to design, build, and deploy regulated systems. They are taking a fresh look at available software development tools and leveraging more COTS (commercial off-the-shelf) packages. And to improve their development processes and product quality and also ensure compliance, they are adopting Good Systems Practices (GSP).

To gain some insight into how these trends are affecting software development efforts, Rational Edge writer Jack Wilber spoke with George J. Grigonis, Jr., senior consultant of QA Edge, Inc.; Matthew Magee, senior systems engineer for IBM Rational; and James Bradburn, principal consultant in the regulatory compliance practice at IBM life sciences. Although the discussion below centers on life sciences, these industry experts agreed that software development organizations serving any regulated industry face similar challenges and are adopting similar approaches and technologies.

Jack Wilber for The Rational Edge: Are development organizations in pharmaceuticals and other regulated industries approaching regulated systems validation differently today than they did in the past?

George Grigonis: Regulatory agencies like the FDA have always expected businesses and nonprofits to use current, sound science and technology practices in their regulated operations. That's why the regulations are largely descriptive rather than prescriptive. The FDA expects organizations to follow good practices, but they let them determine those practices, based on their individual business and manufacturing processes.

Back in the early 1980s, when companies started talking about computer systems validation, they were already validating manufacturing processes, sterile processes, and so on. As a result, computer validation was oriented toward manufacturing and formulation practices, which were predicated largely on "waterfall" engineering methods. They relied on document-centric activities to provide evidence for third parties such as the FDA. That way of thinking quickly became locked in stone and then fossilized. The concepts did not evolve along with uses of computing technology. Now, in the context of, say, utility computing,2 the notions of IQ/OQ/PQ3 just don't fit. That doesn't mean

Contents:Notes

About the authorRate this article

Subscriptions:dW newsletters

dW Subscription (CDs and downloads)

Page 1 of 8Coping with compliance: Meeting regulatory mandates with Good Systems Practices

8/13/2004

© Copyright IBM Corporation 2004. http://www-106.com/developerworks/rational/library/content/RationalEdge/aug04/5566.html

you can't use utility computing in your technology base; it means you have to use a different process when assuring the computing service is fit for your specific business use. Many organizations are still trying to force-fit waterfall methods to COTS products, ASPs (application service providers), and so on, but document-centric thinking does not really apply to these technologies. This is in part what drives project costs up astronomically and causes failures; these organizations don't implement solutions that are tailored to their business, and they can't be agile and resilient.

It's time for regulated establishments to examine how they typically acquire and use computing technology and to change by applying current Good Systems Practices that ensure computing environments will be correct, reliable, and fit for their intended purposes. This was computer validation's goal from the beginning.

James Bradburn: Let's consider this subject at three levels. The first layer is FDA guidance, which has been oriented around process control systems and medical devices. Originally, the agency viewed computer systems as a form of equipment, and applied the IQ/OQ/ PQ structure and the waterfall development V-model4 approach to these systems. The second layer includes various industry standard organizations, whitepapers, and articles that constitute the industry's foundational background. This encompasses everything from the GAMP (Good Automated Manufacturing Practice) committee to the PDA (Parenteral Drug Association), IEEE (Institute for Electrical and Electronics Engineers), and OECD (Organization for Economic Cooperation and Development), among others. The third layer is the validation policies and procedures that companies build, based on their understanding of the first two layers. The old perspectives still show up in procedure after procedure, because people trained in the equipment-based and waterfall documentation methods can't let go. Unless you do validation a certain way, they think the sky will fall on you.

However, I think we are now seeing views change within the life sciences industry; organizations are moving away from the old, prescriptive, waterfall, V-model approach toward a more risk-based approach that focuses on quality and value. It is refreshing to hear the agency5 say that it is logical and acceptable to do risk valuation and have a risk-based decision-making process, and to allocate effort toward those ends. Accordingly, the industry is moving to a more analytical, risk-based approach. But it will take time for that approach to work its way through those three levels—first across the length and breadth of the agency, and then throughout the industry.

Matthew Magee: I agree that it will take time to get a consistent approach in regulated environments. As I go from organization to organization, I frequently see stovepiped systems and groups, and their procedures and practices differ dramatically. In pharmaceuticals, for example, manufacturing practices are very different from IT system practices, although the general principles for good practices are the same.

Some organizations assemble a huge task force to evaluate all of their validation processes and then build a new approach from scratch. However, this is enormously expensive, and the resulting processes can get stale really quickly—as evidenced by the condition the industry is in today. A more progressive approach may be to look for a commercial solution that automatically implements Good Systems Practices and is based on existing standards such as IEEE practices, TQM (Total Quality Management) practices, or the Software Engineering Institute's process model, the CMM6 (Capability Maturity Model).

JW: What impact would consistent Good Systems Practices have on regulated industries?

GG: It has always been good business practice to make sure your computer systems are correct, reliable, and fit for use. If organizations could impose these practices in every corner of the business, ranging from COTS-based systems engineering environments to procurement departments that specify, select, contract, and oversee computing services, we'd see dramatic quality and efficiency improvements. Unlike the linear waterfall processes for traditional computer validation, Good Systems Practices are not document- or project-driven. They are the institutionalized processes that a corporation uses to acquire, sustain, and operate computing bases. These processes are matched to the technologies they are designed for, and they are in lock-step with technology and IT concept evolution. The issue of whether you actually follow your process—in other words, "say what you do and do what you say"—is what most interests the FDA, and the evidence rests with the process tools and artifacts you use. With Good Systems Practices in place, we could stop wasting time on documents we think that the FDA might ask for but are otherwise useless. We could also make our organizations more agile and give ourselves the ability to adapt business processes to market needs. If you're not burdened with heavy documentation, then you can respond more quickly to new demands.

JB: Good Systems Practices is one of the main topics in the life sciences regulatory seminars that IBM conducts. We have to remember that the life sciences industry is pretty wide-ranging—from global mega-pharmas down to bio-tech startups. Across that industry we see everything from companies with very detailed, conservative, and prescriptive procedures to those that haven't quite figured out what their procedures should be yet. We also see companies that overkill everything, so that validation has become a documentation exercise that does nothing to promote real system quality. These organizations are not only incurring unnecessary costs but also creating more compliance problems. The FDA may avoid dictating what to put in our procedures, but as George noted, whatever we put in them we have to follow. So the FDA often cites companies for not following their own procedures—because they are too complicated, confusing, or excessively detailed to follow.

Many companies are aware that they are on either the short or long end of this spectrum, and they're seeking the middle. They are


8/13/2004

asking, "What is an optimum approach to getting quality systems—one that avoids both regulatory exposure and compliance overkill?" Those who have already gone too far are recognizing the need to rethink, simplify, and reengineer, their processes. And those that realize they aren't doing enough are seeking optimal levels of cost and effort to maintain a validated system.

JW: Are these challenges unique to the life sciences industry?

JB: Our industry has system quality needs that are similar to those of others. The banking industry, for example, is just as concerned with security and data integrity as we are. And companies seeking to achieve or maintain ISO 9000 certification must demonstrate process quality in their operations, and that requires robust, reliable computer systems. Also, businesses in many industries are seeking to comply with Sarbanes-Oxley, whose provisions may focus on financial controls and reporting, but some of the actions required for compliance have ramifications for computer systems. In general, numerous regulations and requirements that pertain to business operations are generating a strong need for IT controls, and they affect how we must demonstrate data integrity, data protection, and data security.

MM: The pharmaceutical and medical devices industries are terrific examples because they represent the extreme end of regulatory necessity. People's health and well-being depend on system validity. But banking, finance, defense, and transportation must also meet certain regulations, and businesses in these industries are held accountable.

Say you are a financial institution, and you have a design flaw in a solution that provides online stock transactions or portfolio management. If the flaw is significant, and you are accountable, then you are in trouble. Company stockholders, shareholders, or investors in your company may turn around and sue you. And with Sarbanes-Oxley, you'll be forced to trace, investigate, and determine the problem's source. Without Good Systems Practices in place, this would be extremely difficult and very expensive.

JW: Some companies have one set of practices for developing regulated computer systems and another for non-regulated systems. Is this a valid strategy?

GG: It is true that many companies in our industry segregate systems. We perform one set of procedures for systems that are FDA-visible and another for those that are not. But in terms of quality, there should be no difference between regulated and non-regulated systems. And, in reality, many of these systems are integrated with one another. So separating system requirements— doing this for the FDA, this for the SEC, and this for OSHA—just doesn't make sense.

For example, if you have training records that are now part of HR (human resources), but used to be in another department, then you have to open the HR system to FDA inspection. So it makes sense to establish consistent Good Systems Practices across the business. They'll bring huge benefit to project managers, who will be able to institute logical, interdepartmental workflows. And the business overall will eliminate a great deal of risk.

MM: Yes, and the processes should be consistent regardless of where you are in the organization. Some of today's automated solutions can help you achieve that.

As George noted earlier, regulated industries often look to others to see what is happening, then take what they find and formulate their own solutions. However, as I said before, those solutions can become stale very quickly. But there are methodware solutions that recognize all the methodology tasks and also allow you to contour them to your specific needs. For example, IBM® Rational Unified Process,® or RUP,® is not specifically designed to address a particular regulatory industry's needs, but it offers various plug-ins and tools to help you do that. Rather than a prescriptive approach, you want Good Systems Practices—and that is essentially what RUP embodies.

JB: One positive trend is that pharmaceutical companies that have traditionally treated non-regulated and regulated systems separately are rethinking this approach. As George mentioned, that kind of segregation is more and more difficult, because companies are using more integrated databases now. But a more compelling reason not to segregate is that validation is about quality, and that is just as important for a payroll system, which is crucial to your operation, as it is for a clinical or manufacturing control system.

Companies can use a single process to ensure reliability and sustainability for all systems; and for regulated systems they can add a very thin layer of documentation—which they may ultimately decide to make universal. Then you don't have a 30 or 40 percent cost increase if you build a validated system with a documentation-intensive process. Instead, you're using the same efficient process for everything you build or maintain.

JW: It sounds like Good Systems Practices has a lot in common with industry best practices such as continuously ensuring quality. Is that correct?

MM: Good Systems Practices and quality are really the same thing. Organizations like the Project Management Institute7 say that in order to enable continuous improvement, you must try not merely to create products but also to improve your capacity to produce high-quality systems.


8/13/2004

A process that leverages metrics can help. As a simple example, say a company is approaching the release date for a particular application. With a measurement process in place, the project manager can determine whether the closure rate on defects found during the QA cycle is going down or up. And he can use that simple metric to show an inspector that the team is tracking how well their software development activity is going. Metrics can also help managers and inspectors evaluate the system's stability and overall quality. And project managers can aggregate these metrics in a portfolio to evaluate the organization's consistency and productivity. You can leverage that collective information and the relationships among assets to continuously improve your organization's ability to send high-quality applications into production.

GG: I agree that you can benefit enormously by enabling TQM and metrics-based process improvement. QA and QC teams may have to retool so they can perform different roles in such an environment, but this approach would enable them to do everything they should be doing but don't have time for right now.

I can envision a day when the FDA will not have enough resources to pore over a company's documentation. So instead of asking, "Is this system validated?" they'll ask, "What is the technology process supporting this system? Show me the metrics." The FDA already talks about PAT (process analytical technology) and CAPA (corrective and preventative action) for the manufacturing process. It all relates to TQM; so if you apply TQM to building applications, all your processes will be in perfect alignment.

JB: We keep telling organizations that are going around in circles about compliance to step back and just focus on quality. That's what every agency really wants an industry to do. If you focus on quality processes first, then demonstrating compliance will be much easier.

There are situations in which FDA-regulated companies do not use the pharma-oriented IQ/OQ/PQ and/or V-model approach for their system quality methodology, favoring other cross-industry approaches instead. If your quality procedures reasonably assure that the system functions as intended, and you include controls to assure that it will keep meeting its requirements, then you have a demonstrable quality process.

Just remember that in a regulated environment, you need documented proof of these controls. In addition, to facilitate the inspection, it may help to know how your system quality processes map to the structure that the agency inspector is familiar with. This could aid his or her understanding of your quality approach.

JW: Let's talk more about waterfall versus iterative approaches. Do you think waterfall is on its way out in regulated development environments?

GG: I don't mean to knock waterfall. It has its place. For example, it might be suitable for a unique application that you can't buy off the shelf, one that is specifically tailored to your particular needs. But then you must also have enough time to follow a traditional engineering approach — which is to lock down requirements before you go into design. This can only work when nothing is dynamic in your development lifecycle, and that is extremely rare. Once things begin to change, the waterfall approach tends to fall apart. For example, before you can complete a project using COTS components, there will likely be a product refresh, so change management is important from the very start of your project. Waterfall thinking doesn't take that into account; configuration management and change control typically don't enter the picture until the design phase.

Sometimes, when you talk about iterative development—the need to revisit and reevaluate—to people who are waterfall-centric, they just draw arrows to indicate a backward flow on their waterfall model. But that doesn't work; it just creates a series of disconnected whirlpools resulting in chaos, missed opportunities, lost time, and huge potential for failure to meet an important specification. Good Systems Practices for COTS products have the look and feel of iterative practices that facilitate simultaneous tradeoffs between the system's context of use, design and architecture work, and marketplace influences. You constantly have to go back and reevaluate things—specifications, elements, testing—and without an iterative approach and supporting tools, you cannot manage that kind of complex activity with SOP (standard operating procedure) workflow control.

MM: The focus of an iterative development approach is risk management. With a waterfall-based approach, you're not mitigating risk as you go. Waterfall gives you essentially one iteration, and you have limited ability to recover from mistakes. It's not suitable for today's projects, which must deal with rapid change. The new FDA guidance emphasizes risk assessment as a component of product development, and the objective is to focus on issues that present the greatest risk to people who will use the product.

In contrast to a waterfall approach, an iterative approach like RUP emphasizes risk reduction in every phase of the software development lifecycle. It also focuses on risks associated with package deployment and on the risks inherent in integrating COTS packages with existing systems before implementation begins.

JB: We're seeing more and more people adopt risk-based approaches, not only in the design process but also in defining how much testing is enough. Rather than trying to test for every possible thing with a long laundry list of tests, development teams are using a process to determine the systems' and internal modules' relative risk levels. They then use that information to determine what type of testing to do and at what level of detail.


8/13/2004

Whether you use a series of waterfall steps or an iterative approach, the goal is still the same: quality in the results.

JW: What other issues should companies consider when implementing COTS packages in regulated systems?

GG: Using COTS products effectively is a matter of adopting good practices that help you evaluate, acquire, assemble, use, and service a computing environment largely based on marketplace products. It requires an awareness of what COTS products are, and SEI has a very good definition.8 But even more important is an awareness of what role marketplace dynamics play in the evolution of COTS products, because companies have little control over this. The market and the supplier's business needs drive product evolution; if the supplier is acquired by another company, the product direction may change.

I'd add that tools are absolutely necessary because the developmental, documentation, communication, change management, and quality assurance complexities associated with COTS-based systems are much too labor-intensive for manual oversight and control. Packaged application development is further complicated by open source components, which open the "black box" and make internals accessible and visible; and then you have all the supporting tools and information needed to manage and evolve the open source component. So instead of managing one or several black boxes, you have several black boxes and some white boxes. Tools can play a very important role here.

MM: It's also important to remember that companies in a regulated space influence vendors. When companies partner with vendors to create a particular solution, they can describe what they need and affect the vendor's product direction. For example, we have customer advisory boards that tell us what capabilities they'd like to see in our future products. We are not the only company that does this. Clients should be aware that their COTS products will likely have new releases periodically, and that's why they shouldn't depend on a rigid waterfall methodology. Carnegie Mellon's SEI has a terrific whitepaper by Cecilia Albert and Lisa Brownsword9 on how to select and deploy COTS packages, using RUP as a Good Systems Practices model.

If you are going to use a COTS application with an internally developed application, and both systems must comply with one or more regulations, then you must demonstrate a few things: first, that you used Good Systems Practices to select the package; second, that you can show traceability and linkages between assets to prove you are compliant; and third, that you used a common methodology to implement both the systems and the software used to integrate them. You may also need to show that you are partnering with the vendor to extend the COTS package.

JB: COTS application vendors serving the life sciences industry have been learning about computer system validation and are beginning to use testing tools for their own software quality assurance efforts. They're developing applications that have an accompanying set of testing materials to support their customers' compliancy efforts out of the box. Also, now many customers audit the vendors' quality processes and are not compelled to test the entire application themselves. This is another very positive trend.

JW: How can companies find out more about the quality processes behind COTS packages and software development tools they are considering for deployment?

GG: The PDA's Technical Report 32 (TR-32),10, 11 an audit program, was established to provide software customers in regulated markets insight into what their suppliers do and how they do it.12 With this information, a regulated company can better assess the risk of using a particular technology, not just from a performance perspective, but also from a project management perspective. If you are trying to make an agile response to a particular business need that includes a COTS product under development, but the vendor doesn't have good project management capability, then you are at risk. You may not get the product when you need it.

JW: George explained why development tools are so important when leveraging COTS applications. But how important are they to the overall process of building quality software and compliant systems?

JB: Automated tools have tremendous value in terms of ensuring quality and generating documentation for validation. They also help to keep systems in a continuous state of validation. Remember, every time you change something, you have to consider the change's impact and do regression testing. You simply cannot do these things thoroughly in a manual world. Technology gives you the ability to understand and test the effects of changes.

That companies are gradually incorporating these tools to lower costs and boost quality is another positive trend—long overdue, but definitely happening.

GG: I agree with Jim. Today's environments are too complex for manual processes. Automated tools can track and connect artifacts and assets so you don't make a mess of development. Tools also direct workflow, so you have approval-based accountability all along the way. And people cannot jump in and do something they shouldn't do, because tools control data access, too. The FDA expects all of this control and oversight in a structured development environment. Tools are an ideal way to reduce SOP volume, minimize quality assurance labor, and capture and manage the information appropriate to the work being performed.


8/13/2004

MM: IBM Rational integrates process and best practices right into its toolset to create a "tool-directed environment." Development teams can establish relationships between specific solution requirements and confirmation tests. And these tools provide all regulated industries thorough documentation that they can show to an inspector to prove they have a common process across their organization that encompasses Good Systems Practices. For example, instead of producing IQ/OQ/PQ tests to substantiate system validation, one of our clients simply runs IBM® Rational SoDA,® which reaches into each of the tool repositories and builds reports to demonstrate the traceability between assets: requirements to test, requirements to configuration management, requirements to change request management, and requirements to production code. Companies that use disparate tools from a variety of vendors would have a very hard time doing this. But this client has successfully walked through audits by showing these reports to substantiate compliance.

Of course, you have to educate people to bring them up to speed on Good Systems Practices. I am not trying to imply that the tools in and of themselves will implement Good Systems Practices, but once the team understands how to use the tools, they'll have a powerful head start on achieving good practices.

JW: What are the benefits of using a toolset that inherently supports— and is supported by—a proven methodology?

MM: Among other things, such a toolset can help organizations achieve consistency across all development activities. Development skill sets in many companies are based on old technology. These organizations need to move quickly to new tools and technologies when they are implementing COTS applications or trying to leverage new technologies such as J2EE or component-based development. Links in our RUP product connect the methodware and methodology to specific IBM Rational tools for creating or executing assets, defining requirements, creating models, and documenting change requests. These "tool mentors" educate practitioners new to a specific tool about iterative development and other Good Systems Practices. They tell the practitioner explicitly what to do—what button to push, what setting to use.

This all goes hand in hand with compliance. You really cannot demonstrate compliance to an inspector unless you have:

A consistent process in use within the organization—ideally across all parts of it. The ability to document traceability among implementation, design, construction, and deployment assets.

In a tool-directed environment, traceability will happen automatically if you conform to the underlying methodology. And you can actually shorten your time-to-market or system delivery schedules, because you no longer have to establish traceability manually.

JW: What is a good first step for a company that is rethinking the way it ensures compliance? And where can someone go for more information on this topic?

JB: You can start by looking at various system quality methods and practices and deciding which ones might be appropriate for your organization. Specifically, you should consider using an integrated approach across the entire enterprise, adopting a risk-based approach, leveraging technology in your process, leveraging COTS application vendors' quality processes, and gaining a better understanding of the relationship between software quality assurance processes and business process compliance to FDA and other requirements.

GG: In my view, a first step would be to evaluate uniformity of practice across the enterprise for the various technologies you use. A goal is to eliminate segregation between regulated and non-regulated systems and processes— there should be no difference. Then re-evaluate the disciplines responsible for computing support to make sure the right people are doing the right things. Finally, look outside the company for ways to standardize internal practices; and use Good Systems Practices. Institutionalizing such practices is really the trick.

MM: For more information on the issues we've been discussing, I can suggest a few resources. In addition to the Albert and Brownsword whitepaper I mentioned earlier, there is PDA's Web site at http://www.pda.org, and the Audit Repository Center at http://www.auditcenter.com. I have also written a whitepaper that describes possible strategies for the operation, use, and extension of IBM Rational ClearCase and IBM Rational ClearQuest to address electronic signatures and electronic records compliance, using 21 CFR Part 11 as an example. It can be found at http://www3.software.ibm.com/ibmdl/pub/software/rational/web/whitepapers/2003/germ.pdf. And finally, there is good information on IBM Rational's main compliance page: http://www-306.ibm.com/software/rational/solutions/etrans/compliance.html.

James Bradburn is a principal in the Life Sciences Division of IBM Corporation. He has been involved for more than thirty years in pharmaceutical and medical device manufacturing, including plant operations and production management. For the past fifteen years, he has focused on information technology supporting quality functions and regulatory compliance. A member of the life sciences regulatory compliance team, which is focused on FDA and EU regulations affecting pharmaceutical and medical device manufacturers, he specializes in Title 21 regulations that impact production and quality systems, including those that relate to computer systems. He has been a frequent speaker on manufacturing systems, 21 CFR 11 compliance, and computer validation at industry conferences and seminars for more than a decade.


8/13/2004

George J. Grigonis, Jr., systems quality assurance and systems validation professional, has twenty-one years of experience in pharmaceutical industry process validation and computer validation methods for both regulated and non-regulated environments. Formerly he was affiliated with Merck & Co., Inc. as a manager and consultant in information services. He is presently a senior consultant for both QA Edge, Inc. and Cohasset Associates, Inc. An active member of the PDA, he has served on the committee for computer systems validation and as a leader for both PDA's supplier auditing and qualification task group, which developed the TR-32 program, and PDA's task group on Part 11, which developed the GERM guidance for electronic records management. He is also a member of IEEE Computer Society, an affiliate of the Software Engineering Institute at Carnegie Mellon University, and a member of Pharmaceutical Technology's editorial review board.

Matthew Magee joined Rational Software in 1998 and currently serves the pharmaceutical and regulated industries market for the IBM Rational brand. In addition to developing software for Wall Street and the defense and pharmaceutical industries, his accomplishments include several worldwide deployments of IBM Rational products, and the authoring of Rational's position paper on CFR 21 Part 11 (Electronic Records and Signatures). He holds a B.A. in information systems solutions from Rutgers University.

Notes 1 The Health Insurance Portability and Accountability Act (HIPAA) requires businesses to adopt standards for the security of electronically based health information, to be implemented by health plans, health care clearinghouses, and certain health care providers. The Sarbanes-Oxley Act regulates, among other things, acceptable conduct for public companies regarding retention of business records which to today consist largely of electronic-based information. 21 CFR Part 11, according to the FDA, applies to "records in electronic form that are created, modified, maintained, archived, retrieved, or transmitted under any recordkeeping requirement set forth in Agency regulations."

2 For more on utility computing, see http://www-1.ibm.com/services/us/index.wss/rs/imc/al002842?cntxtId=a1000404

3 IQ/OQ/PQ, or Installation Qualification, Operational Qualification, and Performance Qualification, are protocols traditionally used for implementing compliant systems. These terms were derived from practices used to assure that the manufacturing equipment was installed properly and operates according to predetermined specifications and standards.

4 In the V-model software development approach, testing and development have equal weight, and each set of development activities is paired with a corresponding set of test and verification activities.

5 Unless noted otherwise, the "agency" refers to the FDA.

6 The latest development in CMM is the CMMI®. The SEI describes CMMI as, "The Capability Maturity Model® Integration (CMMI®) project is a collaborative effort to provide models for achieving product and process improvement. The primary focus of the project is to build tools to support improvement of processes used to develop and sustain systems and products. The output of the CMMI project is a suite of products, which provide an integrated approach across the enterprise for improving processes, while reducing the redundancy, complexity and cost resulting from the use of separate and multiple capability maturity models (CMM®s)." See http://www.sei.cmu.edu/cmmi/background/conops.html for more information.

7 http://www.pmi.org

8 From "An Activity Framework for COTS-Based Systems" at http://www.sei.cmu.edu/pub/documents/00.reports/pdf/00tr010.pdf

"A COTS product is a product: sold, leased, or licensed to the general public offered by a vendor trying to profit from it and supported and evolved by the vendor, who retains the intellectual property rights available in multiple, identical copies used without modification of the internals."

9 To download this paper, "Evolutionary Process for Integrating COTS-Based Systems (EPIC): An Overview—Key Elements in Building, Fielding, and Supporting Commercial-off-the-Shelf (COTS) Based Solutions," go to: http://www.sei.cmu.edu/pub/documents/02.reports/pdf/02tr009.pdf

10 PDA Technical Report 32: "Auditing of Suppliers Providing Computer Products and Services Used for Regulated Pharmaceutical Operations." Supplement to Journal of Pharmaceutical Science and Technology, Vol. 53, No. 6, 1999.

11 Case study: "Computer Supplier Evaluation Practices of the Parenteral Drug Association (PDA)." CMU/SEI-2003-TR-011, May 2003.


8/13/2004

12 To examine audits for specific software tools, including IBM® Rational ClearCase® and IBM® Rational ClearQuest,® visit http://www.auditcenter.com.

About the author Jack Wilber has worked with Rational Software as an independent consultant since 1998. In that time he has either authored or co-authored many whitepapers, case studies, product datasheets, and even an article or two for The Rational Edge. When not writing for Rational, Mr. Wilber spends his time developing software out of his home office in South Carolina. He has more than ten years of experience in software development and holds a B.S. in Computer and Electrical Engineering from Carnegie Mellon.

What do you think of this document?

Comments?

Killer! (5) Good stuff (4) So-so; not bad (3) Needs work (2) Lame! (1)

Submit feedback developerWorks > Rational



8/13/2004




Business modeling practices: Using the IBM Rational Unified Process, IBM WebSphere Business Integration Modeler, and IBM Rational Rose/XDEMaria Ericsson Principal Consultant, IBM 6 Aug 2004

from The Rational Edge: This article discusses various contexts for business modeling efforts and provides an overview of the technologies IBM offers to support business transition through three phases of e-business adoption: Function, Integration, and On Demand computing.

As organizations are moving to become more agile and On Demand, the need to take an engineering approach to connecting business and IT is increasing.

If you are interested in understanding why business modeling might be useful and would like an overview of the IBM solutions for business modeling, please keep reading. Ideally, you already have a basic understanding of the concepts of business modeling as defined by the Rational Unified Process. Several introductory articles on business modeling concepts can be found in The Rational Edge archives.

The purpose of this paper is to explore contexts for business modeling efforts and to provide an overview of the technologies IBM offers to support this. As I proceed, I will refer to various IBM software products that offer capabilities relative to the discussion at hand.

The essential concepts These are business engineering, business modeling, and business process management. The Rational Unified Process discusses business modeling using UML modeling tools such as Rose/XDE and how it supports business engineering. The WBI Modeler in combination with the rest of the tools that constitute the IBM WebSphere Business Integration tool set support business process management. Let's review these concepts in order.

Business engineering Business engineering is a set of techniques a company uses to design its business according to specific goals. Business engineering techniques can be used for business reengineering, business improvement, and business creation.

Business reengineering. This form of business engineering yields changes based on a comprehensive view of your existing business and a thorough evaluation of how and why it functions as it does. You question all existing business processes and try to find new ways of reconstructing them to achieve radical improvements. Other names for this are business process reengineering (BPR) and process innovation.

Business improvement, This form of business engineering yields local changes that do not span the entire business. It involves trimming costs and lead times and focuses on monitoring service and quality.

Business creation. This form of business engineering is used when the goal is to create a new business process, a new line of business, or a new organization.

Contents:The essential conceptsObjectives of a business modeling effortWhy do you want to do this?Tools for moving through the stages of e-business adoptionProcess for business modeling ConclusionReferencesAcknowledgementsAppendix A: Translation of Rational / WebSphere conceptsAppendix B: IBM products for business modelingAppendix C: Notes on standards




Page 1 of 24Business modeling practices: Using the IBM Rational Unified Process, IBM WebSphere Business Integr...

8/12/2004


Business modeling Business modeling encompasses all modeling techniques you can use to visually model a business. These are an essential subset of the techniques you may use to perform business engineering.

Business process management (BPM) Business process management is the concept of continuously defining, analyzing, and improving a business process.

Objectives of a business modeling effort In a business world that is increasingly competitive, companies frequently find that their organization structures and processes are not allowing them to adapt to change quickly enough. Many of these companies have decided to adopt modern business practices and IT technology that supports them—a change that affects many aspects of an organization. The following challenges are common:

Business processes are not visible to people working in the organization. If there is no common baseline to start from, it is difficult to discuss what to change. Different parts of an organization use different variants of the same process. This lack of alignment in practices and terminology causes confusion and makes it difficult to assess what should be changed. Technology has become essential to how business is run. Both the business management and IT management teams see the need to collaborate more than they traditionally have. Companies that didn't used to think of their products and offerings as software now find that software is how the products are represented. For example, much of the financial and banking industry has been changed over the past five to ten years through online representations of essential service offerings.

Business modeling, which refers to the techniques for visualizing and reasoning about processes and structures in the organization, is therefore becoming more important. Currently, there are no obvious standards for business modeling notation; there are, however, efforts underway to align several common practices. Standardization initiatives are driven by BPMI (Business Process Management Initiative, www.bpmi.org) in collaboration with the OMG (Object Management Group, www.omg.org). For more background on standards, see Appendix C.

So, let's imagine that you undertake business modeling under the simplest scenario, in which you only want to understand or align business processes. As a depicted in Figure 1, you would do this by visualizing the processes and then make improvements based on business requirements understood through executing the processes. You will see varying levels of formality in the modeling efforts, all depending on the audience and focus of the effort. Some practitioners use white boards and PowerPoint only, while others may use visual modeling tools such as Rose/XDE to do more formal UML modeling.

Figure 1: A simple scenario for business modeling.

A more complex scenario involves business modeling in combination with build process monitoring capability. As shown in Figure 2, as processes are executed, data is collected and performance is analyzed in order to generate business requirements for process change. This is the strength of the IBM WebSphere Business Integration tool set, and the capability it provides is referred to as business process management (as defined above). The current notation used for business modeling and process monitoring is inherited from the Holosfx products that IBM acquired a few years ago. The WBI Modeler also makes it easy to record costs and timing of tasks, which are required to do simulation of a workflow.


8/12/2004

Figure 2: Using performance analysis to refine business requirements for the business modeling effort.

An even more sophisticated variation combines business modeling and process monitoring with application reconfiguration capability (automatic reconfiguration of an application based on changes in the business model). Business requirements for change may require changes in the functionality of existing software applications (IS solutions). In this case, as shown in Figure 3, the software requirements will lead to changes in the parameters by which the applications are run. Again, this type of reconfiguring is supported by the WebSphere Business Integration tool set.


8/12/2004

Figure 3: By intoducing software requirements into the stream of analysis, applications can be reconfigured automatically as a result of matching business requirements to actual performance.

Finally, you may do business modeling in combination with application development. If business requirements demand new functionality in your applications, or if the business is going through an automation effort, business modeling could be input to an application development effort in order to generate software requirements. This is the strength of the IBM Rational Software Development Platform. The business modeling notation used for this comprehensive process and toolset is based on the standard UML.


8/12/2004

Figure 4: When application development is added to your fundamental business processes, businesses are able to tailor IT systems to directly address known business requirements.

Now, let's look at the big picture, as shown in Figure 5. The IBM Software Development Platform combines business modeling with other capabilities to monitor, reconfigure, and/or build application support for the business processes.


8/12/2004

Figure 5: The essential capabilities of the IBM Software Development Platform, which lead to business IT system improvement.

Why do you want to do this? In order to remain competitive, clients we work with are going through three basic stages of e-business adoption, depicted in Figure 6: Function, Integration, and On Demand.

Function. The first stage, typically, is to build processes and IT support for domains or functions in the business. This is where most organizations are today. The understanding of the business process as well as the supporting applications tend to be specific to the functions or domains. For example, international corporations often rely on country-based expense-reporting procedures and supporting applications. The disadvantage is that there is some overlap in features and also perhaps difficulties in communicating information across national borders.

Integration. The second stage is to integrate domain-specific IT systems to eliminate overlap and to align processes and information structures across the organization. This should result in fewer applications to maintain and easier communication across borders or traditional business boundaries. However, as this is implemented, businesses tend to find more overhead in adapting to the needs of the functions or domains, and the organization is perceived as more bureaucratic. For example, adding or modifying an expense-reporting system to meet changes in local laws and regulations for overtime work might all of a sudden become rather complicated and costly as it now affects an application the whole organization uses.


8/12/2004

On Demand. The third stage is to create something that combines the best of both worlds, with a focus on the marketplace in certain niches, but with the ability to integrate business-led processes and IT capabilitites horizontally across the enterprise for efficiency and for speed of response and resiliency. Very few businesses have achieved this. No one would even attempt this unless they had to in order to remain competitive. But increasingly, organizations are striving to reach this stage in order to survive. This is what IBM's On Demand vision is all about.

The demands on business modeling are different in each of the three stages described above; business engineering becomes increasingly important as you go through the stages. Understanding business processes and business architectures is essential in order to deliver the appropriate technology; therefore, business modeling is critical to the IBM On Demand strategy.

Figure 6: The three stages of e-business adoption.

Tools for moving through the stages of e-business adoption Currently, two IBM product sets exist to support business modeling. These product sets are related, although they have different purposes:

The IBM Business Performance Management Framework, including the WBI Modeler for business modeling using a workflow-based proprietary notation. WBI Modeler can import and export UML models. It can also export model information to the WMQ Workflow tool. The IBM Software Development Platform. Includes an approach to business modeling using UML and Rational visual modeling tools (Rose/XDE and the Rational Unified Process). As the name indicates, this set of tools is targeted to the software developer audience; though the business modeling portion of the solution is aimed both for business developers as well as system analysts and architects.

If you look at a more complete picture of support for helping clients migrate through the stages of e-business adoption, you'll see three main components:

A middleware platform that forms the basis for all applications. In this platform you may see an enterprise application server such as WebSphere Application Server, and possibly also a workflow engine such as WebSphere Foundation Server for monitoring of business process execution. A set of application development tools—requirements management tools such as Rational RequisitePro, visual modeling tools such as Rational Rose/XDE, and construction tools such as WSAD. A set of business modeling tools such as WBI Modeler or Rose/XDE possibly combined with workflow orchestration tools such as WebSphere MQ Workflow.


8/12/2004

Figure 7: The tools for e-business adoption. The strengths of the IBM Business Performance Management Framework lie in the left side of the picture, while the IBM Software Development Platform is focused on the right side.

Impact of process characteristics Another dimension to consider is the characteristics of the business processes being described. The IBM Business Performance Management Framework products lend themselves to modeling and managing processes that are repeatable in nature. In these types of processes, known as transactional processes, the same sequence of steps, with minor variations, is repeated every time the process is executed. This is typical for production processes and order handling processes, for example.


8/12/2004

Figure 8: An example of a transactional process, in which the same sequence of steps gets repeated every time the process is executed.

But there are also processes that are much less predictable in nature. For example, in a software development process, it is very difficult to define predictable sequences of activities. Activity and workflow diagrams used in UML-based modeling can never be prescriptive, only illustrative; the same is true of workflow diagrams in RUP. Modeling these processes needs to be focused on the results achieved along the way, what responsibilities need to be carried out, and not a prescriptive sequence of activities. We call these processes non-transactional; the strength of the UML (among other things) is that it supports class diagrams and interaction diagrams in addition to activity/workflow diagrams.


8/12/2004

Figure 9: A UML activity diagram example.

The IBM technology for business process management (IBM Business Performance Management Framework software) is, as mentioned before, most effective for transactional types of business processes. Monitoring and management of more unpredictable processes require an expanded set of business notation and monitoring techniques like those offered as part of the IBM Rational Rose/XDE/UML approach. IBM Rational monitoring technology (the IBM Rational Project Console) focuses project management attention on the status of the deliverables of the process and achievement of milestones, rather than on what steps of the process may have been conducted.

For clients working in scenarios where both the IBM Business Performance Management Framework offering and the IBM Rational RUP/Rose/XDE offering are needed, IBM offers integrations between WBI Modeler and IBM Rational RUP/Rose/XDE. Note that these two approaches have differences in their styles of notation; Appendix A provides a table that translates key concepts used in WBI Modeler and IBM Rational RUP/Rose/XDE, respectively. The notations are overlapping, but offer different strengths as indicated above.

Process for business modeling

The business modeling discipline in RUP Of the scenarios for business modeling outlined earlier in the "Objectives of a business modeling effort" section, any given organization may find itself engaging several. A "common way of working" tends in real life to mean a common framework for how to do business modeling, with variations depending on the usage scenario. In order to be able to harvest and reuse experiences, it is critical that the terminology, approach, and notation you use are standardized. As you gather experiences from business modeling efforts, you may gather a set of adaptations of the framework to typical usage scenarios. Those adaptations should be based on experiences in your organization. For example, the business modeling discipline in the Rational Unified Process (RUP) outlines the workflow shown in Figure 10.

Page 10 of 24Business modeling practices: Using the IBM Rational Unified Process, IBM WebSphere Business Inte...

8/12/2004

Figure 10: RUP workflow diagram.


8/12/2004

This graphic can serve as a framework for all usage scenarios of business modeling, although the emphasis of the steps will be different in each scenario. The original workflow from RUP has here been extended with a workflow detail "Analysis of Model" to further emphasize how models can be explored to analyze expected behavior. RUP offers some guidelines on how to do this, however the addition of the WBI Modeler and its guidelines broadens this guidance.

Let's consider a subset of the workflow details shown in Figure 10 that are relevant to modeling (for complete details of the business modeling workflow, you should refer to RUP itself).

Describe current business. The key benefits of this step are an understanding of current business context, a first insight into business needs, and a definition of the scope of the modeling effort. An example of diagrams that could be produced (via Rose/XDE) are context diagrams (a variant of class diagrams), as shown in Figure 11.

Figure 11: Example of a simple context diagram.

In some cases, you might find the business modeling effort is only relevant for a part of the business, in which case a more detailed context diagram might be produced, as shown in Figure 12.

Figure 12: Example of a more complex context diagram.

Identify business processes. The key benefit of this step is an understanding of what process areas require change compared to current processes. This is facilitated by producing overview diagrams that show high-level processes as business use cases and business actors. What you build here (and what you may continue in the "Refine business process definitions" workflow) is what RUP refers to as the business use case model. An example of this model, created using Rose/XDE, is shown in Figure 13.


8/12/2004

Figure 13: Example of a business use case model.

Refine business process definitions. The benefit of this step is a more detailed definition of the business processes, but one that is still understandable to someone outside the organization. This is about describing from an external viewpoint what the processes are, without including internal details such as information structures or what roles are involved. Typically, this is done via text documents, although some also illustrate the business process flows with simple activity diagrams.

Design business process realizations. The benefit of this step is a description of how roles collaborate to perform the process, and what information objects are used, managed, or produced. This is where you get into the details of the process and start building what RUP refers to as the business analysis model. Common output from this step is one or more activity/workflow diagrams using either WBI Modeler or Rose/XDE, depending on your context. For example, if the process is highly transactional, and is a candidate for being implemented in a workflow engine, you'd choose WBI Modeler. An example of this output is shown in Figure 14.


8/12/2004

Figure 14: Example of a business process realization modeled with WBI Modeler. Click to enlarge.

The same activity/workflow diagram in Rose/XDE using UML would appear as shown in Figure 15.


8/12/2004

Figure 15: The same activity/workflow depicted in Figure 14, rendered here using Rose/XDE and UML.

Another diagram type used more for technical audiences is the sequence diagram, which can be created via Rose/XDE as shown in Figure 16.


8/12/2004

Figure 16: Designed for technical readers, the sequence diagram offers a step-by-step view of the business process being modeled.

So far, we have only focused on the dynamics of the process. It may also be useful to look at static structures among classes using class diagrams. For example, a view of participating objects in a business use case or process is shown in Figure 17.


8/12/2004

Figure 17: Example of a class diagram, showing a view of participating objects in a business use case or process.

You may also explore structures in information, expressed as business entities in Rose/XDE, as shown in Figure 18.

Figure 18: Rose/XDE allows you to model information structures expressed as business entities.

Refine roles and responsibilities. Unless you need to simulate your workflows, you would typically not spend a lot of time on detailed descriptions of the classes in the business analysis model. Instead, you might (for example) investigate state changes to explore lifecycles of key business entities (Rose/XDE), as shown in the statechart diagram in Figure 19.


8/12/2004

Figure 19: Example of a statechart (state transition) diagram drawn in Rose/XDE.

If you are building your model in WBI Modeler and plan to do simulation, you will need to capture details around cost and schedules for resources. Figure 20 shows an example of how business workers/roles are defined in WBI Modeler.


8/12/2004

Figure 20: WBI Modeler helps you capture details around cost and schedules for resources. This portion of the user interface shows how business workers/roles are defined.

Analysis of models. The Rational Unified Process does not currently put a lot of emphasis on analysis or simulation of the models. There is some content that describes activity-based costing, which WBI Modeler tool supports a variant of. Assuming you have captured details of resource cost and activity duration, as described in the previous step, WBI Modeler allows you to generate various charts to analyze the execution of a workflow scenario. Figure 21 shows a sample chart with cost per activity for an instance of a workflow (scenario).


8/12/2004

Figure 21: WBI Modeler allows you to generate various charts to analyze the execution of a workflow scenario. Here is a sample chart with cost per activity for an instance of a workflow (scenario).

Another variant of analysis is to simulate the progress of the process. Figure 22 shows a process "snapshot" created for visual reference within the WBI Modeler tool. The higher numbers of little red boxes next to the activities/tasks indicate higher workload at each task, at the point in time the snapshot was taken.

Figure 22: WBI Modeler allows you to simulate the progress of the process. Click to enlarge.


8/12/2004

Reusable model frameworks

IBM's business consulting services team is currently addressing the business engineering needs of our clients by developing an approach we call Component Business Modeling (CBM). Today's interconnected firms face a business environment that is challenging on multiple levels; organizational structures and strategic alliances constantly shift in response to rapid-fire marketplace changes. The idea is to speed the transformation of an organization by providing frameworks for agile or On Demand businesses. Currently, CBM is primarily available for our customers in the financial industry.

But in a general sense, some CBM principles and techniques, such as the definition of a Business System, are available in RUP today.

Integration vs. development projects An additional set of factors to consider in business modeling have to do with the general purpose of the project; that is, whether you are attempting to integrate process and applications, or to develop or automate processes. Consider the following:

In traditional application development, the effort needs to include architectural specifications (interfaces, mechanisms, components) as well as low-level constructs such as classes and methods. UML offers full support for this activity. The components of an integration solution are somewhat different: adapters, business objects, diagrams representing individual steps of processing, etc., are the key constructs. The work on an integration project is usually concentrated on tying together existing people, processes, and applications (perhaps with some satellite development to support integration at the end points).

Therefore, if you intend detailed business modeling work to result in development work, the choice of implementation technology has a significant impact on the format (and potentially the total content) of the modeling work produced.

At a high level, one can consider the IBM Business Performance Management Framework (using WBI Modeler) approach to be well-suited to modeling transactional processes for integration projects and the IBM Software Development Platform (using Rose/XDE and UML) approach to be well-suited to modeling processes that will result in new system functionality or architectures. The two sets of tools can be used together (e.g., Rose/XDE for modeling business requirements and business analysis models, WBI for transactional business processes).

Conclusion Business modeling is a key to moving to an On Demand enterprise. IBM is focusing on business modeling to drive:

Process monitoring Process integration Process creation Process automation

Currently, IBM Software Group offers two complementary product groups for conducting business modeling:

IBM WebSphere Business Integration (WBI) Modeler, which ties into IBM products for process orchestration and monitoring. WBI Modeler supports a subset of the modeling activities defined in the RUP business modeling discipline—mainly visualization of business use case realizations. WBI Modeler also offers simulation capability, which allows you to analyze the models. IBM Rational Rose/XDE, which supports full-scale UML business modeling and ties in to the IBM Software Development Platform and all modeling activities defined in the RUP business modeling discipline.

Depending on your business modeling context, you may choose either or a combination of these modeling approaches.

References

BPMI (Business Process Management Initiative) - www.bpmi.org OMG (Object Management Group) - www.omg.org


8/12/2004

Continuous Business Process Management with HOLOSOFX BPM Suite and IBM MQSeries Workflow—an IBM Redbook, www.Redbooks.com CBM (Component Business Modeling)—IBM Executive Brief The Rational Unified Process 2003

Acknowledgements I would like to thank Jasmine Basrai, Sr. Product Planner in the WBI Modeler team, with whom I developed a presentation for the Rational Software Development User Conference in Dallas 2004, upon which this paper is based.

Simon Johnston, IBM Rational, who helped sort out which standard does what for whom.

Alan W. Brown, IBM Rational and IBM Distinguished Engineer, who initiated much of this work and helped keep us on track.

Christina Gerde and Anna-Lena Henriksson of AstraZeneca, Sweden, who have contributed much to the understanding of clients' needs in the areas of business engineering, business modeling, and business process management.

Appendix A: Translation of Rational / WebSphere concepts Translation of key concepts in RUP/Rose/XDE to WBI Modeler. For definitions, please refer to the WBI Modeler user guide and to the Rational Unified Process:

Appendix B: IBM products for business modeling

RUP/Rose/XDE Concept WBI Modeler Concept

Business Use Case -

Activity Graph (diagram) Process

Business Actor External Entity

Business Goal -

Activity Task

Business Entity Phi

Business Worker Role

Package Organization Unit

Business System Organization Unit can be used

Business Event -

Business Rule -

Class diagram -

Collaboration diagram -

Sequence diagram -

Term Explanation

The IBM Rational Project Console

A metrics collection tool that dynamically creates a project Web site with a progress metrics dashboard based on data collected from the IBM Software Development Platform.

The IBM Software A platform for automating software development consisting of a set of integrated components covering all


8/12/2004

Appendix C: Notes on standards OMG and UML

The Object Management Group (OMG) is responsible for the Unified Modeling Language (UML) and Meta-Object Facility (MOF) standards. These standards have traditionally been seen as relevant in the software development process and are heavily used and supported by the IBM Software Development Platform.

From the Introduction to OMG UML: The OMG's Unified Modeling Language™ (UML®) helps you specify, visualize, and document models of software systems, including their structure and design, in a way that meets all of these requirements. (You can use UML for business modeling and modeling of other non-software systems too.)

The UML is a general-purpose modeling language although its roots are in modeling software systems and specifically Object-Oriented Programming. The language is based on a modeling architecture rooted in another language, MOF, which provides the language used to describe the UML itself. MOF is also important to IBM as it is the modeling language implemented by the Eclipse Modeling Framework (EMF), which is used extensively by our tools across all brands. Specifically, the UML 2.0 model is the metamodel for the future IBM Rational modeling products. The WBI Modeler implements its own metamodel, called BOM, that is based upon UML 2.0 and reuses substantial parts of the UML while adding extensions to support business process specification. BPMN and BPMI.org The Business Process Modeling Notation (BPMN) was developed by BPMI.org as a method for describing processes in a manner that could be easily translated to technology-specific standards such as BPEL.

From the BPMI.org Web site: The Business Process Modeling Notation (BPMN) specification provides a graphical notation for expressing business processes in a Business Process Diagram (BPD). The BPMN specification also provides a binding between the notation's graphical elements and the constructs of block-structured process execution languages, including BPML and BPEL4WS. The first draft of BPMN was made available to the public on November 13, 2002.

The next generation IBM WBI Modeler has taken into account the BPMN specification when the development team was working with customers and usability teams in the definition of the notation that the modeler implements. This notation borrows elements from the UML 2.0 notation, from BPMN and from existing IBM products such as the previous WBI Modeler 4.2.4.

BPEL or BPEL4WS The WBI Server Foundation middleware includes, among other things, a Business Process Execution Language for Web Services (BPEL4WS or simply BPEL) runtime component. BPEL is thus the language used to technically specify the workflow that is to be

Development Platform functions in a software development project.

The IBM Business Performance Management Framework

A set of tools that together executes and manages processes that span diverse applications and enterprises. This includes the business process modeling and simulation tool, a workflow engine tool, and functionality to analyze the execution of the workflows. It primarily focuses on supporting business process management and also helps with integration of business processes. The notation used is workflow diagrams.

The IBM WebSphere® Business Integration Modeler

Also referred to as WBI Modeler, part of the IBM WebSphere BPM software. A visual modeling tool for business process modeling and simulation.

IBM WebSphere MQ Workflow

Part of the IBM WebSphere BPM software. A workflow orchestration tool.

IBM Rational Rose/XDE

Supports visual modeling with UML. Part of the IBM Rational Software Development Platform.

IBM Rational Unified Process

A customizable framework for the software development process, including business modeling.


8/12/2004

executed.

FDL and WebSphere MQ Workflow FDL (FlowMark Definition Language) is the native file format describing workflows for the WebSphere MQ Workflow middleware. WBI Modeler generates FDL that can be imported into MQ Workflow. This is not a general standard proposed by a standardization body, but is mentioned since it may be referred to.

Architecting and managing business and information systems It is probably too early to discuss standards for business architectures, but various frameworks have been put in place to help manage the business as a whole including its information systems.

A few examples:

The Zachman Framework emerged in the early 1990s, focusing on how you build enterprise architectures to ensure information systems are more business focused. More recently, the Component Business Model efforts in IBM's Business Consulting Services organization, work to define standard frameworks for the on demand business.

About the author Maria Ericsson is a principal consultant for IBM Rational's Strategic Services Organization (SSO). She started working in the field of software engineering and object technology in 1990 at Objectory AB, and co-authored Ivar Jacobson's book, The Object Advantage: Business Process Re-engineering with Object Technology. Since joining Rational in 1995, she has worked as a mentor and trainer in process, business modeling, and requirements management, and also spent three years as a member of the Rational Unified Process,® or RUP,® development team. As part of the SSO, she currently focuses on solution deployment strategies and serves on the IBM Rational field training team. A resident of Sweden, she is based in the

Kista office.


Comments?





8/12/2004




Getting from use cases to code Part II: Use Case DesignGary Evans Independent Object Technology Evangelist, Evanetics 10 Aug 2004

from The Rational Edge: The second of a two-part series on transforming the requirements captured in use cases into implementable representations and code, this article presents the steps involved in Use Case Design activity within the Rational Unified Process, or RUP, where technology-dependent decisions are made.

This is the second part of a two-part series on transforming the requirements captured in use cases into implementable representations and code. In "Part 1, Use case analysis," I traced the major steps in the Use Case Analysis activity of the Rational Unified Process®, or RUP®1. I introduced a simple case study for a vehicle rental system (let's now call it Deals on Wheels) to be developed to offer browser-access to customers wishing to reserve a vehicle, cancel a reservation, view a rental history, etc. I presented a single use case, Reserve a Vehicle, first in a very generic version, then in a supplemented version, introducing more of the external interactions that have to take place in this use case.

Then I used the simple technique of grammatical dissection to identify candidate entities in the use case, and introduced four questions to challenge these candidates. These questions distilled our forty-one candidates down to eight entities that passed the test of being analysis classes. Then I identified the responsibilities of each analysis class, so we could have what James Rumbaugh et. al. call a "crisp boundary" in our definition of each class.

With the eight analysis classes identified, I then introduced four steps that I used to construct an initial analysis class diagram to capture the relationships and multiplicities among these classes. Next, I constructed an analysis-level sequence diagram of the main scenario of the Reserve a Vehicle use case. This was strictly constrained to be an analysis-level diagram containing only analysis classes, and no design or implementation classes. To improve understandability of the sequence diagram, I introduced a generic use case controller object to mediate the messages received by the human customer actor. With the analysis classes and their relationships identified, I then populated each analysis class with attributes (data members) that were in accord with the responsibilities of each class. Lastly, I identified the analysis mechanisms for this little case study.

In Part II of this series, we will perform the steps of RUP's Use Case Design activity.

The elements of Use Case Design in RUP Figure 1 is taken from RUP's Analysis and Design Activity Overview. It illustrates where the Use Case Design activity occurs within the context of the other Analysis and Design activities.

Contents:The elements of Use Case Design in RUPUse Case Design step 1: Create use case realization for each use caseUse Case Design step 2: Describe interactions between design objects Use Case Design Step 3: Simplify sequence diagrams using subsystems (optional) Use Case Design Step 4: Describe persistence-related behavior Use Case Design Step 5: Describe design mechanismsConclusionReferencesFurther ReadingNotes




Page 1 of 16Getting from use cases to code Part II: Use Case Design

8/13/2004


Figure 1: Use Case Design activity in RUP

In RUP the stated purpose of Use Case Design is:

To refine use case realizations in terms of interactions To refine requirements on the operations of design classes To refine requirements on the operations of subsystems and/or their interfaces.

We can alternately express that the goal of Use Case Design is to transform each business abstraction in our system (i.e., the analysis classes) into one or more design classes that are implementable representations, taking into account the properties of the target execution environment. Note that the design classes are implementable representations. The implementation (i.e., coding) of the design classes occurs after design is stable (although we sometimes write code to explore design options). In design, we want to express, in appropriate detail, how we will approach the implementation so that the actual programming is a matter only of language and platform issues. Figure 2 illustrates the steps that constitute Use Case Design.

Figure 2: The steps of Use Case Design


8/13/2004

The activity of design involves maintaining multiple perspectives simultaneously on the software system you are building. Good designers successfully juggle separate, and often competing, views of their target system. Just as a cube has six faces which are independent yet part of the same whole, a software system's design is multi-faceted, including: logical, process, implementation, deployment, and operational views; system and software architecture; components and classes; patterns and interfaces; static structure and interactions, and more. I am not going to cover all of the practices of software design in this article. But I will provide some concrete examples of what we might actually do inside the major Use Case Design steps, as shown in Figure 2.

Classes Object-oriented and component-based design strategies focus on classes—which are named units of data and bound functions that operate on that data. In the design discipline, we describe our classes, and their relationships to other classes or subsystems, in a manner that now takes all technology concerns into account.

Some of these concerns are:

How do I represent a one-to-many, or many-to-many, relationship between two classes? How do I represent access to and from a data store? What aspects of my class should be visible to objects of other classes? Which aspects should be invisible to other classes? How do I provide a simplified interface to multiple objects? How do I appropriately separate the business behavior in my business classes from the technology behavior that my executable system requires?

The Design Model The result of our activities in Use Case Design will produce the contents of the RUP Design Model, including (but certainly not limited to):

The design classes (i.e., analysis classes plus technology classes) needed in our system. Realization of the Software Architecture Document (SAD) content through the design classes. The operations and attributes that each design class will own. Specification of actual data types, initial values, or interfaces for specific subsystems or vendor-provided software. Partitioning of our system functionality by subsystem or architectural category. Interaction (i.e., collaboration and sequence) diagrams showing how our design classes will collaborate to carry out the business process captured in our use cases.

The Design Model will become the raw material that our implementation discipline will transform into executable code. I reiterate that this discussion will focus on a pure object-oriented approach for new system development (often called "green-field" development) in C#, Java, or C++.

The steps we will discuss When I do Use Case Design activity, I follow the steps depicted in Figure 2, with the exception that I add a step, as you can see in the following revised list:

Create use case realizations Describe interactions between design objects Simplify sequence diagrams using subsystems (optional) Describe persistence-related behavior Define design mechanisms (normally a Software Architect activity in RUP)


8/13/2004

Refine the flow of events description Unify classes and subsystems Evaluate your results

It is important to recognize that the order of these steps is not fixed. The sequence you follow may differ based on your understanding of the domain you are designing, your experience with RUP, your personal preferences for the models you use, or the metaphor you follow for characterizing the properties of your design classes (e.g., applying design patterns, or following principles such as Separation of Concerns or the Interface Segregation Principle). What is important is that you achieve a comprehensive expression of the solution approach you will eventually implement.

Of the steps listed above, we will look closely at the subtleties of only the five following steps:

1. Create use case realizations

2. Describe interactions between design objects

3. Simplify sequence diagrams using subsystems (optional)

4. Describe persistence-related behavior

5. Define design mechanisms

Architectural criteria To get started, since design is intimately constrained by and related to architecture, let's consult our Software Architect (SA) and get an idea of the overall architecture envisioned for our reservation system. For this discussion, let's stipulate that the SA has produced a Software Architecture Document (SAD) that indicates our architecture must support the following technical criteria:

1. Scalability. Multiple, simultaneous customer sessions must be supported in the Deals on Wheels reservation system and RDBMS. Each session must be independent of all others, and the system must scale transparently to thousandss of simultaneous customers.

2. Maintainability. Updates to software components must be made easily and centrally, with no distribution of business logic to browser platforms.

3. Technical simplicity. Our rental company does not want to incur the expense of training or hiring staff competent in full-fledged component environments such as J2EE or .NET. These are overkill for our limited needs. The existing staff are knowledgeable in Java, Java ServerPages, servlets, and JavaScript.

4. Leverage technology. The Internet must be the communications medium between browser and servers and the system must use existing programmatic interfaces to legacy platforms.

With these criteria as givens, and knowing that the Deals on Wheels project has Java and servlet developers on staff, the SA adds to the SAD the simple UML architectural models shown in Figure 3.


8/13/2004

Figure 3: Initial deployment diagram for kiosk system

Figure 3 illustrates the simplest view of the layout between an individual client browser and the legacy Reservation DBMS, which contains reservation, customer, and vehicle information. Figure 4 shows more internal detail as well as some inter-process communication.

Figure 4: High-level physical architecture diagram

The diagram in Figure 4 reflects the high-level structure of our application based on the JavaServerPage (JSP) Model 2 architecture. The customer interacts with a browser on a client computer. The browser communicates via HTTP (Hypertext Transmission Protocol) to the Web server, invoking servlets on the Web server. I have shown only the Reserve a Vehicle servlet specifically. The servlet acts in the role of a controller, coordinating the steps needed to carry out the work of the Reserve a Vehicle use case. The servlet creates and interacts with the business objects, such as Reservation, Customer, Protection Product, etc., that the JSPs will need. The business objects (to be implemented as JavaBeans) obtain their content from the legacy relational database management system (DBMS) via a Java Database Connectivity (JDBC) protocol and interface. The servlet forwards the service request to the appropriate JSP.

Use Case Design step 1: Create use case realization for each use case In Part 1 of this series, I indicated that a use case realization is really a collection of several UML diagrams and other textual artifacts, such as RUP's Use Case Realization Specification document, which together validate that we have the classes, responsibilities, and


8/13/2004

object interactions necessary to provide the behavior in our use case process.

In analysis, our use case realization contains only analysis classes and objects, which may populate various UML diagrams such as class diagrams and interaction diagrams. In design, our realization will contain design-level information explaining how the steps of a use case will be carried out by collaborating design objects. Class diagrams, interaction diagrams, and description, of derived requirements will populate our design-level realizations. If you are using a modeling tool such as Rational®Rose or Rational's XDE product, creating a use case realization may simply involve creating a use case model element in the tool, and then creating UML collaborations and interaction diagrams under that. If you are not using a modeling tool, your realization may be a desk drawer containing hand-drawn models or photographs of whiteboard models. Identifying and naming a use case realization provides traceability from the interactions diagrams and textual commentary back to the use case. The remainder of this article is basically a discussion of the contents of the design-level use case realizations.

Use Case Design step 2: Describe interactions between design objects This is a complex step, so be prepared to refer to several different figures throughout this article, and to observe the iterative process at work. First, note that as part of our Use Case Analysis activity in Part 1 of this series, we produced the analysis class diagram shown in Figure 5 with attributes for our vehicle rental system:

Figure 5: Analysis-level class diagram

This is an adequate, initial description of our analysis classes, and the relationships between various classes, but it is not yet in an implementable form. For example, how do we express in design that a given Reservation must have access to zero or more ProtectionProduct objects? What does it mean that the relationship between VehicleInventory and Vehicle is aggregation, but the relationship between RentalLocation and VehicleInventory is composition? What data types do we specify for Vehicle's status and specialEquipment attributes, or Reservation's dates, times, or estimatedCost attributes? Where do we discover operations to put into our classes? And, how do we represent an interface to the Reservation RDBMS that lets us request, or store changes to, a Reservation or Vehicle status?

Before addressing those questions, note that we can also review the analysis-level Sequence Diagram (SQD) we described in Part 1 for the Check in a Passenger use case, as shown in Figure 6.


8/13/2004

Figure 6: Analysis-level sequence diagram (SQD) Click to enlarge, then find the four-headed arrow in the lower right of the screen to expand the graphic to full size.

In Use Case Analysis, our goal was to prove feasibility—that our business objects had the right responsibilities to carry out the steps of the Reserve a Vehicle use case. We simplified our task by assuming all objects are eternal, and we ignored creation and destruction of objects. But now, in design, we have to explicitly demonstrate creation (and destruction, depending on the programming language used) of our objects. We must also ask "who is in charge" of accessing and "hooking up" the objects needed to service a customer request.

Let's move into how we transform this sequence diagram into a design perspective on the interactions among our objects. We start again with the Reserve a Vehicle use case, specifically the "happy path" scenario where nothing goes wrong.

How will this Reserve a Vehicle use case really start? The sequence diagram snippet in Figure 7 captures a plausible interaction among objects. (For simplicity, I am omitting in this sequence diagram the handling of the customer profile that is in the analysis SQD.)


8/13/2004

Figure 7: Design SQD of Reserve a Vehicle Click to enlarge, then find the four-headed arrow in the lower right of the screen to expand the graphic to full size.

When the customer visits the Deals on Wheels Website in Step 1, the home page is visible on the customer's browser, presenting input fields that the customer can fill-in to designate the location, dates, and category of the vehicle she wishes to reserve. When the customer indicates in Step 2 she wants the system to search for vehicles matching these critieria, the JSP posts the form data to the Reservation servlet (RsvnServlet) in Step 3. The reservation servlet has to verify that the selected rental location is actually open on the dates and times selected by the customer, so in Step 4 the servlet creates an empty RentalLocation object, and in Step 5 directs this object to populate itself with the data for the location designated by the selected location identifier. The RentalLocation has to obtain this existing information from our on-line Reservation RDBMS, and in Step 6 I have added a new design class, DataAccess, that implements the Façade pattern. This pattern lets us hide a complicated interface by substituting a class or component with a simplified interface. The RentalLocation object invokes a retrieveLocation(locationID) operation on the DataAccess object, which returns a structured data row. In Step 7 the RentalLocation calls a private operation on itself, fill(dbRecord), which parses the data row and assigns the fields to its appropriate instance variables. At the conclusion of Step 7, this instance of RentalLocation is fully populated for a specific rental location. In Step 8 the servlet asks this specific rental location, "Are you open at these dates and times?" through a validate(dates) operation. In the "happy path" scenario we are discussing, the RentalLocation replies "yes" (OK).


8/13/2004

Since the location is open at the selected dates and times, the servlet will obtain the list of vehicles matching the customer's request and have them displayed to the customer. In Step 9 the RsvnServlet creates an empty instance of VehicleInventory to hold the available vehicles matching the customer's criteria. In Step 10, the servlet invokes a populate( ) operation on the VehicleInventory, passing the selection critieria. Since the location inventory and knowledge of vehicle availability is in an on-line data store, in Step 11 the VehicleInventory invokes a method, retrieveVehicles(collection), on the DataAccess object. The intelligence in this method knows to create an empty Vehicle object (Step 12), obtain the vehicle information from the corporate RDBMS (not shown), and then in Step 13 invoke an operation, fill(dbRecord), on the just-created Vehicle object. In fill(dbRecord) the Vehicle object parses the data from the dbRecord (a data row), and fills in its own instance variables. In Step 14 the DataAccess object adds the now-filled in Vehicle object to the VehicleInventory. Steps 12-14 are repeated until all matching vehicles in the result set have been deserialized into objects. Following Step 14, the VehicleInventory and the servlet are notified that their requests have been completed.

The system now has to display the available matching vehicles to the customer. In Steps 15 and 16, the reservation servlet stores the VehicleInventory object in the Session object to be retrieved by the next JSP, and invokes the RequestDispatcher.forward( ) operation to display the ShowInventory JSP in Step 17.

In just this small snippet of our design SQD, we have almost as many messages as the entire analysis sequence diagram. But in our evolution of the design SQD, we have strictly demonstrated how our analysis and technical objects will interact with each other to carry out the work of our Reserve a Vehicle use case. We have determined that VehicleInventory really is just a collection class to hold multiple Vehicle objects, and if we continued the SQD, we would add another collection class to hold all the ProtectionProduct objects that could be assigned to a Reservation.

At this point, we can reconstruct our class diagram (recall Figure 5) based on the new technology classes and new operations discovered while developing this part of the Reserve a Vehicle SQD, as shown in Figure 8.

Figure 8: First class diagram updated with operations Click to enlarge, then find the four-headed arrow in the lower right of the screen to expand the graphic to full size.

Let's step back and look at the larger picture that is developing. As depicted in Figure 7, we adopted an approach that is known as the Java Server Page Model 2 Architecture. This approach is a variation of the separation of concerns principle embodied in the Model-View-Controller architecture. On the left side of Figure 7, there are two JSPs. These are the presentation pages, or views, that are displayed in a browser. In the middle of the SQD is the reservation servlet, which is a controller that knows the steps that have to be carried out to allow a customer to reserve a vehicle. On the right are the entity objects, known as model objects, that have the business knowledge and business relationships we need.

In our development of the SQD snippet shown in Figure 7, we discovered new operations on three of our model classes: Vehicle, VehicleInventory, and RentalLocation. So far, these are operations only to obtain data from on-line storage. As we develop later parts of this SQD, or SQDs for other use cases, we would discover more operations in all of our classes: operations for retrieval, storage, computation, etc.


8/13/2004

So let's not stop here. We've only looked at the behavior of getting existing data from online storage into our objects. The next part of the rental process starts when the customer selects a specific vehicle to reserve. In Figure 9, I show the SQD for this part of the process.

Figure 9: Design SQD for second part of Reserve a Vehicle Click to enlarge, then find the four-headed arrow in the lower right of the screen to expand the graphic to full size.

In Step 1 the human customer indicates that she wishes to reserve a specific vehicle. This request is posted to the servlet, which asks the VehicleInventory to mark the vehicle, denoted by a vehicleID, as unavailable temporarily. This marking will prevent other rental customers from trying to reserve the same vehicle. The VehicleInventory forwards this request to the Vehicle object for the vehicle selected by the customer. The VehicleInventory does this with the mark(expiry) operation, in which the expiry parameter is a timeout period in case the customer leaves the page without completing the reservation.

Note that I have included a UML Note ("Update DB status as well?") here. As a practitioner of agile development, I highly recommend that you use a modeling tool only for models that have long-term value. But if you have a long-term model with questions still awaiting resolution, feel free to include the questions right into the model. You may not do this often, but the important point is to get the issue into the open for all to see, and not lose it.

After marking the vehicle, the servlet needs to prepare for displaying the Renter Information page, in which the customer verifies our information on her, and the contents of her Customer Profile. Recall that I omitted the SQD section in Figure 7 that described searching for the CustomerProfile, but now we need the profile and the Customer object, too. In Step 5 the servlet obtains (presumably from the HttpSession object, which is not shown) the customerID that would have been entered on the Deals on Wheels home page. With this ID,


8/13/2004

the servlet creates (in Step 6) a Customer object, directing it to populate( ) its instance variables using the customer ID. In Steps 8 and 9 the Customer object follows the now-familiar pattern of requesting its content as a dbRecord from the DataAccess object. With the Customer object now instantiated and populated for our specific human customer, in Step 10 the servlet notifies the Customer object of its corresponding CustomerProfile. In Step 11 the Customer notifies the profile that it is the profile's parent. Now we have a bi-directional assocation between these two objects.

With this processing complete, the servlet is ready to display the RenterInformation page, which it does with a RequestDispatcher.forward( ) operation. In Step 13, the RenterInfo JSP page is active. In Steps 14-16, the JSP will obtain generic information from the Customer and CustomerProfile objects and corresponding data will display for confirmation. At Step 17 the RenterInfo page is displayed to the customer's browser for confirmation and completion. In Step 18 the customer indicates she has affirmed, changed, or entered data onto the RenterInfo page, and she is ready to continue the reservation. In Step 19, I again show generically that the JSP will interact with the Customer and CustomerProfile objects (remember, these are JavaBeans) to update them from the screen data. In Step 20, the RenterInfo page posts to the servlet, indicating the renter information is complete. After this we would model how the system presents the summary information on the reservation, including offers to purchase protection products, etc.

We have mapped out quite a bit of interaction in this SQD, and we have discovered more operations on our classes. By promoting our SQD messages to operations on the receiving classes, we have a new version of the class diagram, as shown in Figure 10.

Figure 10: Second class diagram updated with operations Click to enlarge, then find the four-headed arrow in the lower right of the screen to expand the graphic to full size.

Now we see that we have added new operations to Vehicle, Customer, and CustomerProfile. This is the iterative, use case driven approach at work. Starting with our use case and list of classes, we choose objects from those classes to interact, and we construct a SQD to show that interaction. The messages in analysis are assigned to the classes based on the responsibilities we have given each class. Then, the messages are promoted to operations on the class of the object receiving the message. These operations are (in an ideal world) guaranteed to be needed because they are generated to meet the goals of the use case that contains many of the functional requirements of our system. Recall that this Use Case Design step is named Describe Interactions Between Design Objects. In our SQDs, we have certainly done that, and captured the static structure enabling that interaction in our class diagrams.

Use Case Design Step 3: Simplify sequence diagrams using subsystems (optional) The next Use Case Design step is to simplify repeated behavior by replacing common groupings of objects with a subsystem. In our example, we have already identified a subsystem, the DataAccess object. This subsystem exposes the retrieve ...( ) operations for


8/13/2004

obtaining object data from the datastore. There is really a lot going on inside the DataAccess subsystem: generation of SQL statements, error handling, result set management, etc. But the gory details are hidden from the client objects requesting data retrieval (and later, data storing).

Often we do not realize we need a subsystem (with simplified interface) until we struggle with a complex set of classes and interfaces that we eventually recognize have a natural affinity for each other (e.g., the classes in a Printing Subsystem: Spooler, QueueManager, PrinterDrivers, etc.) A wonderful principle for grouping classes into subsystems is Robert Martin's Common Closure Principle:

The classes in a package [or subsystem] should be closed together against the same kind of changes. A change that affects a closed package [or subsystem] affects all the classes in that package and no other packages.

In other words, we need to group classes that have a common goal within a clear boundary, so that changes within that package/subsystem are isolated to that package/subsystem. And when we simplify our models by introducing subsystems, we replace large sections of the sequence diagram with a single message to the subsystem. This isolates the former complexity within the subsystem, and we can develop separate models for the internal interactions of the subsystem, which can now be a reusable component in our overall system.

Use Case Design Step 4: Describe persistence-related behavior But, while we have correctly hidden some of the DBMS details behind the DataAccess subsystem, we have actually made an egregious design mistake: Our analysis objects (Vehicle, RentalLocation, ...) have knowledge of the DBMS. Analysis objects should only know business-logic, not storage locations or storage methods, or dbRecord structure layouts. Why is this bad? Consider the effect of a simple change, which is the primary key for an object changes, and recall that our current design has our analysis objects (e.g., RentalLocation) passing a specific identifier (e.g., locationID) to the DataAccess object. If this identifier is used as a primary key, and then changes, we will have to change the logic in our RentalLocation object.

Is there a way to limit or eliminate this consequence on our analysis objects?

Yes. We can accomplish this logical isolation, in addition to the explicit physical isolation described in our Software Architecture Document, by introducing three new objects:

A factory object, which creates our analysis objects for us, fully populated from the DBMS A DB object, which is the DBMS-aware "twin" of an analysis object A persistence interface, a subsystem that specializes in getting information in and out of a datastore.

In the sequence diagram shown in Figure 11, we have introduced these three objects and re-created the interactions where the servlet needs a Customer object restored from on-line storage.


8/13/2004

Figure 11: Design sequence diagram with persistence layer Click to enlarge, then find the four-headed arrow in the lower right of the screen to expand the graphic to full size.

This is a much more encapsulated design, and allows for great reuse. For each analysis object (e.g., Customer) that must be stored or retrieved from the DBMS, we create a "DB-aware" ("database aware") object (e.g., DBCustomer) that understands the data fields and format of its "normal" twin. These DB-aware objects will reside on the Web server, and will be created and used by one of the servlets. The DB-aware object knows how and where to connect, how to build a SQL statement and get it executed, and how to take the result data and store it into the analysis object. Now, changes to the DBMS format or result set data structure are isolated behind the ObjectFactory layer, and the analysis objects are not affected at all.

We must also evaluate our storage structure to make sure we can store, and retrieve, the data required by our objects. Tables 1-7 indicate a sample logical model of the Reservation DBMS layout. Notice that most of the fields in these tables match the attributes in our classes. But some table fields (e.g., Table 3: Vehicle Date of Purchase) are not in our corresponding class because fields like this are not relevant to the Reserve a Vehicle use case.

Table 1: Rental location

Table 2: Reservation

Table 3: Vehicle

LocationID <PK>

Street Address State Sales

Region

ReservationID <PK>

LocationID <FK>

VIN <FK>

CustomerID <FK>

StartDate&Time

ReturnDate&Time

QuotedCost

PaymentMethod


8/13/2004

Table 4: Protection product

Table 5: Customer

Table 6: Customer profile

Table 7: Award program

Also, notice how the primary key (<PK>) and foreign key (<FK>) properties represent the relationships on the Deals on Wheels class diagram. The Reservation table knows about its corresponding Customer through the CustomerID <FK> field. The RentalLocation table can obtain all of its assigned vehicles through the Assigned Location primary key field in the Vehicle table. The SQL statement created by the DB-aware objects will be generated based on the physical data model for the database being accessed. The SQL statement "SELECT * FROM VEHICLE WHERE ASSIGNEDLOCATION = 42355 ORDER BY VEHICLE.VEHICLECATEGORY" will return a result set of all vehicles assigned to that rental location, ordered by vehicle category (economy, compact, etc.).

Now we can appreciate this very important statement: the design classes you discover and use will depend on the design approach you take. In our first approach to addressing persistence, we put DBMS knowledge in the object representing our business abstractions (i.e., analysis classes). But in our second approach we created an additional layer of DB-aware objects that specialized in getting data into and out of the DBMS.

Use Case Design Step 5: Describe design mechanisms In our analysis of our vehicle rental system, we identified a need for the following analysis mechanisms:

Persistence. In a normal reservation session, or change reservation session, the Reservation and Vehicle objects will undergo change of data and/or state. So we need a persistence mechanism to store the new object data. In analysis we need only to specify a meaningful name for the persistence store, but the details of how we achieve this persistence will be addressed later in design.

Security. We do not want to show any reservation other than that of the correct individual, so we specify a mechanism to assure authentication security.

Legacy interface. All vehicle information comes from the Corporate Vehicle System that maintains centralized vehicle information for all rental locations. How we talk with the Corporate Vehicle System depends on the design mechanism choices we can use.

In design, we reevaluate and map these into design mechanisms that are specific solutions to the analysis needs for persistence (Table 8), security (Table 9), and legacy interface (Table 10). For each of these, I have additionally shown the implementation mechanisms that will satisfy the design mechanisms, as shown in Table 8.

VIN <PK> Manufacturer Make Model Assigned Location

<PK> <FK>Vehicle Category

Date Of Purchase

CurrentMileage . . .

ProductType <PK>

Eligibility Codes

Lower Limit

Upper Limit

Benefit Increment

CustomerID <PK> <FK> Name Street

AddressEmail Address

BusinessPhone

Other Phone

Drivers License #

IssuingState

License Exp. Date

Date Of Birth

CustomerID <PK> <FK>

Default Vehicle Category

Smoking Preference

Default DamageWaiver

Default PersonalAccident Insur.

Default Supplemental Liability

AwardID/CustID <PK> <FK>

Current Balance YTD Redemptions


8/13/2004

Table 8:

Conclusion In this two-part series of articles, I have provided numerous examples to illustrate how to move from initial use cases containing many of our system requirements, to analysis of our problem domain, to design of our solution domain in the context of the specific technical dependencies of our production environment. There are many more topics that have to be considered in system development, but I believe these examples satisfy my limited goal of offering a practical tour of the process. Given the level of detail in our design-level sequence diagrams and class diagrams, the reader should be able to see that translating these representations into Java class definitions or JSPs is a straightforward exercise. With our design artifacts, we can move directly into coding a proof-of-concept, or writing the production code for our system. Each use case will follow the process in these articles until, when all use cases have completed this iterative cycle, we will have most of our system coded, and we will have coded all of the significant business functionality that was captured in our use cases.

References Rumbaugh et al., Object-Oriented Modeling and Design. Prentice-Hall, 1991.

Robert C. Martin, Agile Software Development: Principles, Patterns, and Practices. Prentice-Hall, 2003. An award-winning book that covers a broad menu of essential design practices.

Mechanism Persistence

Design Mechanism

RDBMS—Analysis identified the need to access and store object information in a persistent fashion. Since our corporation has several relational database management systems (RDMBSs), our design mechanism will also specify RDBMS. We have staff who understand this technology, and our existing data infrastructure includes tools (e.g., report generators) that require SQL as an interface.

Implementation Mechanism

JDBC—We have partially addressed this in the section above that evolved the Factory class, DB-aware classes, and Persistence Interface class. But exactly how will the Persistence Interface access the target RDBMS? In our scenario, let us assume that the Reservation RDBMS is an Oracle 9i system running on a Unix server, but may be replaced by SQLServer running on Windows 2000. In this scenario, using a JDBC interface to the RDBMS would give us the flexibility we need.

Mechanism Security

Design MechanismAuthentication Security—Analysis identified the need to assure that a customer can see or change only his/her own reservations. We will use the customer's Awards ID number, which is guaranteed to be a unique identifier, as an index into the passenger data store.

Implementation Mechanism Unique Customer Number (Awards ID number) as unique identifier; RDBMS index on Customer table.

Mechanism Legacy Interface

Design Mechanism

This mechanism is not necessary in our simple vehicle rental system example. In our Use Case Analysis, we stipulated that our vehicle information was maintained in the Corporate Vehicle System, which was itself an RDBMS. Our persistence mechanism provides the services we need to extract and insert information to our RDBMS.

If we had to communicate with a CICS application on an IBM MVS mainframe, then we would need to specify the communications protocol we would use (e.g., LU 6.2 APPC).

Implementation Mechanism None.


8/13/2004

Eric Gamma, et al., Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995. Required reading for any developer doing serious development today.

Further Reading Scott Ambler, Agile Modeling. Wiley, 2002. How to do more with less modeling and overhead.

Len Bass, et al., Software Architecture in Practice. Addison-Wesley, 1998. Excellent coverage of what software architecture really is.

Notes 1 All references in this series to RUP incorporate the content of RUP version 2003.06.00. All UML models in this series have been generated using IBM/Rational Extended Developer Environment (XDE) Developer Plus for Java version 2003.06.

About the author Gary K. Evans is the founder of Evanetics, Inc. (www.evanetics.com), a consulting company dedicated to reducing risk on software projects through agile techniques and process. He is the author of over a dozen papers on object technology and tools, and is a frequent speaker at major software conferences. He is an avid soccer player, but loves even more working with small development teams, training them in OOAD and agile RUP, and then working with them side-by-side to deliver the right software faster than they ever thought possible. He can be reached at [email protected].


Comments?





8/13/2004




Scaling down large projects to meet the agile “sweet spot”Philippe Kruchten ([email protected]) University of British Columbia 3 Aug 2004

from The Rational Edge: This article describes techniques for applying agile software development methods to a large project that normally would be considered beyond the scope of agility. The author explains the communications capability that a software architecture team can offer coding teams focused on architectural chunks, even when those teams are disconnected by geography, culture, and specialization.

If the mountain will not go to Mahomet, let Mahomet go to the mountain. (Proverb)

I often hear the phrase "scaling up agile processes" to tackle "large" projects, and I also hear varied opinions about whether it can be done or not. In this paper, I view this problem slightly upside down. If agile processes work in some well-identified conditions, is there a way to transform large projects such that most of the work is done under the optimal conditions of agility, and therefore using all the good practices associated with those conditions? What are the consequences, the cost, and the constraints on the organization and the individuals of doing so?

The "sweet spot" for an agile project What is the ideal context for an agile project? Under which conditions are agile methods most likely to be successful?

Small group. When no more than ten to fifteen people are on the project team, it is possible to maintain high-bandwidth, one-to-one communication. But this starts to become difficult when the team size exceeds twenty people. With smaller teams, no specialization of roles (analyst, designer, coder, tester), with handover of artifacts, is necessary. Co-location. Again, this allows high-bandwidth communication and synchronous communication between people. Customer availability. When the project team has direct access to the customer, it becomes easier for course corrections to be made periodically, thus rendering a better product at the end of the project. Better yet, your team has direct and continuous access to a good customer representative empowered to make decisions about the needs to be satisfied by the software under development. Business application. Agile development works better for business software projects in which interactive types of applications are being built. It works less well for other kinds of projects: for example, an embedded real-time system. New development. New software projects, often called "green field" projects, are better suited to agile methods than maintenance projects. RAD programming environment. Agility requires a rapid application development environment that allows fast turnaround from construction of code to testing and collective code ownership; ideally, development teams do daily builds and monthly delivery. Short lifecycle. Projects should not exceed a few months and ideally have a rapid internal loop (one day).

Contents:Introducing the large, but agile, projectProblemsSummaryAcknowledgementsReferencesNotes




Page 1 of 11Scaling down large projects to meet the agile “sweet spot”

8/12/2004


Common culture. Vocabulary and an implicit process are shared, and individual developer practices rapidly align with each other.

There is no doubt that in this sweet spot for agile development projects, there are few elements that require consigning information to some intermediary artifact, such as change management tools or sophisticated modeling tools. Communication is direct, including communication with the most interested stakeholder, the customer. Note that I am not saying that agile development would not work outside of the sweet spot, but simply that the conditions listed above describe the perfect case.

The "bitter spot" for an agile project Conversely, we can describe the "bitter spot" for an agile project. Here are the conditions that cast agile projects into jeopardy:

Large group. When one-to-one communication is possible, but it is hard to identify the right person to communicate with, agile methods tend to break down, especially when some team members become swamped by communication requests. Communication starts to be asynchronous, and teams begin to rely on documents of various forms, including specific channels and storage mechanisms, for managing requests and project information. Distributed team(s). This again makes impromptu, rapid communication more difficult; various electronic media can be used (phone, Internet conferencing tools, instant messaging systems), but often organizations will opt to produce documents for asynchronous communication and rely on travel for resolving critical issues. No empowered customer representative. Under this condition, requirements—and the clarification, negotiation, and feedback about requirements—tend to evolve over long and inefficient routes, via documents, occasional meetings, and decisions that are dragged up and down multiple levels of management and of organization boundaries. Long cycle. When first delivery occurs two years or more after project inception, agile methods don't hold up. Inevitable personnel turnover leads to "loss of memory" within the project; it gets difficult to find meaningful intermediate milestones or short-term objectives for assessing progress or simply for motivating the team. Inefficient programming environment. Lack of automated tools usually means long turnaround and no collective code ownership; handover of artifacts proceeds slowly from designer to coder, from coder to tester, etc. Different development cultures. This is often found when multiple organizations are involved, which often includes subcontractors and outsource partners. To remedy this, the process has to be made much more explicit, but that means the process itself can become the dominant driver, the monster to satisfy: "We have to take this step because the process demands it."

We should note that in the extreme, large projects have traditionally relied solely on extensive documentation for all stages of development, with project managers insisting on written documentation rather than on people to span the gaps that occur in a distributed environment, multiple organizations, long project cycles, and the disruptions of personnel turnover.

Large projects "I never want to work in a large project like that again," someone told me recently. Well, let's face it: large projects are not going to go away because of the emergence of agile processes. "Size is some kind of grit, if not the grit." wrote H. Erdogmus,1 and this grit is causing much friction, slowing projects to a crawling speed. I have spent most of my professional life involved in large projects: telecommunications, command and control, programming tool development, and so on. For the sake of this discussion, let me characterize a very large project to avoid questions about marginal conditions, such as "But how about twenty-five people instead of twenty?" or "What if teams are distributed over just two sites?"

My large project has these characteristics:

200 people Distributed across multiple time zones Workers report to three different companies The lifecycle is 2.5 years to first delivery and 6 months for subsequent updates

There are some conditions that can be achieved equally in small and large projects:

Good programming environment


8/12/2004

Skilled people Access to the customer

Let's assume all three of these optimal characteristics to make it easy to compare my large project to a small one.

However, in large projects, certain testing capabilities may not be readily available to all developers—e.g., there are only so many telephone switches or submarine simulators you can make available to any given programmer in the lab. This deficiency is usually not the case with small projects.

Introducing the large, but agile, project Now that we've considered the optimal and suboptimal conditions for an agile project, and we've described the basic scope of a large project, the question becomes: Can this large project be managed via agile methods? My answer is, yes. It is possible (though not easy) to organize a large software development project so that a good proportion of the software practitioners will work in an environment close to the agile "sweet spot." The rest of this discussion will describe how software development teams can go about this.

Setting up an ideal large project Of the 200 people involved in this project, suppose that I organize ten teams of fifteen people each, and use the remaining fifty as the superstructure, the glue that makes it work. In each of the fifteen small teams, members are co-located, they have their own development playground, and they share a common culture. The key problems are: Who is the customer? Is this customer empowered? How do I communicate with other people outside the boundary of my team? A large system is not just a loosely coupled set of small systems. How do we start on day one?

As the project manager for this monster project, I will need to set up the project from various perspectives:

Organization. I need to define project structure, the various teams, their roles, responsibilities, and objective. Schedule and rhythm. I need to define the milestones and their associated objective alongside a timeline. Communication. I need to decide how the various teams will communicate.

Most of the answers lie in the software architecture, which is produced by the architecture team, and the function of the Elaboration phase (as it is known in IBM Rational Unified Process®2 and MBASE3 process frameworks).

Gradual organization: Inception and Elaboration The development work can start under one team of seven to twelve developers (the architects), working in an agile setting that is co-located, has a common culture, and so on, with direct access to the customer, and using an iterative work flow. The objective of the architecture team is slightly different from a classic agile project, however, since the immediate goal is not to deliver a functioning system, but to identify enough of the significant user requirements (user stories, use cases, nonfunctional requirements) to set up an architectural prototype, that will facilitate:

Organizing use cases (and other aspects of requirements) Identifying underlying mechanisms needed to support the implementation of these use cases, as well as the key nonfunctional requirements (scalability, performance, availability, etc.) Developing an architectural prototype to validate the design choices made, the technologies selected Organizing the code in subsystems And, as necessary, setting up guidelines and tools for further development: e.g., design and code standards, code management, change request and trouble report, and so on

This initial architecture team proceeds iteratively, and the team gradually grows in size. When this team reaches fifteen members or more, they split into:

The architecture team, mostly customer involved, and making key architectural choices


8/12/2004

The prototyping team, gradually building up and testing the architectural prototype, refactoring it, and pushing back up any issues detected, thus closing the loop

The architecture team (seven to twelve people) is the customer for two or three prototyping teams (consisting of thirty-five or more people).

Staffing the project: Construction and Transition Towards the end of the elaboration phase, when the structure of the code starts to take shape and stabilize, additional teams are created, each one "owning" a chunk of the architecture. The implementation view (according to the "4+1 view model" 4) serves as the basis for the partition of the code and the allocation of ownership to a set of teams. Each chunk of code now belongs to one and only one team.

Ideally, each of the newly created teams is seeded with one or two members of the original architecture and prototyping teams to bring some of the project knowledge and emerging culture. Typically, each of these seed members is the new team's "technical lead." This person will play the role of the local architect and serve as the communication conduit to the core architecture team that remains, and, in some cases, with the customer.

There are two types of teams involved in the development of code:

Feature teams. Develop code directly related to user functionality. Infrastructure teams. Develop common elements and services used across the feature teams, or support nonfunctional requirements (e.g., middleware, services, domain-specific framework, and reusable assets).

For each team, we will strive to create the ideal conditions of agility.

The customer for the feature teams is the real customer(s). But the real customer(s) cannot spread themselves too thinly across multiple teams and locations. The customer for the infrastructure team is... all the other teams!, which may have multiple and conflicting needs.

This is where select members of the core architectural team start playing the role of customers. They mediate between the real customer and the various teams, via the tech leads, and identify common needs across the team to be realized by the infrastructure teams. They become the customer representative for the infrastructure teams.

Note that this does not prevent direct access by the feature teams to the real customer, but that contact should occur only when specialty, or subdomain, issues arise. For example, an air traffic control system has several subdomains: flight management, radar data, clearance, aeronautical information, conflict detection and resolution, and so on. As long as the customer involved is competent and empowered to make decisions about these subdomains, the project is in good shape.

The architecture and infrastructure teams will continue to fine-tune, refine (refactor), and expand the architecture of the system throughout the Construction phase, and ideally this will be conducted with minimal disturbance to the other teams.

A note on system integration As we tackle this large project using agile methods, we need to be realistic in our approach to integrating the complete system. We not only need to assemble the pieces built by the various teams, but at the same time we need to provide each team the right "environment" to be able to perform their own builds and tests. Each team cannot do this on its own, as this may require resources beyond its capacity. This is the role of a system integration team, which is at the receiving end of all the software subsystems and commands access to the testing facility for systems-level tests. This system integration team is derived from the initial prototyping team. It may also be responsible for certain types of test such as performance, load, availability, usability, and so on, or when special equipment is needed.

What about those agile "sweet spot" conditions? At this point in our project, we have completed the Inception and Elaboration phases, we have assembled the Feature and Infrastructure teams, and are about to begin the Construction/Transition phases of the project. Figure 1 shows the evolution of team structure over the course of Inception, Elaboration, and Construction/Transition phases. The "blue" teams are the ones that could operate close to the agile "sweet spot." The "green" teams are the ones that provide the necessary communication and glue to make it happen.

During Inception and Elaboration a system architecture is prototyped, based on customer needs, and this serves as the blueprint for a team structure; it shows where small, cross-functional teams are set up and a how superstructure is put around them to allow them to efficiently operate.


8/12/2004

Figure 1: Evolution of team structure over time

Problems I said this was not going to be easy. Several dangers lie ahead in this set up. I'll cover a few of them.

Dependencies between teams We have created a problem that does not typically occur in our ideal (sweet spot) agile projects. As I have indicated, any given team, such as a feature team, is the customer of another team, and is dependent on that team delivering some functionality. To reduce the impact of this dependency on its ability to proceed building and testing its code, the team will not only need to build tests associated with its "user story," but also to build a stub, proxy, or mock-up that will be passed along in its role as customer to the infrastructure team.

Stove pipes Each feature team becomes a single project in itself; large portions of functionality end up being replicated across the team. This is exacerbated by the geographical distribution of the team members in this large project and by the different organizations at work, which do not share a common culture.

The remedy for this? The architecture team must provide some communication glue, participate regularly in design reviews, and bring in people from other teams. It can identify potential for common code or functions to be developed by infrastructure teams, and it can indicate which elements should be reused from one team to another. It can also suggest rotation of staff when possible across teams.


8/12/2004

Figure 2: Reducing the "stovepipe" effect. The improvement from side A to side B represents identifying and developing more common code across various teams and having an organization to match.

Imbalance Because the workload is uneven between teams, overloaded teams can potentially harm the whole project by delays or poor quality. Sometimes the imbalance is more in the mix of expertise among team members. The solutions to this include: change of personnel allocation (ratio) of subsystems to team; rotation of staff to spread knowledge and expertise; and temporary reallocation of staff from underloaded teams to overloaded teams.

Communication breakdown When communication between teams becomes inefficient, delays, frustration, and work blockage can result. The architects must step in and establish and facilitate direct communications between the involved parties.

Too much staff too early In general, it is not a good idea to have too many people involved in the Inception-Elaboration phases. But if your only alternative is idling a portion of your staff during these phases, it may be possible to start using some of the staff in small agile projects to develop parts of the system that are obviously necessary from day one.

Loss of a common goal, lack of focus Again, although small teams operate with their own objectives, they must never lose sight that the real success is in delivering the complete system, not just their chunk. They are not in competition with the other teams. And they must be open to communication, suggest improvements, and always be ready to rebalance the project workload.

Dual rhythms (two nested beats) When the lifecycle is long, measured in years, for example, it may be necessary to introduce two levels in the project timing, in the project "beat." Figure 3 shows development teams working with a monthly iteration cycle (an internal release), with a daily "scrum."5 But this beat is inside a slower system-level beat, with a system major increment of six months, and a system-level scrum of a week or two.


8/12/2004

Figure 3: Two beats (scrum of scrum)

Organizing documentation Some tell me that as soon as I use the unfortunate word—documentation—everything collapses: the project is not agile anymore. But this shouldn't be the case. Project managers should only introduce permanently maintained artifacts (documents, models etc.) where they are needed: that is, when they resolve a key problem and allow the complete project to be globally more efficient.6

Architecture. The architects will need at some point to document the architecture, if only to avoid spreading themselves too thin when the overall team grows and having to explain the structure, the rationales, the metaphors again and again to newcomers. A well-documented architecture provides a single visible reference for all team members. Other non-architectural aspects of the design may never be documented explicitly in a separate document, but only in comments attached to the code, from which tools like JavaDoc can extract it when necessary.

Requirements. Requirements are kept very fluid at first, but as the various feature teams progress in the implementation, use cases are collected, together with any nonfunctional requirements and organized to facilitate tracing and testing, and to serve as the starting point for user documentation and training.

Risks, issues, and backlogs Risk issues and product backlogs should be treated at the level of each team; only the risk issues and elements of backlog that cannot be rapidly resolved at the level of a single team should be escalated to the project level. Technical issues are typically handled by the architecture team, and organization issues (staffing, resources, etc.) by the project management team. Often issues arise where the "owning" team is not clear or the problem straddles two teams. Architects should arbitrate to resolve these problems. A common project-wide tool to manage local and global issues is a useful part of the development environment (for example, ClearQuest).

Process Description of practices, standards, guidelines, and templates is best made explicit in order to increase uniformity of practice across the whole organization. A clear description of the process should support the teams in their efforts to succeed and should not be used in a


8/12/2004

coercive, administrative fashion.

Concrete examples Do I have any evidence that large projects can be managed, at least in part, by agile techniques? Or did I pull this out of my hat? Not quite. Most aspects of this strategy have been applied to large, iterative development projects I was personally involved with. Here are a few examples:

1. Ship System 2000 (SS2000), a family of ship control systems developed by Philips/Bofors/NobelTech/CelsiusTech in Sweden7

2. The Canadian Automated Air Traffic System (CAATS) developed by Hughes Electronics/Raytheon in Canada8

Both these projects had the large-project characteristics I indicated at the beginning of the paper. Ideas about matching the team structure to architecture—i.e., that the architecture team, integration team, and feature and infrastructure teams should match the static view of the software architecture, as shown in Figure 4—grew out of the SS2000 project. These ideas were applied and refined in the CAATS project, with the introduction of use cases. Additionally, we were fortunately able to convince the customer to give us access to the entire team of end users on site. The architecture team took over the communication role, and it was able to resolve issues as they arose across the various teams.9

Figure 4: On SS2000 and CAATS, the team structure exactly matched the implementation view of the architecture (i.e., the code view). For details, see P. Kruchten and C. J. Thompson, "An Object-Oriented, Distributed Architecture for Large Scale Systems," in Proc. of Tri-Ada'94, Baltimore, November 1994: ACM, 1994.

Another large project was of a different kind:

3. The Rational Suite, developed by Rational Software Corporation, with 700 developers and a delivery beat of six months

The product group that produced the Rational Suite had a dozen teams on eight sites around the globe, with a central product definition


8/12/2004

(i.e., the customer representative) in Lexington, Massachusetts, a distributed architecture team, and a central integration team. The feature teams were each co-located on the various sites. Each of these teams was more or less agile depending on its own origin, history, and culture. Each team has its own internal beat, matched to align with the beat imposed by the central product teams. My team in Vancouver, British Columbia, was very agile, accommodating lots of tactical changes, with a daily build and a more major release every other week or so.

A few years ago, my colleagues at Rational Software reported many other iterative projects where these ideas have been implemented, with some variants, and with success.

Summary I have described the gradual organization of a large project as a team of teams, where an objective is to set the ideal conditions of agility for a subset of the teams—the ones producing the quasi totality of the code—and with a few ancillary teams whose main objective is to ensure efficient communications between teams and to implement the external interface that each team would expect to have in a smaller project.

For very large projects this is done through:

Communication inside and outside the team, using the architecture team, the management team, or the integration team as nodes of communication. Scrums and other facilitation and resolution practices at the team level and inter-team level. Builds and releases at the team level and inter-team level. Issues and risks managed at the team level and globally. It is preferable that all teams operate according to the same overall rhythm.

Several ingredients are key to doing this successfully:

The ancillary teams (architecture and integration) must remain at the service of the other teams and not the reverse. Their role is to ensure that the development teams are as efficient as possible, and therefore the ancillary teams must address the needs of the development teams as rapidly as possible through external communication or resources. For example, through communication with the customer or with other teams. The team involved in code development must keep in mind the ultimate global objective, the final system, which has priority over its own team objectives, performance, schedule, and so on. Teams involved in infrastructure must be very responsive to the needs of their customers, who are the other teams. Documentation and an increase in formality (sometimes called the "degree of ceremony") must be introduced gradually and only when and where it allows the organization to be more efficient. Architecture is the key to setting up the team structure in a way that minimizes the need for cross-team communications and dependencies.

The project as a whole needs ready access to a competent and empowered customer representative, who can make decisions about the system. The project needs powerful tools to manage quick turnaround of code, tests, and test suites, and to support change management. And, as in any project, skilled and motivated people are the most important ingredient.

Acknowledgements This paper benefited greatly from discussions with the participants to the Canadian Agile Workshop in Banff, February 2003, the USC affiliates research review in March 2003 and March 2004, and internal reviews by Gary Pollice, Walker Royce, and Joe Marasco from Rational Software Corporation, now IBM Rational, as well as Hakan Erdogmus from NRC.

References H. Erdogmus, "Let's Scale Agile Up." Agile Times, vol. 2, pp. 6-7, 2003.

P. B. Kruchten, The Rational Unified Process: An Introduction, 2nd ed. Addison-Wesley, 2000.


8/12/2004

A. Egyed, B. W. Boehm, D. Port and, M. Abi-Antoun, "Guidelines for the Life Cycle Objectives (LCO) and the Life Cycle Architecture (LCA) Deliverables for Model-Based Architecting and Software Engineering (MBASE)." USC, Los Angeles, USC Technical Report USC-CSE-98-519, 1999.

P. Kruchten, "The 4+1 View Model of Architecture." IEEE Software, vol. 6, pp. 45-50, 1995.

M. Beedle and K. Schwaber, Agile Software Development with SCRUM. Prentice-Hall, 2002.

L. Brownsword and P. Clements, "A Case Study in Successful Product Line Development." Software Engineering Institute, Technical report CMU/SEI-96-TR-035, 1996.

K. Toth, P. Kruchten, and T. Paine, "Modernizing Air Traffic Control Through Modern Software Methods." 38th Annual Air Traffic Control Association Conference. Nashville, Tennessee: ATCA, 1993.

P. Kruchten, "Iterative Software Development for Large Ada Programs." Proc. Ada-Europe conference, Montreux, Switzerland, vol. LNCS 1088, A. Strohmeier, Ed.: Springer-Verlag, 1996, pp. 101-110.

P. Kruchten and C. J. Thompson, "An Object-Oriented, Distributed Architecture for Large Scale Systems." Proc. of Tri-Ada'94, Baltimore, November 1994: ACM, 1994.

P. Kruchten, "The Software Architect, and the Software Architecture Team." Software Architecture, P. Donohue, Ed. Kluwer Academic Publishers, 1999, pp. 565-583.

Notes 1 H. Erdogmus, "Let's Scale Agile Up." Agile Times, vol. 2, pp. 6-7, 2003.

2 P. B. Kruchten, The Rational Unified Process: An Introduction, 3rd ed. Addison-Wesley, 2003.

3 A. Egyed, B. W. Boehm, D. Port and M.Abi-Antoun, "Guidelines for the Life Cycle Objectives (LCO) and the Life Cycle Architecture (LCA) Deliverables for Model-Based Architecting and Software Engineering (MBASE)." USC, Los Angeles, USC Technical Report USC-CSE-98-519, 1999.

4 See P. Kruchten, "The 4+1 View Model of Architecture." IEEE Software, vol. 6, pp. 45-50, 1995.

5 See M. Beedle and K. Schwaber, Agile Software Development with SCRUM. Prentice-Hall, 2002.

6 Let's ignore for now the documents that must be produced for contractual reasons, or for compliance to standards and certification organisms—FAA, FDA, etc.—since these forms of required documents don't necessarily benefit project organization. In any case, they constitute another topic.

7 See L. Brownsword and P. Clements, "A Case Study in Successful Product Line Development." Software Engineering Institute, Technical report CMU/SEI-96-TR-035, 1996.

8 See K. Toth, P. Kruchten, and T. Paine, "Modernizing Air Traffic Control through Modern Software Methods." 38th Annual Air Traffic Control Association Conference. Nashville, Tennessee: ATCA, 1993. Also P. Kruchten, "Iterative Software Development for Large Ada Programs," in Proc. Ada-Europe conference, Montreux, Switzerland, vol. LNCS 1088, A. Strohmeier, Ed.: Springer-Verlag, 1996, pp. 101-110. Also P. Kruchten and C. J. Thompson, "An Object-Oriented, Distributed Architecture for Large Scale Systems." Proc. of Tri-Ada'94, Baltimore, November 1994: ACM, 1994.

9 I describe the role of the architects and the architecture team in P. Kruchten, "The Software Architect, and the Software Architecture Team," in Software Architecture, P. Donohue, Ed. Kluwer Academic Publishers, 1999, pp. 565-583.

About the author Philippe Kruchten is former director and general manager of the IBM Rational Software Process Business Unit, in charge of the Rational Unified Process (RUP). He worked with Rational for thirteen years, in various functions and places: France, Sweden, the US, and Vancouver, Canada.


8/12/2004

Philippe's main interests right now, besides software architecture and design, are software engineering and the development process. He is campaigning, in Canada, for the concept of state-licensed professional software engineers.


Comments?





8/12/2004




Strategic components for identifying and justifying a programMichael F. Hanford Chief Methodologist, SUMMIT Ascendant Methodologies, IBM 3 Aug 2004

from the Rational Edge: Mike Hanford continues the program management discussion he initiated in the May 2004 issue, looking at the decisions involved in creating a program after you apply a strategic process to identify business needs. He discusses steps and deliverables to justify proceeding with the program, and how to plan make the transition to mobilizing the effort.

Typically, marketplace needs and a cohesive enterprise business strategy are the drivers for organizations to tackle large, complex efforts that combine delivery of software elements with changes to business models and organizational structure and capabilities. Yet at the start of such efforts, many program managers/directors discover that goals, boundaries, costs, and other key data about their new programs are ill-defined, poorly quantified, and either partially documented or not documented at all. Often these missing or incomplete items delay the start of work and result in a loss of critical mass and momentum. In this article, we will identify and review the strategic components that organizations must develop and deliver to provide the framework and knowledge required to mobilize a program. This discussion will be based on a portion of the IBM® Rational SUMMIT Ascendant Program Management Method, shown in Figure 1.

Figure 1: Extract from the IBM Rational SUMMIT Ascendant Program Management Method

Creating and validating a strategic vision Large enterprises continuously analyze and validate the company’s strategic direction, role, and marketplace position against the competition. They use a variety of tools and techniques, all aimed at identifying and quantifying potentially profitable opportunities and deflecting competitive threats. Often these analyses indicate a necessary change in the stated direction and help to define contributions that would create more value.

Contents:Creating and validating a strategic vision Strategic planningLifecycle model: Strategic vision and program development and delivery Justifying, planning, and launching the program Getting and maintaining strategic consensus




Page 1 of 9Strategic components for identifying and justifying a program

8/13/2004


The analysis process also delivers a vision statement that outlines potential achievements, identifies possible impact on products and services, and defines a set of benefits or returns to the enterprise that map to goals or desirable results.

At this point, the vision statement must be tested against the enterprise’s current direction as well as competing visions and ideas. Deciding that a vision is good or useful is not alone enough reason to act upon it. In addition, the testing will identify current practices, assets, roles, and products that would be impacted by acting on the vision, the type and degree of impact, and the feasibility of the required changes. It will help managers view all of these factors against the business’ needs to continue supporting current clients and products, and to operate efficiently.

If the vision statement does survive this initial testing and validation, the next step is to quantify the impact that successfully implementing the vision would have upon the enterprise. This assessment has multiple dimensions, including:

Identifying change areas Assessing risks Identifying and quantifying benefits Constructing and validating a financial model Justifying a decision to proceed

The goal is to assess both the vision’s potential value and its implementation cost, providing enough information for executives to decide whether the value justifies the expenditures. If the decision is affirmative, there is justification to move into planning.

As we have seen, in IBM Rational SUMMIT Ascendant, an organization creates a program strategy through a series of iterations, with review and approval gates at the end of each iteration. Through dialogue and oversight at each gate, executive leaders can exercise control to ensure that proposed initiatives fit the overall business direction.

Strategic planning Strategic planning involves defining and setting boundaries for specific initiatives to achieve the vision. These initiatives might involve several projects of several types, each designed to fulfill some portion of the vision. Managers may group similar projects into programs that support the achievement of a single outcome or business result. Then, as a further step, they may group all programs supporting either a major product line or LOB (line of business) within an enterprise into portfolios.

Program managers describe each project within their program in an outline project charter document, including goals, project components, definition of success, and project qualifiers such as assumptions, risks, and limitations.

Figure 2 shows a sample outline project charter as a work-in-progress — some items have yet to be filled in.

Outline project charter Building maintenance project

1. Management summary a. Short description of the proposed project success definition for the proposed project

To keep all buildings within the Hotel Lautec facility in virtually the same condition they were in when they left the builder's hands.

b. Business problem/opportunity the project will address Focus group participants consistently remarked that the hotel’s buildings and common areas looked ”tired.” Management sees an opportunity to transform this image so that the hotel will always look ”fresh” and attract more guests.

2. Goals and boundaries 1. Strategic goal(s) supported

Goal 3 - To create an upscale property that is country club-like in both appearance and amenities. 2. Project boundaries

All maintenance-related activities, which include routine upkeep such as painting, cleaning common-use areas, preventive maintenance, and repair of mechanical equipment

3. Project prerequisites and dependencies (to come) 4. Major expected results/deliverables


8/13/2004

Figure 2: Outline project charter (work-in-progress)

When the outline project charter documents are complete, managers can prepare the program definition document that summarizes content from all the project charter documents and defines the program in outline form (see sidebar).

Both the project charters and the program definition are subject to review and change/validation at the time the program is actually mobilized.

During program implementation, new information, the passage of time, market changes, and shifts in management can all lead to changes in the goals, contents, and needed outcomes for a program and its constituent projects.

So far, we have described the process of identifying, defining, and justifying a major program effort very briefly. In the remaining sections of this article, we will examine several process elements in greater detail.

Lifecycle model: Strategic vision and program development and delivery In IBM Rational SUMMIT Ascendant, evolving a strategic vision, creating plans, and launching the program proceed according to a well-defined lifecycle, as shown in Figure 3.

Figure 3: IBM Rational SUMMIT Ascendant program definition and commitment lifecycle

We will briefly describe each of the model’s five components along with associated results.

Strategic Business Analysis This component relates to analyzing current business and IT directions within the organization, opportunities for new products and changes to existing products and services, threats to existing market share, client relationships, and sales and delivery needs. The major results are a strategic assessment document and one or more presentations, and/or work sessions with the enterprise’s executives. This component is designed to assess the value of both current initiatives and new opportunities; it looks at their potential impact and associated risks.

Strategic Vision Development This component deals with identifying new (or different) directions and initiatives, based on the strategic business analysis results. It is here that the organization seeks to define specific vision components that take advantage of identified opportunities and to identify ways to deflect threats to the enterprise. In addition, it looks at the value and impact of existing initiatives, recommending which ones to continue, enhance, cancel, or defer.

The major result is a strategic vision document with some segments devoted strictly to business topics and others to IT elements required to support the business direction. This document identifies — at a high level — new, changing, and continuing initiatives, along with the business direction or drivers they support.

The software deliverable associated with this project will provide periodic updates to management about the buildings’ condition, current maintenance efforts, near-term projected efforts, and costs in comparison to budget.

3. Project qualifiers (all to come) 1. Management-defined constraints/limitations 2. Assumptions 3. Identified risks 4. Identified impact areas

Program definition

Sets scope and boundaries Defines needed outcomes Assesses impact on IT


8/13/2004

Strategy Justification and Approval It is important to understand the dimensions of potential initiatives defined in the strategic vision. This component deals with developing and quantifying these dimensions to assess the initiatives’ comparative value. In addition to advancing business goals, each initiative should yield some other benefit for the enterprise, such as:

Additional revenue and profit Costs savings or reductions Favorable return-on-investment (ROI)

Assessing and ranking all potential contributions allows executives to make choices about where and how much to invest. In addition, they need ongoing comparisons of costs against returns as efforts go forward so they can decide whether to continue investing in these efforts.

Strategy Program Planning This component represents the beginning of the planning work. It encompasses several deliverables that identify requirements and milestones at a relatively high level. First, the program outline describes the program’s contents and boundaries, effectively serving as a program charter. The program outline lists projects the program will comprise and describes major deliverables, and boundaries among the projects. This is the first document that decomposes the program into a number of discrete projects.

These deliverables form the foundation for later in-depth planning efforts. They also record early thoughts and decisions about the program; collectively, they are a valuable element that traces back to the strategic vision.

Implementation Program Launch This component encompasses a transition from the analytical to the actual. Its activities and results prepare the organization for program mobilization and ramp-up, which entails recruiting additional staff.

Naming an executive sponsor is key to accomplishing the transition effectively. This senior-level officer/executive will be responsible for producing the desired program results. Deliverables for this component include an outline charter document for each project, an initial/draft program budget, and a mobilization plan.

When the launch component work is complete, you will be ready to begin mobilizing staff and other resources to initiate the program work.

Justifying, planning, and launching the program This article focuses on the later stages of developing a strategic vision and program strategy because the process and deliverables required to conduct a thorough business analysis and create a business strategy vary greatly from one enterprise to another. Typically, program leaders have not been assigned yet when these activities take place, so they do not participate in them. That said, let’s look at some practical aspects of justifying, planning, and launching the program.

In examining each of these activities, we will address four practical questions:

Who should do the work? Who else should participate? What work should be done? How complete should the results be?

Justifying the program Justifying a program requires that you understand and quantify the program’s dimensions. Only then can you produce a well-defined set of expectations and value proposition that will make executives feel confident about approving and funding the program. These dimensions include projected costs, existing initiatives that will have to be deferred or delayed, benefits to the enterprise, “fit” with the strategic vision, and impact and risk analyses (see sidebar).

Let’s look at our first question: Who should do the work? A small program enablement team with two to three full-time members should be sufficient to complete the justification and planning and launch work. The team should be a mix of middle-level business managers and IT management/technical staff.

All members of the program enablement team should also have solid planning

Justification dimensions

Projected costs Quantifiable and non-quantifiable benefits


8/13/2004

credentials and a good understanding of business area(s) the program will affect. Their work will consist of planning, developing deliverables, and integrating part-time participants into the process.

One member should be a full-time team leader. Ideally, this person will continue on during program implementation — or at least throughout the mobilization effort — to provide continuity.

Our second and third questions are, respectively: Who else should participate in assembling/validating the justification? What work should be done? Table 1 responds to these questions in terms of the dimensions we identified above.

Table 1: Justification dimensions, staff, and work requirements

A fourth practical question is: How complete should the results be? In the strategic assessment and planning stage, program initiatives address questions and issues at a high-level while competing for attention with other initiatives, both potential and ongoing. How much data do you need, and how complete should it be before you present it to executive decision makers?

The answer to this question is the old, reliable consulting answer: “It depends.” In this case, it depends on the nature of the business enterprise, the types of programs the enterprise typically undertakes, and the degree of risk acceptance in the culture. For example, enterprises that produce competitive products such as airplanes or automobiles are used to undertaking very large initiatives and accepting significant degrees of risk for each venture. In contrast, enterprises that offer services, such as banking or finance, are just coming to understand very large initiatives and are typically risk adverse.

With respect to risk mitigation, the data you submit to executive leaders need only make them comfortable enough to approve continuation of the program definition and planning process. The mobilization effort will provide multiple opportunities and checkpoints for reviewing and refining your justification for the program — and for postponing or canceling it without major financial loss.

Planning the program Planning a program and its constituent projects is an iterative effort that continues throughout the program lifecycle. Following the early, high-level strategic planning, later iterations will flesh out the details and relationships among plan components. At the outset, it is

Deferred initiatives Strategic vision “fit” Business impact assessment Risk analysis and mitigation strategies

Justification dimension Staff/participants Work/results

Projected costs Financial staff member Develop financial model. Access enterprise financial data. Ensure conformance with enterprise financial practices. Validate cost data.

Quantifiable and non-quantifiable benefits

Business area(s) manager Identify potential benefits/returns. Determine scope/scale. Understand benefits’ impact.

Deferred initiatives Business area(s) and IT managers

Enterprise PMO staff

Identify current initiatives. Determine their value. Identify deferral possibilities. Make deferral decisions.

Strategic vision “fit” Vision segment owner Assess program “fit” and suitability. Validate value/potential. contribution to success

Business impact assessment Business area(s) managers

Change management specialist

Identify impact areas and types of impact. Quantify degree/scale of impact. Validate positive downstream potential.

Risk analysis and mitigation strategies Business area(s) and IT managers

Risk management specialist

Identify risks. Assess potential impact. Outline risk mitigation plan.


8/13/2004

important to name an executive sponsor(s) for the program and create various planning deliverables, including a first-cut program budget.

Let’s again begin with the first practical question: Who should do the work?

The program enablement team should continue to drive the effort, drafting the plans, and creating the planning documents. In addition, a budget analyst should help estimate capital and expenses and ROI and should, ideally, become a program office staff member when the program is mobilized.

The budget analyst should be a staff specialist from the finance organization or CFO’s office. He or she should be have in-depth knowledge of the enterprise’s financial systems and be able to apply enterprise standards for financial management, budget construction, and expense records.

Finally, a specialist in size estimation (knowledgeable about models, tools, etc.) should work on projections for both the overall program and its constituent projects.

Again, we will use a table to address the second and third practical questions: Who else should participate in planning, sizing, and financial analysis? What work should be done? Table 2 responds to these questions in terms of planning dimensions.

Table 2: Planning dimensions, staffing, and work requirements

The fourth practical question is: How complete should the results be? The capital and expenses figure should represent the total program

Program planning dimensions

Capital investment and expenses Benefits and ROI Goals and milestones plan Program outline document Program (initial) size estimate

Planning dimension Staff/participants Work/results

Capital investment and expenses

Executive leaders; may include the CEO and Board of Directors

Business area(s) and IT managers

Financial or CFO office staff

Review current/projected budgets, planned expenditures, and anticipated revenue. Define a financial model. Agree on financial assumptions. Iteratively run /review/refine financial scenarios. Define /present /agree on alternatives and initial funding.

Benefits and ROI Executive leaders; may include the CEO and the Board of Directors



Review/refine quantifiable and non-quantifiable benefits and returns. Define an ROI model and agree on assumptions and conditions Iteratively run/review/refine ROI scenarios. Define/present/agree on alternatives and final ROI quantification.

Goals and milestones plan



Agree on a planning model. Plan/hold joint planning work sessions. Agree on goals/goal owners. Agree on major milestones and associated results. Agree on commitments to achieve individual milestones. Assemble/agree on a plan.

Program outline document



Plan/hold joint definition work sessions. Assemble “straw man” program elements: goals, scope, outcomes, limitations, participation. Refine and review/validate program elements. Assemble program outline.

Program (initial) size estimate



Agree on a sizing model and validation model. Agree on sizing assumptions Iteratively run/review/refine sizing scenarios Identify/quantify/resolve sizing issues. Make program funding, goals, and schedule adjustments. Agree on initial sizing.


8/13/2004

cost. Under that dimension, you can segment costs in a way that conforms with enterprise financial practices. But all participants should agree that the estimate is only preliminary. It should cover the program launch and mobilization, but a major financial review is then necessary, near the end of the mobilization effort, to refine the overall cost figures.

Estimates for quantifiable benefits and ROI are also preliminary; the program staff should conduct periodic reviews to check/reset these metrics. Goal and milestone documentation must indicate owners for each major goal as well as others who will perform associated work. The program outline must clearly indicate expected program outcomes and specifically name business areas, products, organizations, and locations that are within — and outside — the program scope.

The initial size estimate should indicate the overall size in terms of both labor hours and other major program cost components — including implementation.

Launching the program The work and results of a launch represent a transition from considering what might be possible to deciding what people will actually do. The launch effort will establish and validate readiness for beginning the mobilization effort, which includes defining governance, acquiring program staff, and establishing a program management office. It requires naming the executive sponsor(s), recording outline information about each project (a charter document), and creating a plan (specifying tasks, needed results, resources, and schedule) for the mobilization effort. It also includes creating the first program budget and validating it against the approved capital and expenses estimate. Finally, during this period, arrangements for outside (or inside) staffing are concluded.

Who will do this work? The program enablement team will continue to drive the effort, drafting plans and creating planning documents.

The program enablement team will provide the new executive sponsor with a complete picture of decisions and activities to date, describing pending decisions or issues needing resolution.

In addition, a liaison person (typically from purchasing) must negotiate consulting and outside staffing contracts. This person might need lead time to review new and existing agreements in order to complete the negotiations before mobilization begins.

Who else should participate in planning, budgeting, and negotiating staffing agreements? What work should be done? Table 3 examines these questions in terms of the launch dimensions.

Table 3: Launch dimensions, staffing, and work requirements

Launch dimensions

Executive sponsor(s) selection Project outlines (charters) Program mobilization plan (Initial) capital and expense budget Consulting and staffing agreements

Launch dimension Staff/participants Work/results

Executive sponsor selection

Executive leadership; may include the CEO and Board of Directors

Identify candidates. Review qualifications/availability. Hold informal discussions. Select the best candidate.

Project outlines (charters)


Leader/team member for strategic vision delivery

Review/refine project list. Iteratively draft/review/refine project outline segments. Validate boundaries/dependencies. Publish/approve the set of outlines.

Program mobilization plan


Enterprise PMO staff

Internal resource owners and managers

Review mobilization goals, needs, and best practices. Iteratively outline/review/refine iteration activities. Quantify/qualify immediate staffing needs. Initiate staff search. Validate/publish mobilization schedule.

(Initial) capital and expense budget

Vision segment owner

Leader/team member for strategic vision

Review/understand outline of capital and expenses – from strategic vision. Define program chart of accounts.


8/13/2004

How complete must the results be? The budget is the key to moving ahead with mobilization. It must clearly identify the available (and immediately spendable) funding. Funding must be sufficient to complete the mobilization effort. The budget document should give specific spending authority to at least one individual involved in the program.

At a minimum, managers should identify the individual projects and goals, and establish a set of boundaries that delimit each project. The mobilization plan must be executable; that is, it should identify tasks and expected results, estimate the work’s magnitude, identify usable resources (including level of commitment and availability), and include a schedule with major milestones and a completion date. It should identify at least one executive sponsor who will be available no later than the mobilization start date (earlier is better). If the program involves outside resources, the process and approvals for engaging them must be completed (or scheduled for completion) no later than two weeks before mobilization — to allow time for interviews and initial staff selection.

Getting and maintaining strategic consensus Because the number of items in the strategic space for which agreement is necessary is so substantial, a key to success is gaining consensus iteratively, for each decision, and then maintaining it as you consider new strategic elements. In fact, there can be no forward movement that is not solidly based upon consensus.

For each lifecycle component shown in Figure 3, there must be consensus-building to validate each set of steps; in the end, executive leaders must agree to continue on to the next lifecycle stage. Ideally, they will do so in a face-to-face discussion, with each member expressing his or her willingness to move forward.

If some people voice doubts or point to seemingly significant issues, it is wise to define a set period in which to address these before moving to the next stage. Once the program effort is underway, it is far more difficult to resolve issues and/or adjust course. In addition, because fewer individuals are involved at this early stage, it is easier to reach consensus about how to address issues.

Table 4 provides a simple list of key points that require consensus.

Table 4: Consensus checkpoints and goals

delivery



Populate/review/refine chart of accounts. Transform funding into budget model. Review/validate/approve initial budget.

Consulting and staffing agreements


Leader/team member for strategic vision delivery

Purchasing liaison for outside staffing

Review/understand outline. Capital and expenses – from strategic vision Review initial budget. Draft/review staffing needs. Outline/validate program staffing model. Outline/validate (initial) program staffing plan. Review/update existing vendor agreements. Initiate negotiations with outside vendors as needed.

Consensus checkpoints Goals

Completed preliminary cost and benefit estimates

Agreement that the overall cost is reasonable and appropriate. Assessment that benefits will provide a suitable return on the cost.

Check for fit with strategic vision “fit”

The goals and desired program outcomes are solidly in the mainstream of the strategic vision.

Completed quantification of benefits/returns

It Is probable, and readily believable, that the scale of benefits and returns will be realized

Completed initial size estimates When judged against the needed outcomes, the initial sizing seems of sufficient scale to achieve them


8/13/2004

This article discusses just a sample of the guidance that is available through IBM Rational Unified Process and IBM Rational SUMMIT Ascendant. Our Program Management Method provides program managers with guidance both pre- and post-mobilization. For more information about this method, please either follow this link: http://www-306.ibm.com/software/rational/howtobuy/index.html or call 1-800-728-1212.

Completed capital and expense budget

Confirm that the budget conforms with both enterprise financial practices and regulatory/legal requirements. Validate that budgeted costs are realistic in scale, viewing them against those of previous projects and programs and/or current marketplace conditions.

About the author Michael F. Hanford is the chief methodologist for the IBM SUMMIT Ascendant methodologies and a member of the IBM Rational commercial methods content team. He has also worked as a methodology author, a manager for large consulting engagements, and a leader of enterprise process assessment and transformation efforts for IBM Rational and PriceWaterhouseCoopers Consulting (PWCC). Prior to joining PWCC, he was director of software engineering practices for Fidelity Investments Systems Company.


Comments?





8/13/2004




Book excerpt: What Is Thought?–“The Mind Is a Computer Program” (Chapter 2)10 Aug 2004

Book excerpt--from The Rational Edge: This book argues that the human mind’s complexity is the outcome of evolution, which has built thought processes that act unlike the standard algorithms of computer science. To understand the mind, we need to understand these thought processes and the evolutionary process that produced them in computational terms.

from What Is Thought? by Eric Baum MIT Press, 2004

This book proposes a computational explanation of thought. Just as Erwin Schrodinger in his classic 1944 work What Is Life? argued ten years before the discovery of DNA that life must be explainable at a fundamental level by physics and chemistry, Eric Baum contends that the present-day inability of computer science to explain thought and meaning is no reason to doubt there can be such an explanation. Baum argues that the complexity of mind is the outcome of evolution, which has built thought processes that act unlike the standard algorithms of computer science. To understand the mind, we need to understand these thought processes and the evolutionary process that produced them in computational terms.

The goal of Chapter 2 is to describe why and in what sense computer scientists are confident that the mind is equivalent to a computer program. In the remainder of the book, Baum describes what that computer program looks like, how it works, and how it came to its present form.

*Chapter 2 is reprinted in its entirety with permission from MIT Press.

Chapter 2 PDF file, 183 kb

Contents:Rate this article




Comments?



Page 1 of 2Book excerpt: What Is Thought?–“The Mind Is a Computer Program” (Chapter 2)

8/13/2004


2 The Mind Is a Computer Program

The goal of this chapter is to describe why and in what sense computer scientists are

confident that the mind is equivalent to a computer program. Then, in the rest of the

book, I describe what that computer program looks like, how it works, and how it

came to its present form. This section does not deal with new material; it reviews

arguments published by Turing in 1936. Readers familiar with the Turing machine

model of computation are invited to skip to section 2.1.

The modern field of computer science began in the 1930s with studies by Alonzo

Church and Alan Turing on the nature of an algorithm, also known as an e¤ective

procedure. The basic idea of an algorithm is very simple. An algorithm is simply a

recipe for computing, like a recipe for cooking, but written without any ambiguity. It

is a list of instructions, with each instruction stating exactly what to do next. It may

not necessarily be deterministic: at step 348 the algorithm may say ‘‘flip a coin and if

it comes up heads add a tablespoonful of sugar and if it comes up tails add a tea-

spoonful of salt.’’ But if it always says what to do next, how explicitly to determine

what happens next, then it is an algorithm.

We are mostly used to thinking about algorithms that have been crafted to do

something specific, like an algorithm to prepare duck a l’orange or an algorithm to

compute the digits of p. But, more generally, we may have an algorithm that just

evolves. In a sense, then, the evolution of any system under a well-defined set of laws

of physics that specify what happens next at all times is an algorithm.

Church and Turing were trying to answer a famous open question posed decades

earlier by David Hilbert: Can one write down an algorithm that would prove all the

true theorems of mathematics? Essentially: could you replace a mathematician with

a computer program. They were, of course, working before electronic computers

existed, so the notion of a computer program had not yet been exemplified. Church

and Turing recognized that to address this problem they would first have to formal-

ize what was meant by an algorithm.

Turing addressed this problem by trying to formalize the notion of a mathemati-

cian. He would see if he could reduce everything the mathematician did to a series of

steps that were simple enough to be explicitly written down.

Turing visualized a mathematician sitting and working on proving theorems, writ-

ing on a big pad of paper. The mathematician might do three things. He might read

what is on the paper, he might erase it and write something else, and he might think.

Paper is ordinarily two-dimensional, but to simplify the discussion Turing sug-

gested considering the paper as one-dimensional, so that the mathematician would

really just write on a long paper tape, which Turing assumed was divided into

squares. It will hopefully become clear, if it is not already, that the restriction to

one-dimensional paper does not a¤ect the generality of the argument, i.e. does not

alter in any way the nature of the proofs a mathematician can generate. After all,

writing on a paper tape is just like writing on lined paper if one cuts along all the

lines and staples the strips together end-to-end to form a long tape.

On each square of the tape the mathematician might write some symbols, like

0, 1, 2, p, a, b, c, and so on. We may assume, Turing supposed, that the number

of di¤erent possible symbols, while potentially large, was finite because otherwise

‘‘there would be symbols di¤ering to an arbitrarily small extent.’’ For example, if the

mathematician used compound symbols that were too lengthy, he would not be able

to tell at a glance which symbol he had written. As Turing noted, ‘‘We cannot tell at

a glance whether 9999999999999999 and 999999999999999 are the same.’’ Turing’s

idea was to break down the functioning of the mathematician into bite-size pieces. If

thought were involved in simply recognizing a lengthy compound symbol, the sym-

bol would be required to be written on di¤erent squares of the paper so that one

could account for the thought processes involved in recognizing the compound sym-

bol. The feat of recognizing the compound symbol would thus be broken down into

simpler pieces: first looking at several tape squares, and then mentally joining their

contents together to perceive the compound symbol.

Now, Turing (1937) wrote,

The behavior of the computer [Turing, in the days before electronic computers, referred to the

mathematician as a ‘‘computer’’] at any moment is determined by the symbols which he is

observing, and his ‘‘state of mind’’ at that moment. We may suppose that there is a bound B

to the number of symbols or squares which the computer can observe at one moment. If he

wishes to observe more, he must use successive observations. We will also suppose that the

number of states of mind that must be taken into account is finite. The reasons for this are of

the same character as those which restrict the number of symbols. If we admitted an infinity of

states of mind, some of them will be ‘‘arbitrarily close’’ and will be confused. (250)

This assumption of the finiteness of the number of states of mind is the crux of the

argument, so I will elaborate for a few paragraphs. Once one allows this assump-

tion, all else follows. If one considers allowing an infinite number of states, one can

achieve so-called super-Turing behavior. However, the assumption of finiteness is

natural, indeed seemingly follows from our understanding of physics.

Surely the brain of the mathematician is finite: it consists of a finite, albeit large,

number of atoms. These can be in some vast number of configurations. This number

is truly vast. If we just consider 1012 or so neurons, and simply assume that each

neuron can be firing or not firing, that already yields 21012

states for the brain as a

whole, which is a number so great as to be beyond imagination. It’s about equal to

the number of electrons in the universe, raised to the ten-billionth power. But the

number of states in which the brain might be is vastly larger even than this, because


each neuron is composed of perhaps 1018 atoms, and each atom can be in di¤erent

states, and the total number of states of the system grows exponentially with its

number of components. Perhaps there are 101030

states potentially available to the

brain, a number so large that it is hard to see how any physical machine could make

practical use of more.

However, the number of configurations the brain can be in, although truly vast, is

e¤ectively finite, for two reasons. Physics tells us that the evolution of the physical

state of the brain of the mathematician is determined from initial conditions by

solving a di¤erential equation.1 If we have two sets of initial conditions that di¤er by

a su‰ciently infinitesimal amount, there will be no measurable di¤erence in the evo-

lution after a reasonable amount of time. If we perturb each atom by 10�20 of a

centimeter, say, we expect no observable change.

It might be argued that, since Turing’s time, we have discovered the property of

chaos, in which systems can be extremely sensitive to initial conditions. Even so, they

cannot be arbitrarily sensitive to initial conditions: the evolution of chaotic systems

can be predicted given su‰cient accuracy in initial condition. It turns out that one

must specify the initial conditions to accuracy of about T bits to be able to predict

the outputs after T time steps. Given that a person’s life is finite, there is some finite

precision that would su‰ce to specify the state of his brain.

Moreover, we know that atoms are constantly subjected to random perturbations

from heat. Thus, there is a natural scale of thermal fluctuations. Fluctuations that

are, say, a million times smaller than that natural scale seemingly cannot a¤ect the

behavior of the mathematician in any useful way.2

Having given these various motivations, let’s stipulate for a moment the assump-

tion that the brain can be in only a finite number of states. We are then ready to

follow Turing’s argument. Turing broke the mathematician’s proof process into a

number of steps. In each step, the mathematician might read what is written on a

number of the squares, say any of the squares within a distance L of the square on

which the mathematician’s pen was last resting. The mathematician could of course

flip through his pages to go further than distance L, but this page flipping is reason-

ably considered to require multiple steps.

Also, the mathematician might think. Turing knew nothing whatsoever about

what thinking might entail, so he made the least restrictive assumption possible.

Whatever thinking is, if it is physical, it takes the mathematician’s brain from one

physical state to another. The physical state the mathematician’s brain goes to is

determined by its current state and by what he reads. That is the meaning of state.

The state of any physical system includes all the information characterizing it, for

instance, the positions of all molecules, and determines, together with any interaction

The Mind Is a Computer Program 35

with the rest of the world, to which state it next transits. Since Turing couldn’t spec-

ify the physics of the brain as it proves some arbitrary theorem, he allowed the brain

to do anything it could conceivably do, constrained only by the fact that its action

was determined by its state and what was written on the tape with which it was

interacting. In this way, he could bound all the possible things the mathematician

could possibly think or prove.

So, Turing allowed that thinking could possibly take the mathematician’s brain

from whichever state it was in into any of the other vast but finite number of states

available to it. A step of the proof process now consists of reading the symbols

written on some squares of the paper, the mind transiting from one state to another,

writing some symbols on some squares of the paper, and shifting one’s gaze some

number of squares to another square on the paper. Exactly what happens, what gets

written on the paper, and what state of mind one winds up in after such a step is

determined by what one’s state of mind was before the step and what one read on the

paper. So, there is e¤ectively a huge lookup table that says, if the mathematician

reads symbol j, and his mind is in state a, then he will write symbol k, and his mind

will subsequently be in state b, and he will shift his gaze some number of squares to

the right or left. Since Turing allowed for all possibilities, this lookup table is arbi-

trary: any possible assignment to each possible entry is allowed. There are thus a

truly huge number of possible lookup tables—a number in fact exponential in the

huge number of possible states of the brain. So there might be 10101030

possible tables.

But, in this view, any mathematician’s mind is perforce equivalent to one such

lookup table.

Inspired by this physical model, Turing wrote down a simple model of a computer,

now known as a Turing machine. This model was only slightly simplified from the

description just given. Instead of being able to look at squares of paper near the one

being glanced at, the computer could only look at one square at a time. Instead of

being able to shift his gaze some distance, the computer could shift his gaze only one

square.

A Turing machine thus formally consists of a read-write head, a finite lookup table,

a finite set of states including a start state and a stop state, and an infinitely long tape

on which is written a finite string of symbols from some finite alphabet (with blank

space extending infinitely far in both directions beyond where the string of symbols is

written). Figure 2.1 shows a Turing machine about to begin computation. Table 2.1

shows an example of a lookup table.

The machine starts with its head reading the start location on the tape and in the

start state. The set of symbols initially written on the tape is called its program. At

each step, the head reads the symbol it is looking at. It consults the lookup table for


what to do when it sees that symbol, given that it is in its current state. The lookup

table tells it three things. It says what symbol to write on the tape replacing the one it

just read, which of the states it should transit to next, and in which direction, right or

left, it is to move its read-write head. Since the number of states and the number of

symbols are each finite, and since the lookup table says exactly what to do for each

state and each symbol being read, this prescription gives an algorithm: it is clearly an

e¤ective procedure that always tells the machine what to do next. The computation

proceeds until the machine enters the stop state (or forever, if it never enters the stop

state). When it stops, if the program did something interesting, it may have written

the answer to some question on the tape.

Turing was able to prove mathematically that this simple model had all the power

of the more general model first presented. Moreover, he was able to show that for

some particular choices of lookup tables, one would get a Turing machine A that was

universal in the sense that it could simulate any other Turing machine program.

Given any other Turing machine, say Turing machine B, there would be a program,

i.e. a particular string of 1s and 0s written on the tape of the universal Turing

0 0 1 0 1 1 0 1 0 0 0 0 0... ...

Read-Write Head

Figure 2.1A Turing machine. Blank squares are indicated by 0. An infinite number of blank squares lie to the left andthe right. The read-write head is scanning the start state, on which 1 is written. Consulting a lookup tablelike table 2.1 next to the entry for S0 (the start state, which it is currently in), and since it is reading 1, theread-write head will write a 0, stay in state S0, and move one square to the right.

0 1

S0 1;S1;R 0;S0;R

S1 0;S1;L 1;S2;R

S2 Halt Halt

Table 2.1A lookup table with three states, S0 (the start state), S1, and S2 (the stop state), and two symbols, 0 and 1.Each entry in the table tells what to do next if the machine is in the state indicated by the row and readingthe symbol indicated by the column. A blank square is indicated by 0. Thus, if the machine is in state S0

and the read-write head is reading a square containing symbol 0, it will write a 1 in the square, move to theright, and change state to state S1.


machine—so that the universal machine simulated B’s operation step-by-step. The

way this works is simple. A’s tape is simply initiated with a description of B’s lookup

table and B’s program. As the machine A reads the tape and follows the instructions

in its lookup table, it goes through a sequence of operations. For every configuration

that machine B goes through, machine A has an exactly corresponding configuration

that it goes through. When eventually B halts (assuming it does), A halts in its cor-

responding configuration with an exactly corresponding message written on its tape.

Machine A’s computation is in logical one-to-one correspondence with machine B’s.

The proof of this is not hard but is not given here. The reader interested in seeing it

in detail is referred to Turing’s (1937) original paper or to one of a number of books

(e.g., Minsky 1967). The proof is constructive. One can describe an explicit lookup

table for Turing machine A and show exactly how it simulates B’s action, whatever

B’s lookup table may be. For each step of B’s action, A keeps explicit records on its

tape of what the state of B is, what is written on B’s tape, where on B’s tape the read-

write head is pointing. Turing machine A’s read-write head just runs back and forth

along its tape keeping these records, mirroring exactly the performance of B. As it is

simulating each step of B’s computation, A simply consults its tape, on which B’s

lookup table is enumerated, to figure out what B would do next, and modifies its tape

to keep the appropriate records. In this process, A goes through a number of steps

for each step that B does, but every step B goes through is faithfully represented in

A’s action, and when B halts, A does, and whatever string of symbols is written on

B’s tape is copied on A’s as well.

The proof that this restricted model is equivalent to the less restricted model that

was described first is essentially identical. One simply shows that the restricted model

can simulate exactly the less restricted model. Again, one gives an explicit construc-

tion where for each state of the less restricted model there is a corresponding state of

the more restricted model, and shows explicitly that the more restricted model halts

in the state corresponding to the less restricted model’s when it halts, with a corre-

sponding message written on its tape.

Moreover, it is possible to show that universal Turing machines can have remark-

ably compact descriptions. Claude Shannon (1956), who is also famous as the inven-

tor of information theory, showed that one can have a universal Turing machine

that uses only two symbols. The two-symbol universal Turing machine can simply

simulate having many more symbols by encoding the states using binary notation,

that is, using a number of symbols to represent what would be represented by a

single symbol in a Turing machine with more symbols. Marvin Minsky (1967), one

of the founders of Artificial Intelligence, wrote down a universal Turing machine

with only seven states and four symbols. This is a lookup table with only 28 entries


that specifies a Turing machine that, with an appropriate program on its tape, can

simulate the working of any other Turing machine and program whatsoever.

Thus, although this discussion started by describing a mathematician with an

enormous number of states in his mind, perhaps 21030

states, everything that he could

do can be exactly done by a Turing machine with only seven states and four symbols

that is fed an appropriate computer program in the form of an appropriate, finite

string of symbols on its tape.

I have presented the preceding argument as if it were a derivation. I gave it in a

form that implies that a Turing machine could e¤ectively simulate the computation

performed by any physical device that could be built, whether built by people or by

evolution. This is a modern, rather strong view of the Church-Turing thesis that

asserts that anything computable can be computed by a Turing machine. Some read-

ers may reasonably complain that the discussion arguing that the brain can have only

finitely many states was not mathematically rigorous.

Indeed, it has been pointed out more recently (Vergis, Steiglitz, and Dickinson

1986; Smith 1993; 1999) that if one assumes some formal set of physical laws, one

can rigorously address the question of whether a physical device could in principle be

built that could not be e¤ectively simulated by a Turing machine. Unfortunately, we

cannot answer this question rigorously for our universe, because we don’t know for

sure the exact physical laws of our universe, and the laws we believe are true are too

complicated to analyze. However, if one studies a ‘‘toy universe’’ in which some set

of simplified, formal laws are hypothesized, one can in principle answer the question.

Physicists commonly analyze toy sets of laws in order to get insights. For example,

anyone who has taken high school physics has answered questions ‘‘assuming ideal

conditions in which there is no friction.’’ Smith (1993; 1999) has studied several sets

of laws of considerable realism and complexity, and has proved that the Church-

Turing thesis holds for some sets of laws and not for others. Also, other authors (e.g.,

Siegelmann 1995) have discussed, without worrying much about the possibility of

implementation, various super-Turing models of computation, that is, models which

could compute functions that are not computable on a Turing machine. Typically

these invoke some model of a chaotic analog system.

All the super-Turing models studied by Smith and others exploit the use of real

numbers of arbitrary precision. Typically they are very unnatural. One essentially

stores the answer to some computation so complex it provably cannot be computed

on a Turing machine, storing the answer in the increasingly high-order bits of real

numbers. If the super-Turing computer only accessed any finite number of bits of

precision, say 1,000, a Turing machine could simulate the program by placing the

1,000 bits on di¤erent memory locations (di¤erent squares on the tape). So, to do


something super-Turing, to compute some function that could not be computed at all

on a Turing machine, one must use infinite precision, compute with infinitely precise

real numbers, and make fundamental use of the infinitely least significant bits of

these numbers. In fact, these super-Turing machines must make use of a number not

even computable by a Turing machine, so in a strong sense the non-Turing nature is

inserted before the computation even begins.

All these approaches are thus hopelessly delicate and nonrobust, and in the pres-

ence of even the tiniest bit of noise, would compute nothing more than can be com-

puted by standard computers. Since noise and friction are omnipresent in the real

world, it seems highly unlikely that such computers could ever be built, even in

principle. Since biological brains are excessively noisy, stochastic environments, and

since they have to have arisen by evolution, it seems very far-fetched that such

models have anything to do with intelligence or mind. The upshot of all these inves-

tigations, then, is that it seems apparent that the brain as well as any reasonable

physical device can be simulated by a Turing machine.

Turing was of course unaware of these later investigations into mathematical

physics, but neither did he suggest that his analysis of the proof process of the

mathematician was mathematically rigorous. Indeed, he o¤ered it only for intuition.

Turing (1937) wrote,

All arguments which can be given [as to the nature of what algorithms might be] are bound to

be, fundamentally, appeals to intuition, and for this reason rather unsatisfactory mathemati-

cally. The real question at issue is ‘‘What are the possible processes which can be carried out in

computing a number?’’ (249)

Turing o¤ered three arguments in support of his view that a Turing machine could

compute any function computable in any ‘‘natural model of computation.’’ The first

is the analysis of the physics of the mathematician, which I have described. He did

not claim it was rigorous but o¤ered it as ‘‘a direct appeal to intuition.’’ Turing’s

third argument, which I do not discuss here, was a version of the first, slightly modi-

fied to ‘‘avoid introducing the ‘state of mind’ by considering a more physical and

definite counterpart of it.’’

Turing’s second argument, which is often judged the most compelling, was a direct

proof that many natural models of computation, proposed by di¤erent people, were

equivalent and could all be computed by Turing machines. I discuss this next.

Can there be other kinds of computers, not Turing machines, that compute other

things? Researchers have written down various other models of what it might mean

to ‘‘compute’’ a function, that is, to execute an algorithm. One model is the class of

general recursive functions, which are all the functions that can be formed by com-


posing a certain set of rules together. For example, if f (x) and g(x) are general

recursive functions, then so is f (g(x)). General recursive functions compose a vast

and complex set of algorithms for computing functions.

Another model of what algorithms might look like is given by the Lambda calcu-

lus. These consist of expressions that at first glance don’t look much like general

recursive functions or Turing machines. However, Church was able to prove that

for every recipe computing a general recursive function, there was a corresponding

recipe in the Lambda calculus that computed the same function, and vice versa.

And Turing was able to show that for every expression in the Lambda calculus,

there was a corresponding program for a universal Turing machine.

Another model of what algorithms might look like is given by the production sys-

tems of Emil Post. In production systems, one has a set of rewrite rules that look for

patterns and rewrite them. The rewrite rules thus have a left side, which instructs one

in what pattern to look for, and a right side, which specifies what results after the

substitution. So, for example, one might have a rewrite rule xBAB y ! xC B y

where x and y are variables that match anything at all. One might also have a

string: ABABC BC. The rewrite rule could then find its pattern BAB in the string,

matching x to the prefix A and y to the su‰x CBC, and substitute its right side C B

for BAB, resulting in a new string: AC BC BC.

Production systems do not at first glance look much like Turing machines; there is

no rewrite head but rather a powerful ability for pattern matching and substitution.

However, Post proved in 1943 that collections of rewrite rules were identically pow-

erful to Turing machines: a universal Turing machine can simulate any rewrite sys-

tem, and rewrite systems exist that can simulate a universal Turing machine.

In fact, as emphasized by Minsky (1967, 227), one can translate Turing’s intuitive

discussion of the Turing machine as the operation of a mathematician into an intu-

itive discussion of Post systems. In the translated version, the mathematician has

access to a finite string on which he can perform three types of operations. He can

scan for specific subsequences of symbols, he can excise these subsequences from the

string and keep track of the substrings that remain when these are chopped out, and

he can rearrange these remaining substrings, inserting fixed strings in certain posi-

tions and deleting certain parts. This can also be seen as starting with some axioms

and deriving new lemmas, the axioms and lemmas being represented as finite strings.

As Post proved, these kinds of operations su‰ce to do all that can be done by a

Turing machine, but they can accomplish no more than a universal Turing machine.

Section 2.2 sketches how the machinery of life is similar to Post’s production sys-

tems, with DNA as well as proteins essentially doing pattern match plus substitution.


Chapter 10 discusses how Post production systems as well as systems similar to

Lambda calculi can be evolved.

Other authors wrote down still other models of computing, but in every case they

were shown to compute the identical set of functions. A universal Turing machine

can simulate step-by-step the performance of any other model that anybody has yet

had the imagination to write down, with the exception of a few models that involve

real numbers of arbitrary precision, or equivalently, an infinite number of states.

As the science fiction author Neal Stephenson put it, factories need all kinds of

tools: lathes, drill presses, typewriters, and so on. But di¤erent types of tools are not

needed for di¤erent kinds of computations, even though one can imagine many dif-

ferent types of computations that one might wish to perform. All one needs is a sin-

gle computer. It can do all the possible kinds of computations that anyone might

want to do.

Some of the proposed models were faster than a Turing machine in the sense that

an ordinary Turing machine would take many more computational steps to simulate

their programs than the models took. For example, a PC or a Mac or any ordinary

desktop computer is a random access machine. Its processor is much like that of a

Turing machine, but instead of moving one step to the right or left on its tape, it can

jump to an arbitrary location in its memory (a location specified by a symbol the

processor reads) and read or write symbols there. So, for example, it has a register

describing where it will next read or write. It can read into this register the contents

of its memory at an address given by another register. This register may contain the

number 1289976. Then it can jump to location 1289976 and read from there an

instruction telling it what to do next.

A Turing machine can simulate the operation of a random access machine, but

slowly. Imagine a Turing machine trying to copy exactly what a random access

machine is doing. When the random access machine jumps to some point in its

memory in one step, the Turing machine will have to use many steps to walk to the

corresponding location on its tape, each step moving only one space on its tape. But

it can be proved that the Turing machine can nonetheless always do the exact corre-

sponding computation, and the number of steps it takes can be bounded. If the pro-

gram takes T steps on a random access machine, it can always be simulated on the

Turing machine in no more than kT 3 steps, where k is some constant number that

does not depend on the program but depends only on the fixed lookup tables of the

random access computer and the Turing machine. Moreover, any program that takes

T steps on one random access machine can be simulated in no more than kT steps on

another random access machine. Thus, any program that runs on a PC can be ported

to a Mac, and vice versa, with at worst a constant factor slowdown.


Generally when someone writes a program for a PC or a Mac or any other mod-

ern computer, they don’t write it in 1s and 0s but rather in high-level instructions.

One can still think of what’s going on as a simple random access machine. The

computer has just had coded into it a lookup table that looks at the high-level

instructions and translates them into 1s and 0s. Using a high-level language can

make it much easier for people to program, but it doesn’t speed up the computation.

The computation still essentially consists of looking at the contents of a register in

memory, accessing a relatively small lookup table to see what that instruction means

to the computer, and executing that instruction. The instruction still just writes on

some location in memory, shifts to another location in memory, and/or shifts the

state of the computer.

A random access machine has a single processor, but other computers are multi-

processors with many independent processors that can read and write and change

state independently. The brain, composed of many neurons, seems rather like a huge

multiprocessor with many little weak computers hooked together in an interesting

way. Highly parallel computers do not have more computing power than a single

appropriately faster computer. A single computer can simulate 100 computers work-

ing in parallel by simulating the computation of each of the computers in turn. It can

simulate the first computer straightforwardly until it tries to read some information

written by another of the computers, say computer 2. Then it has to back up and

simulate whatever computer 2 was doing to figure out what the message was, before

it can go back and resume work simulating computer 1. But it is always possible to

simulate all the computations of all 100 computers if one has a computer that is 100

times as fast, or if one runs a single computer 100 times as long. Using one hundred

di¤erent computers may speed up one’s calculation by at most a factor of 100, but

never more.

The converse, that a multiprocessor consisting of 100 computers can compute 100

times as fast as a single processor, is not true. Sometimes it can, and sometimes it

can’t. It depends on what one wants the computer to do, on what the program is. If

each step of a calculation depends on the last step, having 100 computers will not

allow the calculation to be speeded up at all. Say the task is to compute the ten-

thousandth Fibonacci number in the most straightforward manner. The Fibonacci

numbers are defined as follows. The first two Fibonacci numbers are 1 and 1, and

thereafter the next Fibonacci number is generated at each step by adding the last

two. So the third Fibonacci number is 2, the fourth is 3, the fifth is 5, and so on.

What is the ten-thousandth Fibonacci number? Let’s assume this will be computed

by simply calculating each successive Fibonacci number as the sum of the previous

two. A single processor can compute the ten-thousandth Fibonacci number after


10,000 steps. At each step the last two numbers are added until one obtains the ten-

thousandth number.3 If this calculation is done on a multiprocessor, how could it be

speeded up? At any given time, to do any further calculation, one would need to

know the last two numbers. There is no obvious way to jump ahead and break up

the calculation so that one processor can work on part of it while another processor

works on another part. The multiprocessor, no matter how many processors it has,

will be no faster than a single processor at this program.4 Thus, for this computation,

one processor that is ten times as fast is better than ten processors. Using multiple

processors can only speed up some problems, and only when the programmer cor-

rectly exploits the structure of the problem.

It is important to understand that although the brain is a highly parallel computer,

we can reasonably think about how we would simulate it on an ordinary computer.

There may be advantages to the structure of the brain in that it is designed to com-

pute the exact kinds of functions it needs rapidly. Indeed, it would be shocking if

evolution had not designed the wiring diagram of the brain e‰ciently to compute

precisely whatever the brain needs to compute. The question is, rather, why evolution

has provided human beings with brains capable of doing many things that do not,

at first glance, seem evolutionarily important, such as proving mathematical theo-

rems. But whatever the brain is computing, we can equally well compute it with

the appropriate program on an ordinary computer of su‰cient speed. The ordinary

computer will need to do only as many logical operations as the brain did, in total,

just as we can simulate the actions of a parallel computer with a serial computer.

Moreover, because we can in principle map from one su‰ciently powerful model

of computer to another with no loss of power, we can switch back and forth from

model to model. If we want to talk about using artificial neural nets (a model of

computing discussed in chapter 4), we can talk about programs on an ordinary

computer. Indeed, the artificial neural nets modeled by intelligence researchers are

almost always programmed onto ordinary computers. On the other hand, many

artificial neural net models are not nearly as powerful as random access machines, or

equivalently as Turing machines, because they lack a memory on which to write.

Simple counting is a very hard problem for most artificial neural nets. These neural

nets do not have the ability to perform recursive functions. When they are aug-

mented with the ability to perform recursive functions, the techniques known for

engineering artificial neural networks break, and not much is known about how to

program them to accomplish anything, much less the astonishing range of tasks

people accomplish rapidly.

Recently there have been proposals of quantum computers. Quantum computers

are not claimed to be super-Turing because nobody claims they can compute any


functions not computable on standard computers. It is accepted that standard com-

puters can simulate, state by state and line by line, the operation of quantum com-

puters. However, it is plausibly claimed that quantum computers might violate the

strong Church-Turing thesis by computing some functions faster.

In quantum mechanics, systems can be not in one state or another but rather in a

superposition of states. For example, one can prepare a spinning particle that has

half a chance of spinning to the left and half a chance of spinning forward. The

classic example of the consequences of this is Schrodinger’s cat. Say you arrange a

box with a radioactive source in it that emits a particle with probability 1/2 per hour.

Within the box you place a live cat and a device that if it detects an emitted particle

will release poison gas, killing the cat. You close the box. After an hour, the box is in

a superposition with half a chance of holding a live cat and half a chance of holding

a dead cat. Until you open the box and observe, the cat is in a superposition of

states. When you observe, the superposition collapses and is in one state or the other,

either dead or alive. While the case of the cat may seem a bit paradoxical, it is

not hard to observe superposition e¤ects in very small-scale experiments, involving

quantum-size objects such as one or a few electrons. A quantum computer exploits

the possibility of matter to be in a superposition of states to create a computer that is

not in one state or another. Rather, it is possible to write into its memory many

states at once. By searching over all the multiple states in its memory, the quantum

computer may be able to compute faster than a standard computer.

If a quantum computer could be built, it could not compute any function not

computable by a Turing machine. Rather, it is clear that a Turing machine could

simulate the computation of a quantum computer by carefully enumerating all the

possible states. However, it is possible that a quantum computer could compute

somewhat faster than a standard random access computer. Intuitively, the magical

ability to search over a potentially exponential number of memory superpositions

seems like it could plausibly lead to more power. More concretely, an algorithm for

factoring numbers rapidly is known for a quantum computer (Shor 1994). No such

algorithm is known for a standard computer. On the other hand, neither is it known

whether it is impossible to factor numbers rapidly on a standard computer. The

factoring problem is not even NP-complete, a class of problems believed hard (see

chapter 11). But it is generally believed that factoring cannot be done rapidly on a

standard computer. Computer scientists have invested considerable e¤ort in trying to

figure out how to factor rapidly, without success. Standard cryptographic systems

could be readily deciphered if it turns out that factoring could be done rapidly.

So, the computer science community has a considerable investment in the belief

(hope?) that factoring is hard. If factoring is in fact hard on standard random access


machines, then quantum computers, while they can only compute the same functions

as a standard machine, could compute at least some of them faster.

I noted that a Turing machine can simulate in time at most kT 3 any computation

done by a random access machine program in time T. This is a polynomial slow-

down, because the time taken by the Turing machine grows only as a power function

(in this case, the cube) of the time taken by the random access machine. The strong

Church-Turing thesis says that not only can a Turing machine simulate any compu-

tation done by any other computer whatsoever but can do so with only a polynomial

slowdown.

An example of a slowdown that would be nonpolynomial is an exponential slow-

down. If a tape has T squares, and if each of these can have a 1 or a 0 written on it,

there are 2T possible ways of writing a collection of 1s and 0s on the T squares of the

tape. If a quantum computer could hold all these 2T states in a superposition and

somehow search over them all, it might be able to do in time T what on an ordinary

Turing machine, which would have to successively enumerate all the 2T possibilities,

would take an exponential amount of time.

The extent to which quantum computers are physically realizable is unclear. They

depend on keeping a very delicate quantum coherence among large-scale structures.

Not just one or a few electrons but many objects have to be kept in a complex

superposition. Large-scale objects (such as cats) are extremely hard to keep in

superposed states. If a quantum computer can be built, its functioning would depend

critically on enormously careful isolation from outside sources of interference, on

being held at near absolute zero temperatures, and on exceedingly delicate algo-

rithms that seem hopelessly nonrobust and unevolvable. It is not clear whether a

quantum computer of any interesting size is constructable even in principle, and it is

generally accepted that it would be very unlikely to occur in nature, much less in

nervous systems. Thus, it seems highly unlikely that quantum computation is rele-

vant to the mind. Even if it were to turn out that the brain is a quantum computer, it

would still be true that its operation could be exactly simulated on an ordinary

Turing machine.

To summarize this discussion: Turing formulated the notion of an algorithm as a

Turing machine, which is a simple model of a computer involving a finite lookup

table and an infinite tape. Universal Turing machines can be constructed with as few

as four symbols and seven states, that can simulate exactly the performance of any

other Turing machine. These can also simulate exactly the computation performed

by other models of computation, many seemingly very di¤erent, including every

model that people have had the imagination to write down except a few that make


critical use of infinite precision numbers. All the latter, super-Turing models, do not

seem buildable and are generally not regarded as natural models of computation.

Computer scientists thus generally accept the Church-Turing thesis that any com-

putable function can be computed by a Turing machine. This thesis is unproved

inasmuch as it essentially rests on an intuitive definition of what is meant by com-

putable. In spite of the fact that there is intuitive support for this definition, computer

scientists are open to the possibility that they have overlooked something. However,

it seems highly plausible that Turing machines could compute any function that

could be computed by any physical device, whether made by people or by evolution.

So, the consensus of computer scientists is that, in principle, a computer program

running on an ordinary computer could simulate exactly the performance of the

brain. In this sense, if in no other, the mind is equivalent to a computer program.

Our quest to understand the actions of the human brain within computer science thus

comes down to a quest to understand how the actions of the human brain can be

computed on an ordinary computer.

A skeptic could still validly assert that it has not been mathematically proved that

the mind is equivalent to a computer program. I hope, however, that the preceding

discussion has made it clear why the straightforward picture, the overwhelming con-

sensus of the field, is that the mind must be equivalent to a computer program. If it

turns out that the mind cannot be so explained, we may be driven to a whole new

vision of computer science. My hope is to show in this book that no such extremes

are necessary and that it is entirely plausible that a straightforward and elegant

explanation will emerge within computer science. This is where we should look first.

2.1 Evolution as Computation

In 1948, five years before Watson and Crick discovered the structure of DNA, John

von Neumann (1966) more or less constructed it, answering the question of how one

could construct a machine that could reproduce itself. Figure 2.2 is a copy of a figure

drawn by von Neumann. It consists of a rigid backbone, with a side chain present or

not present at each position in the backbone. The presence of a side chain represents

a 1 and its absence a 0. Readers may recognize this as logically isomorphic to DNA.

Von Neumann didn’t seek to model actual human chemistry; rather, he abstracted

the computational question of how complex structure might reproduce. He imagined

the reproducing automata floating in a sea of parts. The parts were to be simple (for

his first cut at the problem), that is, not at the level of accurate depictions of proteins,

lest he bog down in the details of chemistry. He imagined instead about a dozen

2.1 Evolution as Computation 47

types of parts. These included four logical units: one causing ‘‘stimuli’’ and three

calculating, respectively, ‘‘p or q’’ (whether either of two stimuli was present), ‘‘p and

q’’ (whether both of two stimuli were present), and ‘‘p and not-q’’ (whether one

stimulus was present and another absent). With these, it is possible to build an arbi-

trary logical circuit that could be used for controlling the construction. One can, for

example, build a network of such units (by connecting them in appropriate ways)

that is equivalent to the lookup table and the read-write head of a universal Turing

machine. A desktop computer is built of similar logical circuitry, realized in silicon

chips.

The fifth part type used in von Neumann’s construction was a simple rigid ele-

ment, like a bone, used to build a frame for an automaton. This rigid element could

be attached to other parts by a sixth type of part, a ‘‘fusing organ,’’ von Neumann

wrote, ‘‘which, when stimulated, welds or solders two parts together. . . . Connections

may be broken by a cutting organ which, when stimulated, unsolders a connection.

The eighth part is a muscle, used to produce motion. A muscle is normally rigid. It

may be connected to other parts. If stimulated at time t, it will contract to length zero

by time tþ 1, keeping all its connections. It will remain contracted as long as it is

stimulated.’’ Von Neumann listed a few other types of parts, the details of which

need not concern us here. Basically he posited a collection of part types that could be

said to abstract the functionality of how proteins might behave and that was power-

ful enough to allow computation and manipulation.

Von Neumann envisioned an automaton surrounded by a collection of such parts,

which it picks up and tests to see which type they are, sifting through to find the parts

it needs. He neglected, the first time around, the question of fuel and energy. The

question he dealt with was, how could an automaton organize its construction.

Now, let’s say you had a big collection of car parts and a fully constructed car,

and you wanted to duplicate the car from the parts. You might imagine simply

working your way through the constructed car, looking at it piece by piece. As you

worked, you would be producing a duplicate copy. For each part you came to, you

0 1 1 0 1 0 0 1 0

Figure 2.2A figure like John von Neumann’s, showing rigid backbone.


could find the identical part in your set of parts and assemble it into the correspond-

ing place in your in-progress duplicate. After a while, you’d have a duplicate car.

Maybe this would work for constructing a car from parts, but it might not be as

easy as that. After all, some of the assemblies in a car have a lot of subparts. To even

begin constructing the drive train from simple parts, say, you would first have to strip

the car down far enough to get at the drive train and then break the drive train down

to figure out what smaller parts it is constructed of and how they are connected. By

the time you have done that, you don’t have a constructed car sitting there to copy.

You may have your work cut out for you just reassembling the model you were

working from.

But as unworkable as this approach might be for constructing a copy of a car,

there are more complications still for a machine that must construct a copy of itself.

Von Neumann suggested that it would be extremely di‰cult, or even impossible,

for an automaton to examine its own three-dimensional structure and copy it, for

three reasons. First, since it is examining a three-dimensional structure, there is at

each step the question of which part to examine next. In which direction should it go

to copy next? This question is easy for a linear chain but di‰cult in three dimensions.

At a minimum, one would need to specify some method of controlling this, and it’s

not obvious what the solution is.

Second, there is the problem that as it is examining itself, it disturbs itself. For a

complex automaton composed of pieces that are reactive in complex ways, such as in

von Neumann’s simplified model, this problem appears very di‰cult. If it stimulates

some part of itself to determine what the part is, it may contract, or pass stimuli, or

unsolder a connection. Whatever happens may start a chain reaction and cause the

whole system to do strange things. There is an inherent problem of control under

self-examination, especially when what is being examined is a complex machine.

Third, von Neumann was worried that one might get into the kind of paradoxes

that arise from self-reference. Godel had famously proved that mathematics is

incomplete by discovering how to write, as a mathematical equation, the sentence,

‘‘This sentence is not provable.’’ If this equation holds, if the sentence is true, then

the equation can not be proved. If the equation does not hold, if the sentence is false,

then the equation has a proof. Within logic every statement must be either true or

false and cannot be both. Thus the equation either holds, in which case mathematics

is not complete—it contains equations that cannot be proved. Or the equation fails

to hold, in which case mathematics is inconsistent—it contains proofs of false equa-

tions. Thus Godel proved that if mathematics is consistent it must contain true

equations that cannot be proved.


This answered Hilbert’s open question (mentioned earlier in the chapter) in the

negative: there is no e¤ective procedure that can prove all the true theorems of

mathematics. But it left open a revised form of Hilbert’s question: Can one find an

e¤ective procedure that, given any mathematical assertion, would decide whether or

not the assertion had a proof ? Hilbert believed that the answer would be yes.

Turing answered this also in the negative. Turing showed that it is impossible to

decide, given a Turing machine and a program, whether the computation of running

the program will ever halt. If it will halt, then that fact is provable—the proof con-

sists of simply running the program and verifying that it halts. But one can not in

general establish that it will not halt. One’s inability to decide whether a given Turing

machine computation will halt or not is an example showing one can not decide

whether arbitrary mathematical assertions are provable.

One can have a small Turing machine and a short program such that when one

runs the program on the machine, computation never halts. If one tried to verify

whether or not it would halt, one would wait forever and it would still be running.

One might run another program for an arbitrary length of time, after which it might

halt in the next second. Moreover, there is no algorithm to shortcut the process of

deciding halting. If there was such an algorithm, Turing showed that applying it in

a self-referential way to decide if it, itself, will halt can lead to a contradiction.

Although Turing machines have short descriptions and proceed according to simple,

well-defined, syntactic rules, their actual computation can be arbitrarily complex and

unpredictable. Chapter 14 returns to this point in the context of discussing free will

and determinism.

Given the examples used in Godel’s proof and Turing’s proof, it seems intuitive

that paradoxes are inherent in self-referential systems of su‰cient complexity.5 Von

Neumann was worried that similar paradoxes might befall an automaton that

attempted to examine its own structure and reproduce it. Such an automaton would

perforce be both self-referential and complex.

Von Neumann proposed a very simple approach that avoids all these di‰culties. If

you were going to construct a car from parts, it would help if you started with a plan

telling you what to construct and where to put it, that is, detailed instructions:

1. Start with a part of type 1.

2. Attach to the end of part 1 a part of type 2 using soldering gun 3.

3. . . .

In other words, what is needed is an algorithm, a computer program, that tells you

how to construct the car. Then all you would have to do is follow instructions.


So, von Neumann proposed that the machine he would copy would be analogous

not only to the car but to the car and the algorithm for constructing it. This is a much

easier thing to duplicate than just the car. You begin by copying the algorithm. This

is easy—it’s a linear sequence of instructions. In fact, it is reducible to a Turing

machine program, since I’ve argued that every algorithm is equivalent to a Turing

machine program. Thus it is a sequence of 1s and 0s that can be copied straightfor-

wardly. Then you follow the algorithm to copy the car. The already constructed car

is superfluous, just a distraction. All you have to do is follow the instructions to get a

copy of everything you began with.

And since you want the whole thing to be a single machine, all you have to do is to

write the computer program in hardware and attach it to the car. Writing it in hard-

ware is easy because it’s logically just a sequence of 0s and 1s. In fact, writing it in

von Neumann’s simplified formal model is easy: you can build the whole thing out of

the rigid skeletal elements. That is what figure 2.2 shows: the picture von Neumann

drew of a chain of rigid elements with at each joint either a side element pointing o¤,

representing a 1, or no side element pointing o¤, indicating a 0.

Here is how von Neumann (1966) described the whole construction:

There is no great di‰culty in giving a complete axiomatic account of how to describe any

conceivable automaton in binary code. Any such description can then be represented by a

chain of rigid elements like that [of figure 2.2]. Given any automaton X, let f(X ) designate the

chain which represents X. Once you have done this, you can design a universal machine tool A

which, when furnished with such a chain f(X ) will take it and gradually consume it, at the

same time building up the automaton X from parts floating around freely in the surrounding

milieu. All this design is laborious, but it is not di‰cult in principle, for it’s a succession of

steps in formal logics. It is not qualitatively di¤erent from the type of argumentation with

which Turing constructed his universal automaton. (84)

Now, it is not hard to design a second machine, B, that can copy linear chains of

rigid elements. Fed a description of anything, f(X ), B consumes the description and

produces two copies of it. For example, given the structure in figure 2.2, B produces

two exact copies of it.

‘‘Now,’’ von Neumann writes,

we can do the following thing. We can add a certain amount of control equipment C to the

automaton Aþ B. The automaton C dominates both A and B, actuating them alternately

according to the following pattern. The control C will first cause B to make two copies of

f(X ). The control will next cause A to construct X at the price of destroying one copy of f(X ).

Finally, the control C will tie X and the remaining copy of f(X ) together and cut them loose

from the complex (Aþ Bþ C ). At the end the entity X þ f(X ) has been produced. Now

choose the aggregate (Aþ Bþ C ) for X. The automaton (Aþ Bþ C )þ f(Aþ Bþ C ) will

produce (Aþ Bþ C )þ f(Aþ Bþ C ). Hence auto-reproduction has taken place. (85)


Having thus designed the reproducing automaton (denoted here by Aþ Bþ C ),

von Neumann realized immediately one other thing: his theory immediately gives

rise to a microscopic description of evolution. Again, I don’t think I can do better

than to close this section with another quotation from von Neumann:

You can do one more thing. Let X be Aþ Bþ C þD, where D is any automaton.

Then (Aþ Bþ C )þ f(Aþ Bþ C þD) produces (Aþ Bþ C þD)þ f(Aþ Bþ C þD). In

other words, our constructing automaton is now of such a nature that in its normal oper-

ation it produces another object D as well as making a copy of itself. . . . The system

(Aþ Bþ C þD)þ f(Aþ Bþ C þD) can undergo processes similar to the process of muta-

tion. . . . By mutation I simply mean a random change of one element anywhere. . . . If there is a

change in the description f(Aþ Bþ C þD) the system will produce, not itself, but a modifi-

cation of itself. Whether the next generation can produce anything or not depends on where

the mutation is. If the change is in A, B, or C the next generation will be sterile. If the change

occurs in D, the system with the mutation is exactly like the original system except that D has

been replaced by D 0. This system can reproduce itself, but its by-product will be D 0 rather thanD. This is the normal pattern of an inheritable mutation. (86)

2.2 The Program of Life

This final section of chapter 2 briefly outlines the computational process that is life,

that produces and maintains our bodies. The goals here are to appreciate the fact

that we are nothing but a huge computation, and to marvel at the intricate working

of this machine. A priori, the computation of life and the computation of mind might

be expected to be rather di¤erent. The computation of mind must ultimately sit atop

the computations of chemistry, but many would argue that reduction to chemistry is

the wrong way to go—they conjecture instead that mind is some type of emergent

phenomenon, that is, a phenomenon qualitatively di¤erent in the aggregate than the

simpler processes that compose it. But I will ultimately describe mind in terms not so

di¤erent from the computation of life. I see both as complex, evolved computations

largely programmed in the DNA, both exploiting semantics in related ways. In any

case, it provides worthwhile background to review another example of evolved, nat-

ural computation.

You were conceived when DNA from your mother and DNA from your father

came together in an egg. These two separate DNA programs merged to form a single

program. This joint program was then executed, producing you.

In von Neumann’s day, describing this as a program might have been imaginative

or controversial, but with the understanding we have achieved over the last 50 years,

it has become mundane. The DNA is information: a sequence of bits that is read by


chemical machinery and that causes a sequence of mechanical interactions to tran-

spire, processing the information in the DNA.

The machinery is not exactly identical to the read-write head in a Turing machine,

but as I’ve said, any algorithmic machine can be logically mapped into a universal

Turing machine, and conversely, a Turing machine program is logically isomorphic

to any other model of universal computation. Thus, we can consider Turing machines,

parallel processors, Lambda calculi, Post machines, and many other models to all

equivalently be computers.

The mechanisms of life are most similar to Post machines, described earlier in this

chapter. Recall that Post machines consist of a collection of productions. Each pro-

duction has (one or more) input sequences of symbols and an output sequence. When

a production matches its input sequence in an existing string of symbols, it can sub-

stitute its output sequence in its place. As mentioned, one can think of the proof

processes of mathematicians as starting with some axioms written as strings of sym-

bols and acting on them with productions to derive lemmas and theorems, which are

other strings of symbols. Life is reasonably well described as a giant Post production

system. Again and again in life, computation proceeds by matching a pattern and

thus invoking the next computational step, which is typically similar to invoking a

Post production by posting some new sequence for later patterns to match.

The DNA program is arranged in submodules called genes, organized into collec-

tions called chromosomes. Each chromosome is a strand of DNA. This strand is a

long molecule that is explicitly analogous to a computer program: its organization

conveys logical information encoded in a sequence of discrete symbols that con-

trols the operation. Alternatively, it is a long sequence of symbols as operated on

in Post machines. The DNA molecule consists of a string of bases, where each base

can be one of four possible molecules represented, respectively, by the four symbols

A;C;G;T. Thus, a strand of DNA might be . . . AATGAACTTG . . . .

A gene is a small subroutine of this program containing the instructions for pro-

ducing a protein. As is usual with subroutines, the execution of a particular sub-

routine corresponding to a given gene begins when the subroutine is called by the

execution of other parts of the program.

This is done when molecules called transcription factors (analogous to productions

in a Post machine) are produced elsewhere in the program. The transcription factors

recognize the beginning of the gene by finding specific short sequences of nucleotides

nearby, each transcription factor recognizing certain specific patterns of nucleotides

and attaching itself to the DNA sequence where it recognizes this pattern. Thus, as

with Post systems, execution of the gene begins by matching one or more patterns in

2.2 The Program of Life 53

a string. The combination of such a pattern on the DNA sequence and the tran-

scription factor that recognizes it is appropriately called a control module (Branden

and Tooze 1999, 151).

The matching of the transcription factors e¤ectively posts other patterns, which

are matched by a molecule called polymerase that attaches to the DNA at a point

specified by the pattern matching. The polymerase transcribes the information in the

DNA base by base into the related molecule RNA in a one-to-one mapping, each A

in the DNA sequence being mapped into a U in the RNA sequence, each C into a G,

each T into an A, and each G into a C. Thus the strand . . . AATGAACTTG . . .

would be mapped into . . . UUACUUGAAC . . . . So, at the end of this map,

an RNA sequence with the identical information content as was in the gene is pre-

pared. The transcription proceeds along the DNA, copying the whole gene (or often

a sequence of genes) into RNA until it is stopped upon encountering a specific stop

pattern.

Why is the information copied into RNA before further processing? One theory is

that this structure arose as an artifact of evolutionary history. Because RNA can

serve as a catalyst as well as carry information, it is believed that first life was purely

RNA-based, involving no DNA. Thus, the mechanisms of life evolved using RNA.

As creatures became more complex, evolution discovered that it could use DNA,

which is chemically more stable than RNA, as long-term storage for the valuable

information. But it continues to translate this information back into RNA to employ

descendants of the previously evolved mechanisms for manipulating information

in RNA.

Large portions of the RNA sequence called introns are then carefully snipped out

(see the box What Is Junk DNA?). This is done by small particles (made of a com-

bination of proteins and nucleic acids) that contain RNA sequences matching the

sequences at the junctions between the introns and the adjacent coding regions. Such

particles bind to a junction end, twist its intron into a loop, excise the loop, and

splice the coding regions (the exons) back together.

The RNA sequence that remains when the exons are stitched back together is

the program for producing a protein. It is translated into a protein as follows. The

sequence of letters is parsed into words by reading the bases sequentially three at a

time. Three sequential bases are called a codon because they code for an amino acid.

So AAA is a codon, as is AAC, and so on, through all the 64 possible combinations

of three letters from the set {A;C;G;U}. The codons are translated into amino acids

using a simple lookup table called the genetic code. The RNA is read starting at a

position on the sequence where the codon AUG is found, indicating the start of the


What Is ‘‘Junk’’ DNA?

Only some 1 percent of the human genome is expressed into proteins, the rest consisting of intronsand other sequences that are not expressed into proteins. A small fraction of the unexpressed DNAconsists of sequences that serve as promoters or repressors—tags matched in the control of geneexpression. It is not clear to what extent the rest, sometimes dubbed ‘‘junk’’ DNA, serves any usefulfunction.

Roughly half of the human genome is made up of repetitive elements. These do not seem impor-tant for function. Much of this repetition is most likely an artifact of the evolutionary process.That is, these repetitive sequences are small chunks of DNA that selfishly succeed in getting them-selves copied without benefit to the organism but also without su‰cient deleterious e¤ect to be se-lected out. Evolution selects for whatever gets propagated into future generations, and theserepetive sequences are plainly good at that, propagating hundreds of thousands of copies of them-selves into every human’s genome. Similar artifacts appear in experiments where computer scientistsartificially evolve computer programs (see chapter 10). In such experiments as well, the programgets clogged with sequences of instructions that are not executed but float along for the ride.

While the repetitive elements in the genome do not seem important to function, they are impor-tant to evolution. They consist in part, for example, of copies of genes that have mutated to losetheir original function. But numerous examples are known of chunks of such pseudogenes combin-ing or mutating to acquire new function. Once a gene is copied, one copy may become particularlyfree to evolve, since the other copy may fulfill the original function. For example, the Hox genes arefundamental in organizing development in bilaterally symmetric animals. Humans have 39 Hoxgenes, which likely evolved from a single original gene through duplication events followed by fur-ther evolution (Carroll, Grenier, and Weatherbee 2001). Some 10 percent of the genome is a singlerepetive element called Alu. Alu carries with it promoter sites, so its insertion in the DNA canactivate nearby pseudogenes. Its highly repetitive nature gives rise to recombination errors of rea-sonably well understood types, which can also lead to discovery of new genes (Makalowski 2000).

Within the genes themselves, the presence of introns (which are snipped out and not expressedinto proteins) between the exons (which are expressed) also apparently aids evolution. You havetwo sequences of DNA, one from your mother and the other from your father. When you create anegg or sperm cell (depending on your sex) these sequences cross over, swap corresponding sequencesin a randomized fashion. Much of the genetic variability from generation to generation is createdby this crossover. Because relatively short exons, which are expressed into proteins, are separated bylong introns, which are snipped out before proteins are created, most of the swaps break intronsand not exons. Exons are usually swapped whole, surrounded by portions of introns. Thus, theexons can serve as building blocks, and crossover can serve to mix the building blocks in search ofideal combinations. If it weren’t for the introns, crossover would break the building blocks in themiddle, resulting in many more inoperable programs. Suitably positioned introns thus plausiblyrender creatures more evolvable.

The exonic units, in fact, often code for domains within the protein, units of the protein that foldcompactly and may have some particular function (De Souza et al. 1998). A protein may be madeof several such domains (sometimes corresponding to several exons in the gene coding for the pro-tein) that can sometimes each be interpreted as having some particular ‘‘semantic’’—higher-levelmeaning in the context of the system—by virtue of their function. This is one of many exampleswhere evolution has discovered computational units that can be combined into higher computa-tions. The evolution of such units may contribute inductive bias (see sections 12.2 and 12.5). Evo-lution has made discoveries that allow faster search of semantic space by combining meaningfulunits. By contrast, ‘‘genetic algorithms’’ used by computer scientists who are trying to evolve solu-tions to optimization problems, do not have the benefit of a billion years’ evolution of evolvabilityand thus su¤er greatly from the destruction of useful structure by crossover (see section 5.6).

Aside from their impact on evolution, the locations of the introns have computational roles. Thelocation of introns is used in proofreading, to prevent errors (Schell, Kulozik, and Heutze 2002).Moreover, the fact that the genes are broken up into a collection of exons potentially allows large


sequence for a protein, and ending with one of the codons UAA, UAG, and UGA

that indicate the end of the sequence for a protein. As the reading progresses, each

triplet encountered is translated into amino an acid. So UUU is translated into the

amino acid phenylanine, UUA is translated into the amino acid leucine, and so on,

according to the lookup table. An extensive and marvelous molecular machinery

enforces this translation, creating each amino acid as it goes along reading the

sequential program and attaching it to the growing sequence of amino acids. This

machinery is implemented as a number of Post production–like molecules that each

match a pattern (e.g., a three-letter codon) and post another pattern (in this case, the

corresponding amino acid).

In this way, a molecule is created consisting of a long chain of precisely coded

amino acids. This molecule, called a protein, then folds up into a tight ball according

to physical attractions and repulsions between the amino acids in the sequence and

the substances (such as water) outside. In principle, the physics by which the amino

acids attract and repel each other is well understood. Basically, it occurs because like

charges repel and unlike attract, according to the laws of electrodynamics. One could

thus, in principle at least, simulate the physics of the folding on a computer and say

exactly how a given sequence of amino acids will fold. In practice, such simulation

requires more computational resources than scientists yet have available.

The protein sequences that are specified in the DNA fold quite tightly. It is possi-

ble to synthesize any sequence of amino acids in the lab, but almost all sequences so

synthesized will not fold nearly as tightly as the ones used in natural proteins. The

proteins coded in the DNA have been carefully selected (by natural selection) to fold

numbers of alternative splicings, allowing the same stretch of DNA to code for numerous proteins.This mechanism allows di¤erent cells to express di¤erent proteins. For example, di¤erent soundreceptor cells in the inner ear of birds express 576 alternative splicings of a certain RNA, thus layingdown a gradient of di¤erent proteins that is utilized for perception of di¤erent sound frequencies(Black 1998).

It is not entirely clear whether or to what extent the sequence of bases in the introns (rather thanmerely the location of the introns) is important. Thus, it is not clear to what extent we should countthe sequences within the introns in estimating the information content of the DNA program. Anear-consensus holds that after the introns are expressed into RNA, the RNA degrades withoutfurther function, and thus the code in the introns could be altered without modification of function.If this view is accurate, the length of the introns should not be counted toward the informationcontent in the program. I therefore neglected the introns and the rest of the ‘‘junk’’ DNA in the10-megabyte estimate of the length of the DNA program (see the box How Big Is the DNA Pro-gram?). This view is supported by our growing knowledge of genetic regulatory circuits, whichinvolve proteins and regulatory elements but do not involve most of the DNA. However, it has alsobeen proposed that this intronic RNA may be computationally important, interacting to influencewhich later genes are expressed into proteins (Mattick and Gagen 2001).


into tight molecules. In fact, additional molecular machinery has evolved to help

the proteins fold into their most compact, lowest-energy state even in the crowded

molecular environment found within cells.6

After the sequence has folded, the shape of the folded structure determines the

subsequent chemical reactions it will undergo. Some regions on the surface of the

folded protein can react with regions on the surface of other proteins. This occurs,

roughly speaking, because the shape of the surface and the pattern of positive and

negative charges exposed there fit the pattern on another protein. One fits into

another like a key fits into a lock—the two patterns match. As with a key and a lock,

How Big Is the DNA Program?

How much e¤ective information content is there in the DNA program? Neglecting the ‘‘junk,’’ Iestimate this to be 10 megabytes. The human genome comprises roughly 30,000 genes, each codingfor a protein roughly 300 amino acids long on average. Each amino acid requires less than 5 bitsto specify (since there are 20 alternative amino acids). Thus, one arrives at 45 million bits, or sincethere are 8 bits to the byte, roughly 5 megabytes. The estimate of 10 megabytes thus allows for afudge factor of 2 to take account of additional regulatory information.

It’s possible that this 10-megabyte estimate is overgenerous. The true information content ofthe DNA program might be quite a bit smaller because the functionality of the program would beidentical under many changes in the amino acids. Proteins form three-dimensional structures byfolding up linear molecules made of sequences of amino acids. The function of the protein, how itinteracts in the program, depends on its folded shape, but the fold is not sensitive to many changesin the amino acids. Di¤erent proteins evolve having the same fold and function that have only 30percent or less of their amino acids in common. The proteins fold in such a way that amino acidsthat are hydrophobic are on the inside of the fold and amino acids that are hydrophilic are on theoutside. It is an exaggeration to claim that this is all that counts about amino acids, but in manycases this a reasonable approximation. This approximation would lead to the estimate that the trueinformation content per amino acid is actually closer to 1 bit than 5. Support for this picture comesfrom experiments that led Keefe and Szostak (2001) to the estimate that about 1 in 1011 sequencesof 80 amino acids has a given, specified chemical functionality. This figure suggests that the trueinformation in a gene is only log2 10

11@ 36 bits. Such arguments suggest that the total informationcontent in the program for a human being might be closer to 1 megabyte than 10.

There are other reasons why the e¤ective information content of the DNA program might besmaller than our estimate, potentially even smaller than 1 megabyte. A naive count of the weightsin a neural net can overcount its e¤ective number of parameters, especially when it is trained usinga procedure called early stopping (see section 6.2). Similar e¤ects may well apply to counting thee¤ective number of bits in the DNA program.

The 10-megabyte figure could be too low if we are underestimating regulatory information. Cur-rent understanding does not allow a confident calculation of the useful information content outsideof the genes. However, the full genome, including all ‘‘junk,’’ is only about 3 billion base pairs long,and thus contains less than 6 billion bits. This absolute upper bound is still quite compact comparedto the naive complexity of the world and to the length of the training data. If information from 1035

or so evolutionary learning trials has a¤ected the program, then even with all the ‘‘junk’’ included,the length of the DNA program is vastly smaller than the information that went into its evolution(see chapter 5). Why the relevant comparison is to the useful information content rather than to thetotal length is discussed in chapter 6.


the proteins in the body are evolved to be very specific about what fits where. Each

key can fit only in locks designed to be a close match.

If we understood how to predict the folding, and better yet, how to design sequences

that would fold into di¤erent shapes, we could design drugs for many purposes.

Whatever key we needed for a given lock, such as a virus or an enzyme, we would

presumably design. The massive pharmaceutical market has seen to it that this is one

of the most heavily studied fields in science. Nonetheless, our computers are not yet

powerful enough to reliably simulate the folding of proteins, and scientists do not

yet have a shortcut allowing them reliably to predict how proteins will fold and thus

cannot generally create proteins that fold nearly as tightly as the ones coded for in

DNA. The proteins, in deciding how to fold, do an analog computation of great

complexity.

The proteins within the body react when they find an appropriate mate. In other

words, we again have a Post system–like pattern-matching.7 The proteins float

around, looking to match a pattern, and create other patterns when they do.

Often the proteins react catalytically, which means that they trap some other pro-

teins long enough, and in an appropriate way, to get them to react together. An

enzyme may have patterns that match patterns on two di¤erent specific proteins well.

It will then hold them in place for a while, and perhaps influence their shapes, so that

they react with each other. The enzyme then emerges, free to catalyze other reactions.

Three-party interactions like this, where one entity gates a reaction between two

others, are quite useful for building universal computers. Indeed, that is essentially

what a transistor does. Like transistors, the interactions between proteins allow uni-

versal computation.

Enzymes, for example, serve to gate reactions. Consider the reaction shown in

figure 2.3. Here A, B, C, and D are compounds and 1, 2, and 3 are enzymes cata-

lyzing the reactions, respectively, A ! B, ! C, and ! D. In the presence of enzyme

1, compound B will be produced from A. In the absence of enzyme 1, very little of

B will be produced. Similarly, the presence or absence of enzyme 2 or 3 determines

whether the reaction forks to produce compound C or compound D. To a computer

scientist, this is nothing more than a simple flowchart showing the results of some

if-then instructions.

The usual first step in writing a computer program is to produce a flowchart. A

flowchart is a graphical representation of what the program will do that consists of

nodes representing the instructions in the program connected by arrows showing the

flow of control through the program. So, an arrow will connect one node to another

if the program will first execute the one instruction followed by the other. Because


programs branch depending on the partial results as the computation proceeds, a

given node may have several arrows leading from it to other nodes. Typically the

nodes are represented in a flowchart by ovals, squares, or diamonds depending on

the type of instruction the node represents, and the instruction is written within the

figure (e.g., within the oval). Figure 2.3 can be thought of as a very simple flowchart

showing flow from state A to state B if 1 is present, and flow from state B to state C

or to state D depending on whether enzyme 2 or 3, respectively, is present.

The human metabolism is an enormously complex program with an enormously

complex flowchart. It is broken down into modules called pathways, tightly orga-

nized metabolic systems in which certain enzymes react to produce products necessary

to other reactions. The whole reaction runs in an organized fashion, all programmed

originally in the genes, to do a huge logical calculation that causes a person to

develop in precise fashion, behave properly, and keep on going for 70-odd years.

Thousands of proteins are created at the right times, in the right concentrations.

Which products are created next depend on what molecules are present, in a way

analogous to the pathway represented in figure 2.3. The pathways feed back on

themselves in complex ways to control the metabolism so that it works precisely.

An awe-inspiring wall poster, about 3 0 ft� 10 0 ft, titled ‘‘Biochemical Pathways’’

(Michal 1998), shows the entire flowchart of the metabolism (or, at least, all the

portions that have been worked out). The poster is filled with a flowchart of fine lines

labeled by fine print and would be completely illegible if I tried to reproduce it here

on a book-size page. When one views this poster, one’s first impression8 is awe at the

complexity of the program and respect for the extensive scientific e¤orts that have

gone into unraveling it.

A

B

C D

1

2 3

Figure 2.3A simple enzymatic pathway. Compounds B, C, and D are produced from A depending on the presence ofenzymes 1, 2, and 3.


To some extent there is a random element in this machinery. Molecules move

somewhat randomly within the body (which is composed largely of water) and react

when they run into the molecules they are designed to interact with. This does not

detract from the nature of the system as a computer: it is a somewhat randomized

computation that, however, is designed (by evolution) to work well in the presence of

this randomness.9

But there are also carefully constructed molecular sca¤olds for conveying infor-

mation in precise ways and to precise places to cause precise reactions. For example,

there are long molecules called receptor tyrosine kinases that serve to bring infor-

mation into cells. These stick out through the surface of the cell, and extend into the

cell into molecular complexes. When a hormone molecule docks at the sensor por-

tion of the molecular complex that sticks out of the cell wall (which it does upon

recognizing an appropriate pattern there), molecular changes in the complex occur,

enabling enzymatic modules in the protein complex to pass along the information

that a specific hormone has been received. The information is thus conveyed to spe-

cific molecules within the cell that act in a deterministic way. DNA within the cell

may be activated to produce a certain protein, for example, insulin if the cell is a

pancreatic cell and has been appropriately stimulated. Similar molecular complexes

serve as the brain of the E. coli, causing the bacterium to swim toward nutrients and

away from poisons (see section 13.1.1).

2.2.1 Genetic Regulatory Networks

I have described how a gene is transcribed into a protein but have yet to speak of

the complex structure that controls which genes are transcribed when. This control

structure again is machinelike, indeed Post system–like. Each cell in the body con-

tains a full copy of that body’s DNA. What determines whether it is a muscle cell or

a brain cell is which genes are executed, that is, turned into proteins. (The usual term

is that a gene is expressed. I occasionally use executed to emphasize that gene is

basically a small subroutine.) To keep the metabolism working, specific genes have to

be expressed at specific times. That is, a thread of the execution of the full program

of life has to call the subroutine of that specific gene at the correct times. This sub-

routine calling is done by a vast genomic regulatory network, which interacts with

the metabolic flowchart.

The details of the regulatory network are far from worked out, but pieces of it are

being elucidated. For example, the gene regulatory network that controls substantial

portions of the development of the sea urchin embryo is surveyed by Davidson et al.

(2002). The regulatory networks that control development in bilaterally symmetric

animals, and the evolution of these networks from the earliest such animals to more


complex vertebrates, are surveyed by Carroll, Grenier, and Weatherbee (2001). Gene

regulation proceeds through a complex network where the presence or absence of a

given product determines the branching of a program. Genes are turned o¤ or on

depending on whether repressors or inducers are present, which in turn depends on

whether other products are present. The system explicitly realizes a logic: production

of one product may depend on the conjunction or the disjunction of other products,

and on more complex logical circuits.

As development proceeds, it utilizes memory to control the program flow—

memory, for example, in the form of DNA rearrangements and molecular mod-

ifications such as methylation patterns. Methylation occurs when a molecule called a

methyl is attached to a protein or a nucleic acid. For example, some of the cytosines

in a gene may be methylated, and which particular cytosines are methylated influence

whether the gene is active. When a liver cell divides, it produces new liver cells, which

implies that it remembers which genes should be active and which inactive in a liver

cell. Other genes would be active in a neuron or a skin cell. An important way that

this memory is stored is in the methylation patterns, which are carefully conserved

when the DNA is replicated. Stem cells, which famously can develop into any type of

cell, have all these memory mechanisms initiated blank (Maynard Smith and Szath-

mary 1999, 113–114).

All this logic is implemented in a Post-like production system. For example, a

repressor, which is a protein that binds to DNA, represses the expression of a gene

by matching a specific pattern on the DNA and attaching itself where it is matched.

This then has some e¤ect that suppresses expression of the DNA, such as covering up

a nearby location on the DNA where a promoter might otherwise match to induce

expression.

Development proceeds as a program execution, starting with a single cell. Molec-

ular computations of the kinds just outlined determine which genes are expressed,

and the cell then duplicates. More molecular computations occur, and each of the

two cells duplicate again. At precise duplication points, the cells begin to specialize,

to become liver and brain and muscle cells. Everything proceeds according to logic,

as specified in the DNA program. The DNA program continues its clockwork exe-

cution for 70 or more years of a person’s life, the liver cells, for instance, remember-

ing all that time how they should act to be liver cells. They act correctly by executing

a complex program of molecular interactions. The working memory of the program

(e¤ectively the tape of the Turing machine, or more aptly, the collection of strings

and productions in the Post machine) is stored in the proteins present in the cells, in

nucleotide sequences present in the cells, and in modifications of these proteins and

nucleotides.


2.2.2 The Evolution of Development and Mind

The regulatory networks coding for development, and the evolution of these net-

works, have aspects that mirror features in thought. For one thing, although devel-

opment is a mechanical computation, certain syntactic elements can be seen as

having acquired ‘‘semantics’’ by virtue of their position in the computation and their

interaction with the rest of the network. For example, development is largely deter-

mined by the expression and suppression of certain toolkit genes, such as Hox genes,

and other cell-specific selectors. These encode proteins that bind to DNA, regulating

expression of each other and ultimately of other genes. They interact to turn each

other on or o¤ in appropriate regions of the developing body. Which such genes are

expressed determines the future development, and these toolkit genes can thus be

seen as having meaning. For example, expression of the ey gene during development

seems to code for creation of an eye. Recall that the fly’s eye is quite a complex

object, a compound eye composed of many tiny eyes, in appropriate order, with

appropriate pigment, and so on. Yet expression of the ey gene on the wing of the fly

during development will lead to a well-formed eye growing on the wing. Of course,

the ey gene does not by itself build an eye, but it sets in motion a pathway, a set of

signaling events between di¤erent cells involving a number of genes, that directs

construction of the eye.

Development in all the bilaterally symmetric animals, including, for instance,

insects and human beings, is controlled by closely related toolkit genes. Human

toolkit genes code for proteins that are quite similar to those of very simple animals,

and indeed most of our proteins are similar. The toolkit genes are su‰ciently related

that the expression of the mouse eye gene on the wing of a fly will cause a well-

formed fly eye to grow there. Note that the mouse eye gene, when artificially inserted

and expressed on the fly wing, causes a fly eye to form, not a mouse eye, so the

semantics of the mouse eye gene depends on its context: what it says is ‘‘activate the

pathway for making an eye,’’ or in other terms, ‘‘call the subroutine for making an

eye,’’ and then the subroutine (coded elsewhere in the genome) creates the appropri-

ate eye.

Because the genes and proteins themselves are so similar, most of the evolution

in development from the simplest bilaterally symmetric animals, through insects,

through human beings is thought to have occurred through evolution in gene regu-

lation: additions, mutation, and deletions of promoter and repressor sites near genes,

especially near the toolkit genes themselves (Carroll, Grenier, and Weatherbee 2001).

Because there are changes in when genes are called, the developmental program is

modified from creature to creature even though the proteins manufactured by the


genes are very similar. These kinds of changes can swap whole existing subroutines,

calling them at di¤erent times, or slightly modify subroutines, essentially changing a

few lines of code.

Development is modular, as is the program of mind. Changes in regulatory ele-

ments can thus e¤ect gene expression in one localized structure. Evolution discovers

new structure by putting together (perhaps with small modifications) previously dis-

covered subroutines into new programs, as well as possibly making small changes in

the existing subroutines. I argue that mind, and the evolution of mind, does the same.

This is also related to the phenomenon of inductive bias (see section 12.2). Evolu-

tion worked long and hard, for many hundreds of millions of years, to discover the

first useful subroutines, including the pathways that develop and run single cells, and

then circuits for organizing simple multicelled creatures. As more useful subroutines

were discovered, the search for new organization became increasingly focused and

productive. Evolution built on top of the previous subroutines, finding new ways to

write programs based on the previous subroutines rather than learning from scratch.

I argue that evolution did this not only in discovering body organization but in dis-

covering mind, where the subroutines include computational modules such as basic

concepts. Thought and learning are rapid because they are based on such concepts,

which themselves took a long time to discover.

An additional parallel worth mentioning between the genetic circuits determining

development and the modular organization of mind (see chapter 9) is that both mind

and development reuse their modules in many contexts. The genes in the genetic

toolkit are pleitropic, meaning that they are reused in di¤erent contexts for di¤erent

functions, for instance, acting at di¤erent points in the development of one tissue,

or in di¤erent tissues. For example, the wingless10 pathway is invoked early in the

development of Drosophila to organize segment polarity, then called again in the

embryo to initiate formation of legs and wings, and then days later is employed

in the larva to organize polarity, outgrowth, and sensory organ patterning in the

developing wing, and finally is essential to organize polarity of the eye and leg,

among other tissues (Carroll, Grenier, and Weatherbee 2001, 39–42).

Of course, the DNA, and thus the genetic regulatory circuitry, codes not only for

a creature’s development but for its mature life processes and its interactions with

the world. For example, when a person forms long-term declarative memories, for

instance memorizes some fact, certain specific proteins have to be produced in the

neurons, thus modifying the connective strengths of synapses between the neurons

and encoding the memory. To do this, genes for these proteins must be turned on

at the appropriate times. The networks that turn on the appropriate genes, and

the cascades of reactions that lead to strengthening synapses in order to lay down


long-term memories, have been worked out to a remarkable extent (see Kandel

2001). In this way, the thought of creatures all the way through human beings builds

directly on subroutines that evolution discovered in early multicelled creatures.

The molecular biology of memory is better understood than the computation

itself, that is, the molecular biology that has been worked out describes the mechan-

ics of the process, how the synapses are modified. Relatively little is known about the

computational logic of the process, about how the brain computes which synapses to

strengthen. This is somewhat analogous to understanding the basic physics of the

transistors in a computer but having little clue as to the software. This book is gen-

erally more concerned with the computational logic than with the transistor-level

hardware.

However, the history of the evolution of development of the body suggests that

mental processes and such synapse modification hardware evolved similarly in that

meaningful subroutines evolved and then were swapped around as evolution pro-

gressed. We know this is true for subroutines such as those just described for

strengthening synapses and for building an eye (recall that the retina is composed of

neurons and is thus sensibly considered to be part of the brain). Why shouldn’t mind

also have been built up out of semantic units coding for computational or cognitive

modules, perhaps for things such as calculating how to manage valuable resources,

or understanding topology, and for subconcepts from which these bigger units may

be composed. If there is some kind of universal memory-writing unit, why not vari-

ous units that interact to specify the logic of when and which synapses this unit

strengthens?

In this picture, the human mind evolved from simpler minds partly by an evolution

of genetic regulation putting units together in new ways, much as the plan for the

human body evolved. In this picture, mind is composed of interacting, semantically

meaningful computational modules that call one another, as genetic regulatory net-

works do. In this picture, evolution was able to explore the space of mind programs

much more e¤ectively once it discovered semantically meaningful chunks it could

manipulate. It took evolution vast computation to find the first useful concepts, but

once it was reasoning semantically, it may have built us with relatively little (though

still vast) computation.

2.2.3 Life and Post Production Systems

I went through this broad-brush overview of biochemistry to provide background, to

spur some appreciation of it as program execution. The program of life is an incred-

ibly complex program, but it is similar to the evolution of a big Post production

system. In string rewrite systems such as Post systems, a program consists of a col-


lection of strings of symbols and a collection of rules like ‘‘if you find some pattern in

a string, make a substitution to add a new string’’ or ‘‘if you find some pattern in one

string, and you also find some pattern in another string, add some particular string to

the population.’’

Again and again in the machinery of life we find computations of this form. DNA

enzymes search for patterns in DNA, for instance, promoter sequences, and execute

a complex series of actions when they find one. Enzymes search for two specific

partners (two proteins with specific patterns) and create a new molecule, with new

patterns, when they find them. The entire computation is digital, that is, it depends

on the presence or absence of specific molecules. It is largely pattern-based. It is

programmed as a deep sequence analogous to string rewrites, where each step

involves productions matching patterns and posting new patterns that recruit later

productions.

The computation that goes on in the body is incredibly deep, with layer followed

by layer of computation. There is computation at the level of DNA string reading, at

the level of the genetic code where it is mapped onto proteins, at the level of protein

folding, at the level of protein interactions, feeding back to control again which

strings are read.

Chapter 10 describes some computational experiments with evolutionary pro-

grams, including evolutionary Post systems. The results are intriguing, but they are

dwarfed by the complexity of life. Those experiments were able to execute only a few

tens of millions of productions. By comparison, the execution of the program that

creates life is incredibly vast. It has parallelism in the tens of trillions of cells that

each execute programs, and in the large numbers of parallel chemical reactions going

on in each cell, including pattern searches involving many molecules in each cell.

However, all this is controlled by a DNA program only a few tens of millions of base

pairs long (omitting the noncoding DNA). So a relatively short program encodes an

enormous calculation of enormous logical depth.

The model I propose for the computation of mind is analogous to the one for the

computation of life. It is another giant program execution, like a giant Post system.

Like the computation of life, the computation of mind is rich, with modules con-

nected to modules flowing in complex flow patterns. Like the computation of life, the

computation of mind is the result of evolution. And it is coded by a short program,

as the computation of life is, so that there is an underlying order.





Measuring upGary Pollice Worcester Polytechnic Institute 5 Aug 2004

from The Rational Edge: Software developers are notoriously bad at planning projects because they can’t estimate schedules or cost reliably, and they fail to evaluate what happened at the end of a project. This article discusses how software developers can measure their work and arrive at more accurate estimates for planning purposes.

When your project manager asks you how long it will take to deliver defect-free results for a particular development task, how confident are you of your answer? As software developers, we are notoriously bad at planning our projects. We don't estimate schedules or cost reliably. At the end of our projects, we typically fail to evaluate how we did, so it's hard to learn from the past and improve future performance.

This month, I discuss how you, as a software developer, can measure your work; I describe what to measure and the benefits of doing so. Next month, I'll step up a level to look at project and organization metrics.

Computer science is different from software engineering. Compare these definitions:

Computer science. "The systematic study of computing systems and computation. The body of knowledge resulting from this discipline contains theories for understanding computing systems and methods; design methodology, algorithms, and tools; methods for the testing of concepts; methods of analysis and verification; and knowledge representation and implementation." 1 Software engineering. "(1) The application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software: that is, the application of engineering to software. (2) The study of approaches as in (1)."2

Computer science forms a basis for software engineering, but software engineering is broader in scope and less concerned with theoretical results. In short, software engineering is like other engineering disciplines: It focuses on applying science and technology to practical problems.

Engineering requires observation, measurement, and calculation of things. For example, an engineer might ask how long it takes to build a certain type of bridge, or how much stress can be placed on a ceiling beam. Established engineering disciplines use a rich set of measures and metrics to predict how long a project will take, how much it will cost, and how well the project succeeded.

By contrast, software engineering practice needs more work. As industry practitioners, we do a poor job of gathering the data necessary to help us plan effectively and deliver products of high quality. Here are a few examples of the reasons—excuses really—for our poor performance:

Software is unique. We never really do the same thing twice. Software engineering is a knowledge-based discipline that often requires innovation and experimentation. How can you plan for that?

Contents:Measures, measurements, metrics, and indicatorsWhy measure your personal work?What should you measure?

Analyzing the resultsNot all projects are equalFinal thoughtsFurther readingNotes




Page 1 of 9Measuring up

8/23/2004


Technology changes so rapidly that any metrics based on one set of technologies may not apply to a different set.

Instead of apologizing for why we don't, or can't, measure, I plan to show you what we can do. In this article, I discuss what we can measure and how our observations can help us be better software engineers.

Measures, measurements, metrics, and indicators The terms measure, measurement, and metric are often used interchangeably. However, for software engineering purposes, these terms have distinct definitions:

A measure (noun) is a quantitative indication of an attribute, such as its size, dimension, or capacity. An example of a measure is the number of defects found in a program. The measure is the raw number. Measurement is the act of determining a measure. To perform a measurement, we measure (verb) the number of defects in the code to give us the defect measure (noun). A metric is a quantitative indication of the degree to which a system or component possesses an attribute. So a metric might be the number of defects per thousand lines of code, which is the defect density metric. In other words, a metric is described in terms of the relationship between measures.

We also need to consider an indicator, a metric or combination of metrics that provides insight into our process or product. An indicator is often expressed as a trend, such as the defect density per month. A decrease in defect density is a good indicator of software quality.3

Why measure your personal work? You might wonder why it's a good idea to make the effort to measure your work. It takes time to capture data, and that's time you really cannot afford. You probably have a good idea about your strengths and weaknesses as a software developer. However, are you really confident about your current knowledge, the quality of your deliverables, and the speed at which you work? Can you accurately predict the effort required for your next assignment?

Measuring your work is about becoming more accurate and confident in your estimates. It's about improving the quality of your work. It's about knowing yourself better. When you understand your strengths, you can capitalize on them. When you recognize your weak areas, you can work to improve your skills.

To illustrate these points, and to show how intuitive thinking doesn't help us, I relate two stories from a tutorial I attended on the Personal Software ProcessSM (PSP) given by Watts Humphrey from the Software Engineering Institute. Watts presented an overview of how PSP helps improve personal productivity and increase the quality of our products.

At several points during the tutorial, someone would question the usefulness of a practice Watts proposed. An interesting dialog ensued:

Attendee: I doubt if that practice is really useful.

Watts: Well, my data shows that it is effective and useful.

Attendee: But it doesn't work because ...

Watts: Show me your data to back that up.

At this point, the dialog ended. Watts Humphrey is an engineer. He focuses on data, not theories. He measures things and sees what works. Now, at that time, he gathered much of his data from classroom experiences, but it was data nonetheless. He also knew that none of us in the session had data to back up our feelings. So he "won" these exchanges by default.

Another incident occurred during a break. I had worked in a research and advanced development group in the mid 1980s. This group was unique in part because of its intense focus on quality. We worked hard to achieve zero defects in everything we did. Our practice was to ask the rest of the company to use our completed work. The first person to find each defect was entitled to lunch, courtesy of the software's author.

My first large project was to write a back end for our compiler. I was afraid that when I released it, I would lose my mortgage money buying lunches for people who discovered defects in my code. My fears were unfounded. In the first year, only three defects were


8/23/2004

reported. By the way, that was about average for our group. We tested, reviewed, and tested again before we released something. We had a personal stake in the product.

I thought I'd impress Watts with how well I had done. He'd surely be glad to hear that quality was not a dead issue in the real world. When I proudly told him that only three defects were reported in the first year, he replied: "How many defects did you remove before you released?" What could I say? I had no idea how many defects I had removed. I knew there were a lot. The point of Watts' inquiry was not to tell me I had done a poor job. Rather, he was inviting me to examine why I was able to deliver high-quality software and encouraging me to search for areas in which I could improve.

I returned to my job determined to use metrics to help me improve my work. Since then, whenever I've taken the time to measure my own work, I find that my estimates are more accurate, and I end up delivering higher quality work. When I don't measure, I slip backwards. I think it's like Weight Watchers. When you are on the program, you have a certain goal, and you can track your progress with data that you collect. When you go off the program, you know what you have to do, but you tend to fall into your old bad habits bit by bit.

The bottom line is that if you measure yourself, you can see where you are, where you're slipping, and where you're improving. Measurement is a tool for improving your software self-image.

What should you measure? You can probably think of a lot of things that you do or produce that are candidates for measuring. Watts Humphrey advises that if you measure time, size, and defects, you can derive all of the metrics you need. So, let's talk about each of these from a typical software developer's point of view.

Time Time is pretty simple, isn't it? Just measure how long it takes you to do a task. But what tasks should you measure by the clock, and should you group them? I suggest starting with your development process. If you use PSP, you will probably measure your planning,4 design, coding, testing, and compiling time. There's also a phase for postmortem activities in which you reflect upon your process and adjust it accordingly. If you do code reviews, you might record the amount of time you spend there, too.

If you use a process such as an instance of the Rational Unified Process®, or RUP®, I recommend that you record the time you spend in different workflows and workflow details for the roles you fill. If you use an agile process such as Extreme Programming (XP), you can time your programming, estimating, testing, and so on.

The decision about which groups to assign to your activities is not that important. The categories should make sense to you and the work you do. It is important that you time as accurately as possible and that you are consistent. I have found that measuring time in minutes is the easiest way to record my time data.

Figure 1 illustrates an example of my time data for a project I'm currently working on. The columns represent my planned time, the time I've actually spent so far, and how much time I've spent in each phase of PSP to date on all of my completed projects. Later in this article, I show how I get and present the data.

Figure 1: Time recording

Figure 1 shows no planned or spent time for compiling. I use the Eclipse development platform for my work, and the compilation occurs


8/23/2004

incrementally. So I'm not able to easily measure compile time. That's fine as long as I record the information in a consistent way. Even though I don't count the compile time, I do record defects the compiler finds.

Size Size can be a controversial measurement. How can you measure the size of your software? Do you measure the number of bits in the final product? The number of lines of code (LOC), function points, use case points, or something else? There are so many choices. Again, my advice is to take a simple approach and measure consistently. None of these measures is perfect, and many of them are hard to determine.

I use LOC to measure the size of my work.5 This is fairly simple and reasonably accurate if you have a coding standard and use software to do the counting. You are gathering data for your own use, so you can count however you like. For example, consider Code Sample 1 below. How many lines of code are there? The answer can be anywhere from five to eighteen, depending on how you count. It doesn't matter, as long as you use the same method to count each different piece of code.

Code sample 1: How many lines of code?

An easy way to measure the size is to use a code formatter that organizes the code according to your coding standards, and then use a program that counts the LOC for you. There are many formatters and LOC counters available from the open source community. (Look in http://www.sourceforge.net.) Your IDE may also have some of these features built in.

Once you establish a good counting standard, you still need to decide what to count. There are a lot of choices. Do you count deleted lines? What about modified lines? How easy is it to determine which is which? Then, there's the issue of code that is generated by another tool, such as your IDE. Do you reuse other code? If so, should it be treated the same as code you develop?

There are no easy answers to these questions. The tool I use has a line counter that lets me count new code as well as changes to a code baseline. Figure 2 shows a sample report on new code I wrote, and Figure 3 shows a sample report on code I added to the base.


8/23/2004

Figure 2: Size record for new code

Figure 3: Add-on code measurement

Size can be as important as time for estimating project length. If you keep a record of both, you will find out whether you're better at estimating the time you think it will take or how big you think the project will be. If you are better at estimating size, you can arrive at a good time estimate by looking at your historical data.

Remember that if you use LOC as your metric and you work with different programming languages, your metrics will vary, especially if the languages are dissimilar. For example, the LOC required to write a program in a language such as Scheme is different from the LOC for the same program written in Java or C++. If you use a lot of different programming languages in your work, then I suggest you not use LOC to measure size.

Defects In addition to time and size, you need to measure defects, both where you introduce them and where you remove them. There are many definitions of defect, and people often use other terms, such as fault, error, or bug. Let's agree that when you develop software, you make errors that cause defects to be injected into the software. At some point, defects are found—through some sort of observation—and removed.

Recording defect data serves several purposes. First, it gives you an idea of your defect injection rate, that is, the total number of errors you make that introduce defects into your product. Even if you immediately remove the defect, you still made the error. If you have kept records for a while and know that your defect injection rate is about 30/KLOC6 and your current work is showing only 5/KLOC, you need to figure out why. Maybe you've generated a lot of code, or you've taken quality pills to reduce your errors. Or maybe you've not done a


8/23/2004

lot of testing. But something is responsible for the drastic drop. Until you find out what that is, you probably shouldn't release the product.

If you group your defects by type or category, you can observe a distribution of your most common mistakes. This information can help guide your inspections and reviews. If you know, for example, that when you use loops you have a tendency to be off by one on the loop control counter, you might spend a little more time coding the loops, or at least, reviewing the code before you check it into your production system.

Knowing when you inject the error and when you discover and remove it provides you with valuable information about your testing process. You want to remove defects as soon as possible after they've been injected. A well-known maxim of software engineering is that the longer a defect goes unfixed, the more costly it is to repair. If the majority of your defects are discovered either late in your development process or after you deliver the product, you probably need to revise your process.

PSP suggests a small set of defect types that will cover most of the defects you will encounter. Measuring defects is easy. You identify where an error was injected, where you fixed it, and how long it took to fix. Figure 4 shows the defect injection and removal data for one of my projects. Because I use a test-first approach for much of my development, I injected a significant number of defects during my test phase.

Figure 4: PSP defect data

Analyzing the results Data is just data. You need to work with it to turn it into information. In some cases, you analyze data by consolidating and reporting, as shown in the previous figures. We can gain even more insight by performing statistical analysis on the data we collect.

Statistical analysis can help us determine whether our data is significant and how well different collections of data correlate to each other. For example, is there a relationship between the amount of time we spend on software development and the size of the final result? After completing two projects, we can see an excellent correlation between the two, as shown in Figure 5. However, the correlation is not useful; two data points are insufficient to be statistically significant.


8/23/2004

Figure 5: Data from two projects

After you've measured several projects, you have enough data to help you predict your performance. Figure 6 shows a chart from eighteen of my tasks, comparing the actual time I spent on the project against the time I planned to spend. The value R² is the square of the correlation. If R² is > 0.5 (ideally >= 0.75), then the data is useful for estimating and planning purposes. The closer it is to 1.0, the better the observed correlation is. In this case, the value of R² is 0.86, showing that I'm pretty good at predicting the actual time it will take me to complete a task. Some people are better than others at estimating, and it's taken me some time to get good at it. Experience helps, but having the PSP data to back me up has vastly improved my estimating ability.

Figure 6: Correlation between planned and actual time

You can perform many types of statistical analyses on your data and choose from among many software programs to help you. For example, the analysis in Figure 6 was done using Microsoft Excel. The key is to ensure that you consistently record and enter the data.

Determine what questions you want answered from your data and then select the analysis that is right for you. Watts' book in the Further Reading section of this article provides a good starting point.

Not all projects are equal


8/23/2004

Be careful when you collect data from your software development work. Earlier, I mentioned that using different programming languages can affect the LOC measurement. Another difference can arise because of the type of work you are doing. It takes longer to produce a line of code when you are enhancing software, especially if you are unfamiliar with it. It takes longer to develop software when you are using technology that is unfamiliar to you, or when you're working in a domain that you've never worked in before.

If you record all of your data in one large group, without the ability to sort out similar projects, it will be much less valuable than if you identify a few categories of work that you are likely to do and keep separate data for them. I have different data repositories for my new software development projects, my maintenance and enhancement (of other people's) work, and for Eclipse plug-in work. Each of these affects my overall performance differently.

Final thoughts Recording and analyzing data can be tedious, but tools can help you. A book I co-authored, Software Development for Small Teams, describes one such tool for capturing PSP data. That tool is primitive and not ready for wide distribution. More recently, I have found another tool that meets my requirements; it's extensible and open source. The tool is the Process Dashboard and is available from http://processdash.sourceforge.net/. I recommend it if you want to start measuring your personal process.

Figures 1 through 4 were taken from the Process Dashboard reports. Capturing the raw data is simple. A small Dashboard control panel sits at the top of my computer display, and I simply click to start or stop timing, change projects or activities, or display a defect reporting form. The Dashboard control panel is shown in Figure 7.

Figure 7: Process Dashboard control panel

Process Dashboard offers both ease of use and an extensive set of built-in analyses and reports. It's worth trying it if you want to start a personal measurement program without a lot of pain.

To improve your engineering capabilities, think about capturing your personal data. You don't have to share it with anyone. It's yours, and it's a valuable resource. After you've measured your work for a while, think how differently you'll feel the next time your project manager asks: "How long will it take you to do this task?" You will be able to answer with greater confidence. More important, think how your manager will feel when she realizes that your estimates are pretty good.

Further reading Watts S. Humphrey, A Discipline for Software Engineering. Addison-Wesley, 1995, ISBN 0201546108.

Norman E. Fenton and Shari Lawrence Pfleeger, Software Metrics: A Rigorous and Practical Approach. Brooks Cole, 1998, ISBN 0534954251.

Lloyd Jaisingh, Statistics for the Utterly Confused. McGraw-Hill, 2000, ISBN 0071350055.

Freedman, Pisani, and Purves, Statistics, Third Edition. W.W. Norton & Company, Inc., 1998, ISBN 0393970833.

Pollice, Augustine, Lowe, and Madhur, Software Development for Small Teams: A RUP-Centric Approach. Addison-Wesley, 2003, ISBN 0321199502.

Notes 1 http://www.hpcc.gov/pubs/blue94/section.6.html.

2 IEEE Standards Collection: Software Engineering, IEEE Standard 610.12-1990, IEEE, 1993.

3 Of course, declining defect density might mean that our testers have all gone on vacation. No single metric or indicator is sufficient for most purposes.


8/23/2004

4 Planning for PSP includes the time to develop and refine requirements.

5 Of course, that's only for my software tasks. As I write this article, I'm measuring the number of words, as counted by the software I'm using.

6 KLOC = thousand lines of code.

About the author Gary Pollice is a professor of practice at Worcester Polytechnic Institute, in Worcester, Massachusetts. He teaches software engineering, design, testing, and other computer science courses, and also directs student projects. Before entering the academic world, he spent more than thirty-five years developing various kinds of software, from business applications to compilers and tools. His last industry job was with IBM Rational Software, where he was known as "the RUP Curmudgeon" and was also a member of the original Rational Suite team. He is the primary author of Software Development for Small Teams: A RUP-Centric Approach, published by Addison-Wesley in 2004. He holds a B.A. in mathematics and an M.S. in

computer science.


Comments?





8/23/2004




Book review: Enterprise Patterns and MDABen Lieberman Principal Architect, BioLogic Software Consulting 11 Aug 2004

from The Rational Edge: Business archetypes and pervasive universal concepts occur consistently in multiple business domains, and have a deep history within most business domains. Based on this premise, the authors provide a catalog of business domain archetype patterns to use as part of a model driven architecture (MDA).

Although many problem domains—business domains in particular—have recurring themes and share significant similarities, we who write software for a living often find ourselves "reinventing the wheel." Unfortunately, there are few frameworks to help analysts and architects create new software systems for specific business problems. An early attempt to provide such reusable support was Martin Fowler's analysis patterns; these structures attempted to capture recurring system design solutions and common business domain structures using a standard format based on design patterns. Although these patterns indicated elements and high-level relationships between core business concepts (i.e., purchase orders and inventory), they had several significant drawbacks. First, because of their high abstraction level, anyone who wanted to adapt these patterns required significant domain expertise. Second, the patterns offered no direct way to generate executable code. Together, these two drawbacks were serious enough to limit analysis patterns' use in software development projects.

Arlow and Neustadt take a somewhat different approach in this book by introducing archetypes and archetype patterns. Their basic premise is that there are business archetypes—universal concepts that occur consistently in multiple business domains. These concepts are pervasive, self-evident to experts, and have a deep history within most business domains. Based on this premise, the authors provide a catalog of business domain archetype patterns to use as part of a model-driven architecture (MDA). For example, the first model-driven archetype they identify is a Party. Common to virtually all businesses, a Party represents an identifiable entity that may have legal status, such as a customer, supplier, or client. Parties may enter into PartyRelationships, with specific roles and responsibilities for each Party; for example, there is a PartyRelationship between a buyer and a seller of goods. The authors represent these archetypes with UML class diagrams that contain sufficient detail to be translated directly from the model into code—a primary goal of model-driven development.

To support the diversity found in business environments, the authors also provide mechanisms to easily modify or extend the archetypes and archetype patterns. They describe a "principle of variation," which holds that different businesses require related, but slightly altered, versions of the same domain elements. They divide variations into three extension categories, as follows:

1. Archetype variation. This involves eliminating archetype options (attributes or methods) that are not necessary to the development model.

2. Archetype pattern variation. This refers to including/excluding optional archetypes while maintaining critical dependencies (e.g., a PartyRelationship always requires a Party).

3. Pleomorphism. This refers to the wholesale modification of part of a pattern while retaining the general theme. Usually this means making an abstract concept more specific (e.g., Product becomes UniqueProduct or IdenticalProduct).

Together, these extension mechanisms and the archetype pattern catalog provide a significant advancement in analysts' ability to approach complex, but common, business problems. By studying the examples and techniques the book describes, you can readily tailor the archetype models to the needs of diverse business domains—either by directly translating model elements into code (e.g., one-to-one correspondence to a class), or by mapping the core element into a larger code structure (e.g., many-to-many correspondence). Because the archetype pattern catalog is based on business domains—versus business rules, which are much more variable—the patterns are quite stable and require very little modification to be highly useful.

Contents:Notes



dW Subscription (CDs and downloads)by Jim Arlow and Ila Neustadt

Addison-Wesley, 2004 ISBN: 032111230X Cover price: US$49.95 528 pages

Page 1 of 2Book review: Enterprise Patterns and MDA

8/13/2004


In addition to the catalog, the authors provide a powerful presentation technique that they call literate modeling. This technique presents each part of a complex archetype business model (such as the domain model for inventory management) in an orderly way to improve comprehension of, and accessibility to, the model semantics. It references each model section in a "roadmap" UML diagram, accompanied by descriptive text that explains and demonstrates the use of each pattern archetype. In fact, this is an effective way to describe any UML model, and it represents a good solution to the general problem of how to document software models.

In my opinion, the book has only two weaknesses. First, the concept of business rules gets only a limited discussion in the section describing the Rules archetype pattern. Although the authors state that business rules are highly variable, I would like to have seen a few more examples of complex rule interactions, such as forked, parallel, and chained rules, in addition to the standard Boolean connectors (i.e., and, or, xor, not). Of course, this would have tripled (at least!) the book's size, so it is understandable why the authors chose a simplified approach.

The second weakness is that the book neglects several key business archetypes, such as resource scheduling and rental versus sale of inventory. Adding these would also have significantly increased the book's size, but perhaps we can look forward to seeing these archetype patterns in a revised edition or supplement.

Collectively, the archetype patterns in this book offer a stable foundation for creating a powerful catalog of customized business domain patterns. Just as the publication of Design Patterns1 sparked a revolution and prompted widespread adoption of design patterns, I predict that the availability of detailed archetype patterns for analyzing and defining business domains will engender a similar revolution. I certainly intend to contribute to the ongoing development of this analytical approach in my own work, and I would highly recommend this book to anyone involved in the discovery and/or definition of software domain models.

Notes 1 Erich Gamma et al., Addison Wesley, 1995.

About the author As the Principal Architect for BioLogic Software Consulting, a software architecture consulting firm, Ben Lieberman provides clients with training, mentoring, and practical advice on architectural issues for software systems. He became what he calls a "Consulting Architect" more than five years ago, having arrived at this role via a circuitous route. Prior to that, he had spent nearly ten years as a research scientist involved in numerous software projects relating to biological laboratory research data collection and analysis (e.g., Bioinformatics). He holds a B.S. in Molecular Biology from University of California at Los Angeles, and a Ph.D. in Biophysics and Genetics, from the University of Colorado.


Comments?




Page 2 of 2Book review: Enterprise Patterns and MDA

8/13/2004

© copyright ibm corporation 2004. ... · developerworks > rational > ... and the pressure to...

Documents