obstacle driven development
TRANSCRIPT
09/02/2015 1
Obstacle Driven Development
Obstacle Driven Development is an engineering process designed to incorporate
international standard safety critical engineering processes with the latest software
development methods.
ODD is used for software, hardware and embedded system development and
compatible with safety critical processes such as Safety Integrity Levels. ODD is
currently untested but based upon a combination of engineering and other processes
which are proven and commonly used.
Figure 1: Obstacle Driven Development
Obstacle Driven Development is directed at the problems to be solved and this is
reflected in the name and structure of the process. Obstacle was chosen for the
name as a unit testing process is used throughout and problems to be solved are
divided into four stages: Analysis, Specification, Solution and Production.
Each stage is completed according to an appropriate engineering process and linked
to other stages by creating and passing tests. In a process similar to TDD, each
stage will create a unit test using the previous stage which will then be solved giving
verification and validation.
A TDD process has effectively been extended throughout an entire development
process by ODD. The ODD process has also combined with the structure of a V-
model and extended to give an M-model. These are combined by using the stages of
ODD to give structure with tests used to link adjacent stages.
09/02/2015 2
ODD has been created with an intention of setting a new standard for engineering
through an implicit and comprehensive testing process to give stages which are
separate but linked. Numerous other engineering processes are integrated into ODD
to facilitate the approach including ISO specifications and requirements analysis.
ODD is an engineering process which is designed to be
fully traceable
fully validated and verified
fully tests all levels, from material to product
links all stages of development
The ultimate goal of ODD should be to test everything involved with a development
process to ensure all situations, behaviours, solutions and production are tested and
linked. It is often difficult, impractical or even impossible to test everything and so this
should be considered an ideal form of ODD.
ODD was invented by combining various international automotive safety standards
and processes and inspired by some of the latest principles in hardware and
software engineering.
The processes which have incorporated include
Test Driven Development (TDD)
V-model development
Agile
ISO specifications
Requirements analysis
SOLID principles
ODD is an extension of a TDD process throughout development so each element of
each stage is tested. Through unit testing it is possible to link stages and ensure
identified requirements are met throughout development. Each element in each
stage of an ODD process should each have a test created and solved.
The structure of ODD originated from a V-model for development which has been
extended to include requirements analysis and production stages of development.
The result of extending V-models is the ODD M-model which contains stages of
Analysis, Specification, Solution and Production.
09/02/2015 3
Agile principles are implemented through division of development into smaller
elements. The elements of development are then created through creating, solving
and passing a test. Each element in each stage will be linked to the previous and
next stages through tests.
ISO specifications are used in order to create a development model compatible with
international standards. By using specifications as a basis for tests then a product is
created which is guaranteed as acceptable to the ISO, assuming the tests pass.
Requirements analysis is used to ensure solutions created are those which solve the
identified problems. Use of Safety Integrity Levels ensures requirements are
processed and classified which allows the most important to be identified.
Figure 2: Obstacle Driven Development and M-model
SOLID principles are required for ODD and are implicit throughout. The Single
Responsibility principle is particularly important to ODD and is used for all elements
and tests, without this ODD would not work. Using the SRP it is possible to both
decompose and integrate into desired elements which can be classified into levels of
product hierarchy, such as systems, subsystems and components.
Software has not been created to obtain the full benefits of ODD and therefore
testing the process should begin with small scale developments and be scaled for
larger developments. ODD is scalable and so a smaller project can be integrated to
become a large project.
ODD Stages
Stages of ODD are used to develop a product from a basic understanding of
problems through to creation of a product which fulfils a customers’ requirements.
Tests are used to link stages and provide verification and validation of elements in
each stage.
09/02/2015 4
The ODD process is separated into the following stages and testing processes:
Analysis: Utilisation and Elicitation
Specification: Verification and Validation
Solution: Testing and Design
Production: Quality Assurance and Control
Figure 3: Obstacle Driven Development Stages
Each of these stages and testing processes are used to ensure the correct problems
are identified and a correct solution is created. Each element in each stage is tested
which links the stages to ensure correct problems are solved. Elements are divided
into levels representing a products hierarchy to simplify development.
Using levels of elements allows for effective linking of stages by the creation and
solving of tests. If each test in a stage relates to an element in a previous stage then
solutions will fulfil a developments objectives.
If tests created by a stage are separated into their levels then elements which result
from solving these will also be separated into required levels of product hierarchy.
This improves development by separating development which improves efficiency in
division of labour.
09/02/2015 5
Every stage has clear boundaries to separate development which are then linked
through tests. Tests are used to link each element of development with those related
elements in the previous and next stages. Each element in a stage should be linked
to elements of the same level in the adjacent stages.
Linking of stages is loosely described as follows
Requirements through analysing product situations
Behaviours specified which cover requirements
Solutions designed according to behaviours
Production and quality control of a solution
Analysis is used to investigate situations which a product is expected to cover.
Specification is used to describe behaviours of a product and can be considered
initial design. Solution is where a product is created and can be considered final
design. Production is used to ensure a solution is created effectively and efficiently.
For each stage there is also a checkpoint which are Consolidated Requirements,
Documentation, Prototype and Product. These are useful for ensuring the
development is on track and provide further means for verification and validation.
Each checkpoint should link horizontally to the other checkpoint at the same level in
an M-model. Therefore Consolidated Requirements should link to Prototype and
Documentation should link to Product. Comparing these provides a further check to
ensure a development is proceeding correctly.
To begin a development it is very important to understand the problems to be solved
and therefore requirements analysis is used. An analysis stage investigates
individual situations and processes these into Safety Integrity Levels which are
consolidated into requirements.
A specification is often underused in engineering and is the largest difference in
structure to traditional engineering. An ODD Specification is used as a separate
stage to describe behaviours of a solution and can be verified and validated against
requirements. It is also possible to use behaviours as tests similarly to Behaviour
Driven Development (BDD).
A specification is a complete description of a product up to and including simulations.
It is theorised that a specification enables identification of problems such as
contradictions, errors and omissions in a product before design begins. This is
performed in a similar way to a cross-examination in a court room using unit tests in
the form of questions.
09/02/2015 6
Figure 4: Behaviour Driven Development
The solution stage is used to design a product according to behaviours specified.
This stage is where a product is developed into one which could be sold to
customers. By designing a solution according to unit tests then testability and fault
finding is improved.
Finally the production stage is used to create a solution with sufficient quality and
numbers. This stage is also used to find errors with a solution in terms of
implementation, including if parts required are too expensive or rare.
An ODD process can be repeated continuously to identify any further errors with a
product after production. Individual projects within a larger project can also be
divided into smaller ODD projects.
For example, a car manufacturer creates a product which has components from
suppliers. If the suppler also uses ODD their product can be developed using a
smaller process within a larger ODD process.
ODD Elements
To facilitate a development, an ODD process is divided into elements which make up
various levels of stages and hierarchy. Elements describe individual parts of each
development stage and are required to be linked to appropriate elements of the next
stage.
The term element has been chosen as testing processes used in ODD are implicit
throughout ODD and so element is chosen to describe a result of development in
any stage. A type of element depends on the stage and is also divided into a
hierarchy of product levels.
09/02/2015 7
Figure 5: Obstacle Driven Development Elements
Elements are divided according to development stage and level in a products
hierarchy. A complex product is expected to be composed of systems, subsystems,
components and materials and these, along with a product itself, make up a
hierarchy of elements.
Each element should consist of the smallest part which implement a hierarchical
level of a product to be created. Higher level elements will consist of integrated lower
levels with products made of systems through to components made of materials. An
element is the smallest possible system/component/material which can implement
what is required of it and be described at a product level.
For example, system specification elements should link to those of a solution
system. The links are through the creation and solving of tests and for these
elements the links are through testing and design of a system in a product.
Relative height in an M-model indicates a level of product hierarchy with the highest
indicating a product and lowest for materials. An ascending slope indicates
integration from low levels; a descending slope indicates decomposition from high
levels.
Relative distance between elements in an M-model also indicate how involved a
testing process should be. For example, a product level behaviour contained in a
specification should be very similar to a system requirement, while a system solution
would be more flexible in implementing system behaviours.
As the SRP is used throughout ODD it is possible to both integrate and decompose
levels of development. Depending on the stage it can be useful to integrate or
decompose, and these can also be reversed for fault finding.
Integration is performed for analysis using situations to find requirements and a
solution from materials to systems. Decomposition is performed on a specification or
production to ensure higher levels of product elements are maintained.
When an ODD process is implemented correctly there will be links for each stage
through tests joining each element according to its level. Element are intended to
09/02/2015 8
implement the simplest solutions to problems and higher level elements are
composed of low level elements.
Each element in the ODD process is made up of smaller elements as follows:
Product is a collection of system elements
System is a collection of subsystems
Subsystems is a collection of components, and so on
Low level elements such as materials will be composed of fewer elements than
those of higher levels. Higher level elements such as systems will contain elements
of lower levels and additional elements to achieve its objectives.
If sufficient testing is performed at low levels, such as components, then it may not
be necessary to test them again when combined into higher levels. However this
cannot be assumed and elements may have unexpected reactions to other
elements.
It is often impossible to practically test systems for all expected situations but by
verifying and validating elements from a low level it is easier to predict how reliably a
product will perform. For example, internal states of software can be tested using
functions directly which determine a change in internal state.
ODD Attitude
“Anything that can go wrong, will go wrong.” - Murphy’s law.
A large difference between ODD and traditional engineering processes is one of
attitude. Using ODD all practical attempts should be made to fail a product at the
earliest stage. Of course the intention is not to fail a product but to achieve success.
The purpose of this attitude is to ensure all potential errors are removed as early as
possible. As success is defined as a lack of failure, then it follows that identifying,
correcting and preventing failures is a suitable way to achieve success.
An ODD development is guilty until proven innocent. This means that a product or
development is assumed to be wrong until such time as it is proven right and
reduces the chance of assumptions of success. Once the tests of a development no
longer fail then a successful product has been achieved.
Every element added to a development begins with a red light related to an element
from the previous stage. An amber light is assigned after a test is created and during
the solving of the test, and is only given a green light when the test is passed.
09/02/2015 9
To determine whether a development is correct there will be tests applied. These will
be used to determine and ensure whether an element does what is required for a
development.
It is inevitable that a product with potential for failure will fail and identifying any
potential for problems should be performed at the earliest opportunity. When early in
development errors can be corrected at a relatively cheap cost with any which are
allowed to propagate through stages costing exponentially more to fix.
In order to identify failure, elements of each stage will create tests which the next will
be required to pass before being accepted. Tests will use a unit testing methodology
with a test for each element according to the SRP.
While an extended development process such as ODD will increase costs initially, it
could also result in lower overall costs by preventing errors from propagating through
stages. It is acknowledged that research and development is expensive and Figure 6
from requirements engineering demonstrates this.
Figure 6: Cost of Design Changes in Stages
Figure 6 shows how resources are committed to stages of a development and a
desired practice indicates stronger emphasis must be made on this stage. ODD
allows this by extending a specification and facilitating a unit testing process similar
to a cross examination.
Preventing errors from propagating through a development process is very important
as for each stage an error is missed there is a 10x increase in the cost of fixing the
error. This figure is a rule of thumb but can be considered accurate in demonstrating
the nature of exponentially increasing costs related to errors.
Preventing failure is not a cheap process and so failures prevented must balance or
outweigh cost of development. Potential problems with ODD are that more of the
09/02/2015 10
cost will be upfront, product development more involved and potentially time
consuming.
Errors missed at each stage will result in the following approximations in costs
Analysis error: 1000x more costly to fix at production.
Specification error: 100x more costly to fix at production.
Solution error: 10x more costly to fix at production.
Errors allowed to be continuously produced can result in vast expense such as a
recent fine for Toyota of $1.2bn. This was a result of not responding to errors even
after customers and legislators had made them aware of errors.
ODD Testing
Testing is often given a lower importance than it warrants as without a test there is
no way of knowing whether a product has achieved its objectives. It is important to
test everything that can be tested, especially assumptions, to ensure a development
is proceeding correctly.
Figure 7: Generic Model for ODD Testing
Without a thorough testing process a customer is effectively testing a product.
Obviously this can have disastrous results and result in far greater costs than testing
and identifying errors from the outset.
Testing for product development is often close to the end of a products development
and any errors identified will be costly and difficult to fix. It is proposed that by
09/02/2015 11
making testing implicit for each element then it is quick and effective to identify and
correct errors.
The process for testing using ODD is to create a test and solve it with a solution
being an element in a stage. Testing is often described as being 2x more difficult
than writing code, therefore it follows to create tests first. Creating tests first also
improves testability and allow test suites through the use of unit testing processes.
An ODD process creates new elements by selecting one element from the previous
stage. A test for the new element is created using an element from the previous
stage and solved to ensure this is covered. The new element created by this process
is then used to create further elements in the next stage.
Figure 8: Traffic Light Diagram showing ODD Testing Process
If every element is linked to elements at the same level of adjacent stages then it is
possible to ensure development is focused on requirements. To create an element in
the current stage, an element is selected from the previous and a test created with a
solution being a required element.
The tests are implemented using unit testing with each element and related tests
given traffic light colours depending on its progress.
Red is assigned for an element to be assigned a unit test
Amber is assigned when a test is created
Amber continues through solving a test
Green is assigned when a test is passed
09/02/2015 12
To ensure each element of development is correct, a test is created to ensure the
objective is known and solving the test ensures the element is correct. Linking tests
to related elements of the previous stage ensures the objective is targeted at
creating a correct product.
Tests are created by taking an element of a stage which is rewrote as a question
before devising and creating an appropriate testing process. Tests can be simply a
rewording of an element depending on level and stage.
Complex tests can be created from unit tests by integrating unit tests and can give
an infinite number of test cases created in a bottom-up process. Tests can also be
created by using a top-down process and start with complex tests involving a
number of unit tests before decomposing these into units as required.
Figure 9: Testing Process combined with M-model
Figure 9 shows a testing process combined with the M-model to illustrate how the
traffic light system works and stages are linked. The traffic lights should each
represent an element being linked to the next stage.
It can easily be seen whether bottom-up integration or top-down decomposition is
used from the slope of each stage in the M-model. An ascending slope means
integration while a descending one means decomposition. These can also be
reversed for fault finding and development checks.
Tests for behaviours will compare a behaviour with a related situation and deciding
whether it covers the situation. This is a process which is similar to a cross
examination in a court room as behaviours are compared with situations to identify
errors, contradictions and omissions in described behaviours.
09/02/2015 13
Creating elements according to tests gives improved testability and unless you
design a solution according to your tests, then you may be creating tests according
to your solution. This can lead to unreliable tests and products.
The testing process used in ODD is similar to TDD and this has been extended
throughout an entire development process. Similarly to TDD, the testing must be
implemented in units to ensure testability and use of test suites.
Testing in the ideal form of ODD should be on every element of every stage to link
development and identify errors. Obviously this can be impractical, or even
impossible, so it is important to recognise there will be practical limits to applying
ODD.
The testing process will test individual and collected elements using a traffic light
system in this way. Using elements of each stage to create tests allows linking of
stages after these have been solved.
Creating tests also insures effective communication with suppliers by allowing
communication of exact requirements without problems related to intellectual
property. Suppliers could also be given tests to ensure their product performs as
required for a larger product development.
For example, a system used in a product which is supplied by an outside company
has software which is intellectual property. To verify and validate a system does as
required it may be necessary to reverse engineer software. By creating tests for
suppliers it is possible to ensure a correct solution unambiguously.
ODD Validation and Verification
Validation is most simply expressed by a question “Is it built right?” and verification
by “Is it built in the right way?” Therefore definitions of validation and verification lend
themselves to the creation of tests. Using tests for validation and verification also
allows what is often a subjective process to be quantified into pass or fail.
Tests are used to link stages and also provide both verification and validation.
Verification is achieved through the creation of a test ensuring the objective is
known, while solving the test provides validation and ensures the objective is
completed.
A successful product must be built right and in the right way, therefore ensuring
these conditions are met requires validation and verification.
09/02/2015 14
Separation of stages and related verification and validation processes are as follows:
Analysis: Utilisation and Elicitation
Specification: Verification and Validation
Solution: Testing and Design
Production: Quality assurance and control
Figure 10: Verification and Validation Processes
Adding tests between stages allows a linear process of analysis, specification,
solution and production to become nonlinear. In simple terms a design process
becomes similar to snakes and ladders, with failing tests being snakes and those
passing being ladders.
For each stage the verification process is on the left and is a creation of tests, while
the validation process is on the right and is solving of tests. Creation of tests will
ensure verification of the objective with solving the tests considered validation.
Therefore appropriate verification and validation is performed between each stage.
Each element is linked to the previous stage by using an element to create tests, and
linked to next by being used to create tests. This links both adjacent stages and is
used throughout development.
Creation of tests give potential for a traffic light system with a stop, warning and go
sign, represented by the colours of traffic lights. Each new stage is began with a stop
light meaning a testing process needs to be performed. The amber light is assigned
when a test has been created. Finally a green light is assigned when tests have
been solved.
Through tests each stage can be linked to others and is fully traceable allowing
identification of errors and consequences of any editions to the design or
development. Using a traffic light system, which is an extension of TDD, developers
09/02/2015 15
know immediately whether it is suitable to proceed to the next stage i.e. when a
development stage has all green lights.
Note it is not necessary to have all green lights before moving onto the next stage
but will be described in this way to provide simplification. This point can be
considered similar to differences between waterfall and Agile.
Implementing tests as units gives increased flexibility to testing and improved fault
finding abilities. Tests can be created through integration and decomposition
depending on a process. The slope of the M-model indicates whether integration or
decomposition is performed giving bottom-up or top-down processes.
Tests also provide inflexibility in achieving objectives while increasing flexibility of
solutions. This is because tests are used to determine if an objective has been
achieved while not specifying how. It is desired that a development is inflexible in
achieving its objectives but flexible in how it achieves them.
ODD is not a magic bullet or guarantee of working systems, when implemented
properly it is simply a guarantee that all reasonable efforts have been attempted to
prevent failure. These efforts are continued throughout a products lifecycle and all
relevant tests should be ran for any changes at any stage.
ODD Flowchart
The flowchart shows how ODD combines Test Driven Development processes with
waterfall development. Division of work into units also makes this process
compatible with Agile development. ODD is also compatible with safety critical
requirements analysis and specifications.
Figure 11: Obstacle Driven Development Flowchart
The smaller feedback loops in each stage demonstrate how the process creates and
solves tests. Larger feedback loops are used to enable integration, or
09/02/2015 16
decomposition, of each element in a stage into a checkpoint. The loops are repeated
until all tests are passed for a development stage at an appropriate level.
Feedback between stages is used to decide if implementation of an element is
practical. For example, if a solution is consistently failing tests then a specification
may be modified in order to facilitate more practical solutions.
Figure 12: ODD Flowchart with Feedback between Stages
There are four stages to an ODD process and these are intended to be repeated as
required. The stages are used to first understand requirements, specify behaviours
to cover requirements, design a solution according to behaviours before assuring
quality in production of solutions.
If modification of any stage of a products development is performed then any or all
related tests should be ran to identify any errors which may have resulted. It is
suggested that by using unit tests it is possible to find dependencies and test for
these to avoid running all tests again.
It is not always going to be possible to pass all tests as some elements may be
contradictory to others, such as increasing speed and safety of cars, and therefore a
successful development in ODD could be one which fails the least tests.
The increased cost and effort of testing in ODD is hoped to be mitigated by
decreased risk of failure and improved products. ODD would have larger upfront and
development costs related to development and testing which are balanced by
increased quality of a product from testing.
09/02/2015 17
Note the complexity of ODD is intended to match the complexity of creating a
successful product. While other development processes may look simpler they also
assume success with any failures difficult to locate and fix. Therefore what is
increased effort in the early stages could lead to shorter and cheaper product
development.
There is also opportunity to test assumptions about solutions early in a development
using a specification designed to test behaviours with requirements they cover.
ODD Analysis
"We fail more often because we solve the wrong problem than because we get the
wrong solution to the right problem." -Russell Ackoff.
Requirements analysis is perhaps the most critical stage in development as it
determines the problems to be solved. Without a clear idea of problems to be solved
there can be no guarantee of an effective solution.
Figure 13: ODD Analysis Testing Process and Flowchart
Requirements analysis for ODD incorporates the Automotive Safety Integrity Levels
(ASIL) ratings system defined by the ISO. Each situation is analysed and used to
give ratings for potential probability, severity and controllability of an event.
Multiplying these together then gives an ASIL rating for the level of hazard which an
event may cause.
A set of requirements can be created for a product by taking each of the highest
rated ASILs from those found. For a successful product it is necessary to fulfil
requirements made upon it. Unfortunately a product designer’s requirements can
09/02/2015 18
also change at any time meaning a process which reacts to these is important, which
ODD is designed to allow.
Engineering requirements are used directly in most engineering V-models without a
specification. ASIL ratings show this to be a missed opportunity because there are 3
potential behaviours to reduce hazards associated with any event. Behaviours which
reduce either probability or severity or those which increase controllability are all
possible to fulfil requirements.
Behaviours which improve either probability, severity or controllability (or any
combination) are all applicable to fulfil a requirement. Therefore behaviours may
cover requirements by improving any of these properties and therefore using a
specification gives increased flexibility.
Using requirements directly for a design process limits abilities of designers to adapt
to problems and devise new solutions. For this reason a specification, or interface
between problem and solution domain, has been extended to become a complete list
of behaviours with associated validation and verification processes.
It is proposed that a decision tree, which is a modification of a probability tree, can
be used to model an infinite number of situations while also being combined with
ASIL ratings to produce requirements. Combining these gives safety critical
requirements for an infinite number of situations.
Figure 14: Example of a Decision Tree with ASIL Ratings
The ASIL ratings and process are simply combined into a probability tree by
multiplying them. This can be used to investigate any number of events or conditions
by using a decision tree to model them and could cover all code branches for
software.
09/02/2015 19
If a decision tree is implemented comprehensively then all expected situations for a
product can be found through investigating branches of a decision tree. A situation is
defined using a branch in a decision tree and an ASIL rating found from multiplying
along the branch.
Using a decision tree is very useful for investigating code branches of software as a
decision tree can be modified to match code branches of a product. To find ASIL
ratings for each situation a decision tree is used in the same way as a probability
tree and you simply multiply along a required branch.
ODD Specification
Specifications are an important and often underused aspect of development. By
using a specification as a separate stage used to specify behaviours and can help
quickly and easily identify errors, contradictions and omissions.
Figure 15: ODD Specification Testing Process and Flowchart
Creation of a specification is very important to ODD and facilitates a testing process
similar to TDD which is called Behaviour Driven Development (BDD). The
development of ODD originated with the question regarding TDD which was, “where
do the tests come from?”
Given a development process is named BDD then it follows to use behaviours in a
specification as a basis to design an ODD Solution in the next stage. Linking
behaviours to situations also allows for testing to ensure every expected situation is
covered.
09/02/2015 20
A traditional engineering process has problem and solution domains with a
specification as the overlap between these. When a V-model was compared to
problem and solution domains it was found that a V-model does not extend fully into
a problem domain.
Figure 16: V-model compared with Problem and Solution Domains
It can be observed from Figure 16 that a V-model does not effectively cover an entire
development process because it does not extend throughout the problem domain. A
further problem is the processes for requirements analysis and solution designs are
performed with different processes.
Overlaying a V-model with a requirements analysis model showed a specification
could be used more effectively and extended into a separate stage. Developing this
stage resulted in an N-model which has since been extended into the ODD M-model.
From this observation it was decided that a V-model for automotive development and
models for requirements engineering could be improved through a combination of
both.
Given specifications are an interface between problem and solution domain then
ODD process which extends a specification also extends the interface. Therefore a
specification could be used to define behaviours which satisfy requirements and also
create tests to design a solution.
Creating a list of all behaviours required of a product means engineers will know
exactly what a product is required to do and enables errors, contradictions and
omissions to be identified and corrected. The list of behaviours can also be
converted into documentation for a user.
A description of behaviours can include computer simulations of expected behaviour.
Using these simulations potential customers can determine if a product fulfils their
requirements before any work on a solution is performed.
09/02/2015 21
A customer generally cares about what a product does, not how it does it. Therefore
when using descriptions of a product, a customer can be informed of what a product
is expected to do without requiring any knowledge of how it works.
The ODD process has used an extended specification to achieve the following goals:
integrate safety critical requirements analysis
incorporate validation and verification tests
use of behaviours to generate design tests for the solution
allow decomposition of product requirements to component behaviours
Using a specification also allows for a top down process to defining behaviours
which ensures that higher level behaviours are preserved. This is an improvement
over TDD as tests will often begin at low levels and be integrated.
Figure 17: Extended Specification with Testing Processes
With a specification as a separate stage, the problem domain and solution are now
fully separated and allows testing between stages. Testing is appropriate to what
links the stages, and between analysis and specification will consist of a cross
examination of the situations and behaviours. Tests between a specification and
solution will be similar to BDD.
To build a specification it is necessary to form a basic model of a product which
describe input and output of systems, subsystems, components etc. Creating a
model of I/O allows for investigation of information flow and ensures a model is
complete with no omissions.
09/02/2015 22
Further work on a specification is used to build a basic model into one which
contains expectations of behaviours of systems and components etc. Once
expectations are completed these can be turned into tests and work on a solution
can begin.
Design tests generated by a specification stage are tests to which a solution is
designed and must conform. Tests are generated beginning with behaviours of the
input and output of systems unto which detail is then built up until expected
behaviours are generated to which the designed solution must conform.
Figure 18: Creation of an ODD Specification
Creating an ODD Specification allows its reuse and allow plans for future
developments in tandem. For example, behaviours may be described which are to
be attempted as a solution in a future development. This would ensure compatibility
while also reducing costs of creating a specification.
For companies which sell several similar products they could compare specifications
of their products to identify areas of similarity. Once similarities are identified this can
help increase reusability of solutions.
Various companies offer software which applies this part of the ODD process and
demonstrates the approach is applicable. Companies offer this service through
software which creates C/C++ code compliant with ISO defined behaviours through
unit tests.
ODD Solution
An ODD solution is a collection of designs for various product levels created and
assembled to allow required behaviours to be produced. A solution is designed
according to tests generated by behaviours in a specification.
09/02/2015 23
Solution has been chosen over design for the name of the stage as design is treated
as a verb rather than a noun. In ODD you design a solution rather than solve a
design.
Design is a process to create each individual element of the solution which are
integrated into a combined solution. A solution will consist of elements from materials
used which are tested and integrated to become a prototype.
Figure 19: ODD Solution Testing Process and Flowchart
A solution is created in a similar process to Behaviour Driven Development using
behaviours described in a specification. Expected behaviours of a product are used
to create tests which a solution is then designed for. Designing a product according
to a specification ensures a solution performs all desired behaviours.
Beginning with designs at material level which are integrated into components as a
solution is designed in a bottom-up process. By using a specification to create tests,
which allows top-down process, then higher level behaviours of the product can be
maintained while integrating from low levels.
The stage finishes with a prototype which is compared to requirements to verify and
validate a product performs as required. A prototype is all elements of a solution
combined and assembled which should represent a final product, although changes
could still be made.
An ODD solution is designed according to tests generated by a specification stage
therefore guaranteeing a fully tested solution corresponding to agreed behaviours.
09/02/2015 24
Various companies already offer software which use relevant specifications, such as
ISO, to give a customer readymade unit tests. Using software such as this allows
time to be saved from creating tests.
There is potential for anyone creating software using ODD to distribute tests so
users may create or modify the product while still maintaining the behaviours
required for compatibility and legislation. This would be especially useful for software
and potentially allow a whole community to work on software independently.
Tests created for a production stage will consist of quality assurance tests for
verification and quality control for validation. These are to be performed for every
element in a solution. Remember that once a solution is designed it still needs to be
produced which is often the most difficult part, especially when dealing with mass
production.
ODD Production
Production is where a solution to problems is manufactured and assembled
according to quality assurance and control. Production often consists of a great
many products produced by numerous companies in different locations all
contributing to a single production line at different times i.e. it is very complicated.
For this reason a production stage has not been investigated in great detail due to
having the most complicated and varied processes of any stage required for a
development.
Figure 20: ODD Production Testing Process and Flowchart
09/02/2015 25
As ODD is intended to be a generic process used for hardware, software and
embedded, the possible range of products is far larger than can be described in this
report. Therefore there is no great detail of the production stage in this report and
scope is limited to creating quality assurance tests and passing these for quality
control.
A production stage in ODD is linked to the solution and analysis stages with
appropriate tests to link these. A product is created using ODD by decomposing a
solution into product elements so it can be produced effectively and efficiently.
Tests related to a production stage could be extended beyond quality assurance and
control to include production problems such as delivery times, cost of materials and
labour etc. The process to do this would further extend ODD and improve
capabilities.
Figure 21: Production Stage and Links to Adjacent Stages
Tests generated by a production stage will be related to utilisation of a product as
verification and elicitation from customers as validation of a product. Production is
often the most potentially dangerous and costly period of development and only
when utilisation and elicitation tests are successful can a product be considered
successful.
Once a product is created development should not stop and depending on the type
of product this is actually the most critical stage. Any problems which have not been
identified will now be found by a customer. For this reason an ODD process should
continue to ensure a customer’s problems are solved and provide a framework for
developing future products.
If a safety critical product is created then any problems which have slipped through a
development process will have potential to cause harm to a customer. When a
customer is harmed there is likely to be bad publicity and legal action. Note the costs
of these are increasing all the time and so cost of failure continues to increase.
By continuously analysing a products performance then errors such as single points
of failure can be quickly identified and corrected. Requirements for future products
09/02/2015 26
can also be determined using this stage and an ODD process can be ran
continuously.
Following from production is analysis as the process is designed to repeat and
identify any errors which were missed. To link production to analysis is more difficult
as customers will provide this step and therefore it will not be performed as efficiently
as would be possible with company employees and suppliers.
Once a product is utilised then utilisation tests created are used to ensure
communication with a customer. When these are completed then elicitation has been
suitably performed. Note elicitation is performed to obtain customer opinions on a
product and potential improvements rather than gain approval for it.
ODD Example: Torque Vectoring
While ODD has been developed as a product development process it has a wider
range of practical uses. The original motivation for creating ODD was to deal with
complexities involved in creating a torque vectoring system for a vehicle.
Figure 22: ODD M-model with Modifications for Automotive Control
Torque vectoring is an automotive technology where varying amounts of force are
applied to each wheel to produce turning moments. The turning moments generated
are used to increase cornering ability and improve stability. Various methods and
systems are used such as applying brake torque or biasing torque split to a
differential.
Applying concepts of ODD to automotive torque vectoring
Analysis: varying torque applied to wheels improves cornering abilities.
Specification: vary torque to wheels to improve cornering.
Solution: apply brake torque to individual wheels to improve cornering.
09/02/2015 27
Production: use brake pressure valves to alter applied torque to wheels.
Note the example is simply a basis used to illustrate the concept of linking all stages
of a product development in a unit basis. Using ODD a torque vectoring system
designer would have to find a behaviour, solution and production for each individual
situation, of which there are many. Once a situation is identified by a vehicle
dynamics controller then behaviours can be selected to fulfil requirements identified.
A solution is calculated to implement behaviours for each expected individual
situation. Finally production of a solution is according to control of torque vectoring
components, such as valves of a braking system.
Behaviours and solutions are selected depending on conditions e.g. a braking
system is used when decelerating, and an engine used when accelerating. Torque
vectoring can be used to increase cornering abilities or increase stability by applying
a counter turning moment.
When multiple situations are considered such as accelerating and decelerating, left
and right turns, control and prevention of skids and improving maximum cornering
behaviours then it becomes clear how complex an automotive control system is. All
behaviours and solutions are also affected by environmental conditions, especially at
extremes of hold and cold temperatures.
Figure 23: ODD M-model Adapted for a Torque Vectoring System
Examples of elements used for a braking system in a vehicle are as follows. Note for
a vehicle there are many more materials, components and systems involved.
Material: Brake pads and tyre produce sufficient friction
Component: Brakes are applied effectively and tyres are durable
Subsystem: Brakes lines are split to create redundancy
09/02/2015 28
System: Brake system controls pressure to ensure effective braking
Vehicle: Brake system responds to driver input and decelerates
An intention of ODD is to produce solutions for behaviours to cover individual
situations. Situations can be identified from analysing and combining individual
components of each potential situation encountered. Behaviours are specified to
cover each expected situation. Solutions are designed to implement all behaviours in
the specification. Production is everything involved in creation of solutions according
to quality assurance and control.
Summary
The ODD process and structure has been demonstrated in this report and has
shown to be an innovative process based on current engineering processes. New
models and processes are introduced with explanations of their use and abilities.
While ODD is a new engineering process it uses traditional engineering except the
order is changed to create tests first and a unit testing process is implemented
throughout development. Thus ODD is not about reinventing the wheel, it is about
making a wheel turn more efficiently.
ODD is implemented in four stages which each have related testing processes for
verification and validation. ODD Stages are Analysis, Specification, Solution and
Production with the appropriate testing processes used between each. Using these
stages with suitable verification and validation give the most comprehensive testing
process.
Numerous original models have been created to demonstrate the process including
a torque vectoring model to demonstrate the control uses of ODD. The models
demonstrate the various stages and how they are created and linked with flowcharts
and other models.
Acknowledgements
The ideas which formed the basis of Obstacle Driven Development are from a wide
range of sources and a paper with full referencing is to follow. The author is not
considered an expert on the processes which support ODD and so any input is
welcomed.
All processes on which ODD is based are popular and have been developed
elsewhere. There is not much “new” about ODD except for how processes are put
together and presented. The largest differences are the ideas to create tests first and
create an extended specification.
09/02/2015 29
The author considers that James Grenning’s work in both Test Driven Development
and Agile has played the largest inspiration in creating ODD but there are many
more.
The author would also like to thank anyone who helped or inspired the creation of
Obstacle Driven Development and odd.enterprises.
Further Information
Please see the odd.enterprises website and Facebook pages for more information.
Presentations on ODD can be found here.
odd.enterprises is currently the only company to offer Obstacle Driven Development
solutions such as consultancy and education.
If you’d like to provide feedback or ask any questions please email Jonathan Herring
Figure 24: M-model of Services Offered by odd.enterprises
Use of ODD Materials
Obstacle Driven Development is not a patented idea and is free to be used, with
rights reserved for copyright of diagrams and logos. Use of diagrams is free for any
purpose other than marketing. Logos and the name odd.enterprises are for use only
by odd.enterprises.
Please contact us at odd.enterprises if you would like to use ODD materials for
marketing of products or services.
09/02/2015 31
Appendix 2: Obstacle Driven Development M-model with related stages and
verification and validation
09/02/2015 32
Appendix 3: Obstacle Driven Development M-model showing stages and
elements
09/02/2015 33
Appendix 4: ODD Model in
a flowchart form with all
testing processes
09/02/2015 34
Appendix 5: ODD
Model in a flowchart
including feedback
between stages