SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Architecture Analysis of Evolving Complex Systems of Systems
Technical PresentationSoftware Assurance Symposium
2008Principal Investigator (PI): Dr. Mikael Lindvall, FC-MD
NASA POC: Sally Godfrey, GSFCTeam members:
Chris Ackermann, Dr. Arnab Ray, Lyly Yonkwa, Dharma Ganesan (FC-MD)
William C. Stratton, Deane E. Sibol (APL)
Fraunhofer Center for Experimental Software Engineering Maryland (FC-MD)Fraunhofer Institute for Experimental Software Engineering (IESE)
Johns Hopkins University Applied Physics Laboratory Space Department Ground Applications Group (APL)
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Outline
• Motivation• Background: (static) SAVE• Dynamic SAVE Vision• Dynamic SAVE examples• Applicability Throughout the Life Cycle
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Problem/Approach
• Systems are often difficult to understand– Systems of systems adds to the challenge– Makes system verification difficult– Interfaces often source of problems
• Approach– Architecture analysis focusing on interfaces
• The new tool, Dynamic SAVE, – extends the already existing static Software Architecture
Visualization and Evaluation (SAVE) tool
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Background: The (static) SAVE Tool Software Architecture Visualization and Evaluation
• Does the actual implementation match the planned architecture?– Define a planned architecture– Create an actual architecture from source code– Identify architectural violations through comparison
• Applied to APL’s Common Ground System, Research Infusion project
• Conclusions (Aerospace 2007)– The SAVE approach is useful and practical– One can quickly model, visualize, analyze, find static architecture violations– Good for single software applications– But for systems of systems, some questions remain unanswered…
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Application-Specific Modules
Encapsulation of client/server interface
Encapsulation of socket communications
A Planned Architecture
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
The Actual Application Architecture
Where’s socket implemented?
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Dependency in actual, not in planned
Dependency in planned, not in actual
The Actual Architecture vs. The Planned
But, who does socket communicate with?
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
The Common Ground System
level_zero
indexer
archive_server
Planning &Scheduling
CSCI
SSC POCClients
1:N
ArchivedTelemetry
TelemetryCSCI
spooler
Real-TimeTelemetryPackets
Plotter
eng_dump
Assessment CSCI Telemetry Data Flow Diagram
MOPs/ I&TUsers1:N
ArchivedTelemetry
Level 0 Data FilesAncillary Products
ArchiveServerDirectives
DecommutatedPoints File
Non-Real-TIme Pkt
Files
Non-Real-Time ExtractedTelemetry Packets
Archive ofPkts andIndexes
ArchiveServerDirectives
TelemetryPkt Files
RequestedPoints File
ArchiveServerDirectivesArchived
Telemetry
*Timekeeping System expanded separately
Web Data Server
SortedTelemetry Pkt
Files &Indexes
merger*Timekeeping
System
ArchivedTelemetry
PlotsArchivedTelemetry
ArchiveServerDirectives
LHerrera 08/03
instant_replay
instant_ replayDirectives Telemetry
gap_reporter
Server
Client
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Dyn-SAVE Vision
TelemetryServer
TelemetryClient
Specify PlannedBehavior
Form ActualBehavior
Specify Level of AbstractionFor analysis
Capture DynamicInformation
Compare Planned and ActualBehavior
• Who does socket communicate with?• Is communication according to specification?• Check Sequences, Parameters, Values, Timing
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
DynSAVE in practice
10
The Common Ground System
These systems all have ICDs(Interface Control Documents)
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Focus on:Interface Control Documents
– NASA systems often developed by different teams– Interface Control Documents (ICD) is key, but
• ICDs often interpreted differently because• ICDs implicit, lack important details etc.
– Cause subtle critical deviations from specified behavior • Deviations difficult to detect• Emerging behavior difficult to predict
– Can result in severe problems, e.g. lost data, performance– Need to
• Detect deviations before deployment• (Specify expected and actual behavior before creating ICD!)
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Some Research Questions
• Sequence diagrams– Can we use sequence diagrams to model,
abstract, and detect such deviations? – Can sequence diagrams express what we
need?• Iterative modeling
– Can we start with abstract models, add details as necessary?
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Approach
• Collect concrete examples from APL– Model planned behavior
• Use specification from ICD – Capture actual traces
• Use Archive_Server and Eng_Dump• Generate Client scenarios, observe how Server
responds
• Identify common patterns
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Planned sequence diagram
The “simplest” diagram that describes the planned communication behavior described in the ICD
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
An illegal extra filter is sent after BeginPlayback and Data messages have been sent. The illegal filter is difficult to detect because it is in packet 869.
Example 1: Illegal filter
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
1. Start time must be less than stop time2. Data type of each of the received data messages must match specification
Detailed sequence diagramexperimental notation
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Example 2: Illegal Type specification
STF ordered – STP received.
=STF)
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Adding Timing Constraints
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Checking for Timing Problems
client server
Filter
BeginPlayback
Data
Data
Data
EOT
0s
1.85s
0.35s
0.1s
Data
0.1s
0.1s
0.1s
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
CFDP – A Mission Data System Protocol
• CFDP provides reliable downloads of recorded on-board data
– The implementation is distributed across flight and ground systems
– The protocol runs on top of unreliable CCSDS command and telemetry layer
• At APL, CFDP is mostly automated, but…– Operators turn off CFDP uplink during critical
command load sequences– Operators freeze and thaw timers - pending
transactions don’t time out between contacts
• Improper CFDP operation can lead to unnecessary retransmissions, wasting precious downlink bandwidth
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
DynSAVE monitoring of CFDP
• Monitors macro-level behaviors without affecting flight or ground software
• Detects improper CFDP operation, for example:– timers were not frozen and uplink was disabled on
the ground for an extended period, causing multiple retransmissions when the uplink was finally enabled again
• Detects issues in CFDP implementation, e.g.:– sender continues to send file data after the
transaction has been cancelled• These types of behaviors can go undetected (file
transfers still work) but are important to detect (they can result in data loss!)
Dyn
SA
VE
X
X
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Planned CFDP Sequence
Sample Rules: 1.Check that received FD are not NAKed 2.Check for duplicate FDs 3.Check that we have all FDs upon FIN
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
FileData: 430704-431700FileData: 432698-433694FileData: 433695-434691FileData: 434692-435688FileData: 435689-436685FileData: 436686-437682FileData: 437683-438679FileData: 438680-439676FileData: 439677-440673FileData: 440674-441670FileData: 441671-442667FileData: 442668-443664FileData: 443665-444661FileData: 444662-445658FileData: 445659-446655FileData: 446656-447652FileData: 447653-448649FileData: 448650-449646FileData: 449647-450643FileData: 450644-451640FileData: 451641-452637FileData: 452638-453634FileData: 454632-455628FileData: 455629-456625FileData: 456626-457622FileData: 457623-458619FileData: 458620-459616FileData: 459617-460613FileData: 461611-462607FileData: 463605-464601FileData: 464602-465598FileData: 465599-466595FileData: 466596-467592FileData: 467593-468589FileData: 468590-469586FileData: 470584-471580FileData: 471581-472577FileData: 472578-473574FileData: 473575-474571FileData: 475569-476565FileData: 476566-477562FileData: 477563-478559FileData: 478560-479556FileData: 479557-480553FileData: 480554-481550FileData: 481551-482547FileData: 482548-483544FileData: 483545-484541FileData: 484542-485538FileData: 485539-486535FileData: 486536-487532FileData: 487533-488529FileData: 488530-489526FileData: 489527-490523FileData: 491521-492517FileData: 492518-493514FileData: 493515-494511FileData: 494512-495508FileData: 495509-496505FileData: 498500-499496FileData: 499497-499999EOF: Condition Code=No ErrorACK(EOF): Condition Code=No ErrorNAK: 19940-20937;27916-28913;36889-37886;56829-59820;72781-73778;76769-77766;82751-85742;101694-102691;111664-112661;115652-116649;121634-122631;130607-131604;139580-140577;146559-147556;153538-154535;155532-156529;170487-171484;197406-198403;203388-204385;220337-498500
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Actual CFPD Sequence
Needed FDs: 502Sent FDs: 840Potential Waste: ~70%? – Further analysis needed.
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Zoom in on CFDP sequence
Rule 2 Violation: duplicate FDs!
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Life Cycle Support
SystemArchitecture
Use DynSAVE toSpecify and Test Communication
Add to ICD
Sub-SystemDevelopment
Use DynSAVE toDevelop and Test
based on ICD
SystemIntegration and Test
Use DynSAVE totest based
on ICD
Initial use of Dyn SAVE
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Create System ArchitectureNo Server, No Client ExistUse DynSAVE to• Specify Planned communication
– Sequences– Parameters, Values– Timing constrains
• Create Tests– Correct, Incorrect behavior
• Specific incorrectness• Automatically generate defects
• Ensure that communication protocol can handle all tests
• Add Diagram, Specification, Tests to ICD
• “Generate” information for ICD
Client Server
Communication
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Status
• Dyn-SAVE works for telemetry protocol• Currently adding functionality to evaluate
CFDP protocols• Applying Dyn-SAVE to APL’s systems• We’d like to apply to other systems
SAS_08_ Architecture_Analysis_of_Evolving_Complex_Systems_of_Systems_Lindvall
Summary
• Analyze, Visualize, and Evaluate – structure and behavior using – static and dynamic information– individual systems as well as systems of systems
• Next steps:– Refine software tool support– Use approach to review, improve ICD
• E.g. add planned sequence diagrams, rules to ICD– Apply to other systems to get feedback