eosp summer 2011

Project Overview Demonstration Operations and Processes Planning and Adjusting Risks Quality Measures System Architecture Tools Overview Lessons Learned Client Feedback

Team HandSimDroid 2

Team HandSimDroid 3

Development/ Support Manager

Team Lead Planning Manager

Process/Quality Manager

Mentors

Team HandSimDroid 4

Bosch RTC

UC Berkeley

• Elizabeth Latronico • Charles Shelton

• Christopher Brooks • Edward Lee

Bosch Research & Technology Center (Client)

Bosch uses an open-source tool called Ptolemy for modeling, simulation, and design of concurrent, real-time, embedded systems.

Android application that can run simulations of Ptolemy models on handheld devices.

Team HandSimDroid 5

Team HandSimDroid 6

• Show simulations running on the handheld

• Enable UI customization by model and per user

• Create demos that showcase usefulness of functionality to engineers

Act as a proof of concept for ASCET tool ◦ Inspire innovation at Bosch

Freely extend open source software

Improve operations & reduce cost of calibration ◦ Running simulation on the handheld on the go

◦ Customize UI for different purposes & users

Team HandSimDroid 7


Team HandSimDroid 8

Team HandSimDroid 9 http://bitURL.net/handsimdroid

http://bitURL.net/handsimdroid

http://bitURL.net/handsimdroid


Team HandSimDroid 10

Show simulations of three Ptolemy models on android.

Enable UI customization by model and per user.

Strict adherence to the coding, styling, and commenting standards set forth by the Ptolemy Project

Complete all must-have requirements

Adhere to (and adapt) the architecture


Priority Completed Total

Must Have 21 21

Good To Have 11 16

Nicety 1 5


What? Why?

Define integration points Difficult to tell when components could be integrated together

Update workbooks every Tues/Thurs/Sat

Weekly updates were no longer sufficient with 192 hr weeks

Provide weekly engineer reports Communication breakdown with task status

Calculate correlations between data to better estimates

Universal fudge factor of 30% was no longer accurate because of varied types of tasks

Team member who closes most bugs get free lunch

New features took precedence over addressing defects

Team member who reports most bugs gets free ice cream

Wanted to ensure that corner-cases were addressed


Hours get lost in the context switching, during breaks and incorrect tracking ◦ Measure the gap and allocate it as buffer

Group your tasks into sprints for easier tracking ◦ Helps in keeping the big picture in mind without being

bogged down with details

Improve and tailor your processes continuously, use postmortems to add PIPs to the process.



Release 1 Release 2 Release 3 Release 4

2011 Summer

May June July Aug.

Cycle 5 Cycle 6 Cycle 7 Cycle 8

ptdroid

Tool Setup

Homer

ptserver

Documentation

Bug fixing

0

500

1000

1500

2000

2500

1/19/2011

2/2/2011

2/16/2011

3/2/2011

3/16/2011

3/30/2011

4/13/2011

4/27/2011

5/11/2011

5/25/2011

6/8/2011

6/22/2011

7/6/2011

7/20/2011

8/3/2011

Actual Hours

Earned Value

Planned Value


Continuous problems: - Logging time (4 PIPS) and TSP tool - Only task hours, no context

switching - Some tasks not logged

PIP 8 PIP 15

PIP 17

PIP 19

Snapshot: 8/3

813.81

47%

565.62

33%

118.13

7%

3%

3% 2%

1% 1% 1% 1% 1% 0% CODE

MGMT

DOC

PLAN

IT

DLD

CODEINSP

PM

REQ

HLDINSP

DLDINSP

REQINSP

Was 42% during Spring. Less meetings overall as cycles became longer.


y = 1.26x - 0.21

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

0.0 2.0 4.0 6.0 8.0 10.0 12.0 14.0 16.0 18.0 20.0

Actu

al ti

me (hours

)

Estimated time(hours)

Overestimated Tasks

(Visualization interface)

Underestimated Tasks

(UI tasks, Homer infrastructure)

Team HandSimDroid 19 Correlation: 84.51%

Challenges Possible solutions

Continuous tracking of time data was lacking and imprecise. Continuous effort is required.

• Tools help, but don’t really solve the problem currently. Context-aware applications could be a big help for developers.

• Team members being more disciplined. This didn’t work quite well for us.

Task volatility and tracking, especially since the TSP tool is not collaborative.

• Standups helped coordinate and mark tasks. • TSP workbooks are hard to be updated during

these meetings. Centralized customized workbooks or other management tool (e.g.: Jira)

Task boundaries and exit criteria were not defined well, due to unknowns or uncertainties.

• Spend more time defining the tasks. • Facilitate more frequent communication between

the members.


Risk Mitigation

Team had trouble in integrating modules; schedule and quality might be affected

• Design Reviews focusing on integration points

• Continuous Integration

Team is becoming more relaxed as functionality is implemented; Schedule might be affected - not enough time to focus on quality

• Baseline modules • Allocate last cycle for quality

activities • Friendly competition on bug

bashing

Team developed some code without following Ptolemy style guidelines; Berkeley team might become alienated; project does not have impact

• Checkstyle & continuous integration for publicly tracking style issues


0

5

10

15

20

25

30

35

Ptdroid Ptserver Homer General

Normal

Minor

Major

Enhancement

Critical

Blocker


Injection Phase Detection Phase

35

56%

27

44%

Design Code

37

60%

20

32%

4

6% 1

2%

Test Code Review Design Review Code


34.12

4%

860.11

96%

Time Spent

Review Code/Test


24

39%

38

61%

Bugs Found

Review Code/Test

Name

ptserver.actor 51% 42/82 67% 115/172

ptserver.communication 80% 60/75 87% 291/336

ptserver.control 42% 45/108 69% 231/335

ptserver.data 49% 35/72 73% 134/183

ptserver.data.handler 97% 58/60 95% 186/195

ptserver.util 43% 73/169 46% 149/325

Conditionals Lines


• Total LOC: 4235

• Total Comments: 4279

• Defects: 33 ptserver

• Total LOC: 5810


• Defects: 23 ptdroid

• Total LOC: 3624


• Defects: 7 homer


Defining a fault model would have resolved several design defects (state & concurrency).

Needed to realign development goals with stakeholder goals when working on Homer (functionality over quality).

Continuous integration was extremely helpful in getting a quality snapshot at any point.



1

2

3 4

QA Priority Difficulty Scenario

Performance High High Latency between capturing sensor data from a source and processing it on server is no more than 2 seconds per source Latency between completion of the data processing on server and piping it into a sink is no more than 2 seconds per sink

Extensibility High High A new graphical actor or source sensor is incorporated in UI designer and operational in the Android application within 2 person/weeks

Reliability High High In case of network issues, simulation errors or runtime exceptions, system must display error message, reset to default state and log the error and gracefully end simulation on the client and server.

Usability Medium Medium The UI of a model looks conceptually the same as it was shown in the desktop preview.

32 Team HandSimDroid

Dynamic Perspective

Proxy ModelInfrastructure

Legend

Object

Android Handheld

PtolemyServer

Thread

MQTT Broker

A B

A sends attributechange token to B

A B

A sends aremotetoken to B

A B

A sends an MQTT message to B

A B

A sends a token to Busing Ptolemy filter connector

Process

Ptolemy SimulationEngine

Multiplicity

Proxy Sinks

ActorsProxy

Sources

Blocking Queues

Token Listener

Token Publisher

Token Batch

Monitoring System

A B

A blocks B's thread

Ptolemy Server Hessian Servlet

A B

A sends synchronous command to B (http)

Proxy Model Infrastructure

SinksSources

Blocking Queues

Token Listener

Token BatchMonitoring

System

Proxy Sinks

Proxy Sources

Proxy Attributes

Attribute Change Listener

Attribute Change Listener

Token Publisher

New Component

Existing Component

Modified Component


Sources & Sinks must run on handheld

Reliability: Ping / Echo

Performance: Heavy Processing

on Server

Performance: Thread throttling

1 2

3

4

5

6

7

8

Mirrored Design

Architecture was largely detailed enough to be implemented as designed ◦ Minor issue: Didn’t design how to inject proxy sinks and sources

on the client side

Runtime view was critical for understanding the system and reasoning about it ◦ Code (static view) is sometimes tightly coupled for non-obvious

reasons Thread synchronization and locking Hard to understand this without runtime view

◦ Had an edge case that threatened implementation of our design Bosch and Berkeley team helped us overcome it with the help of runtime

perspective diagrams Issue: Type inference system was not designed for distributed

environment


Homer (UI Designer) Static Perspective

35

Actors depend directly on Desktop Java [Lattix]

Extensibility: Abstract Factory & dependency injection - Easy to add new Actor for Android

Usability – one to one mapping from preview to the UI

Generally worked out quite well ◦ Detail level was high enough since complexity was lower

Good decision: Netbeans Visual Library ◦ Designed WYSIWYG layout editor in 3+ weeks ◦ A lot more usable that what Ptolemy currently has ◦ Saved a lot of time

Not so good decision: Google Guice dependency injection framework ◦ Guice’s license was compatible with Ptolemy’s

Berkeley team asked us to remove it because of potential complication with licensing issues

Core Ptolemy had to depend on Guice

◦ Relatively easy fix for us because only small portion of Guice was used

◦ Reflection: keep an eye on licensing issues and if functionality is simple enough, implement it yourself even if this impacts maintainability


Tool Name

Continuous Integration Jenkins

Coding Style Checkstyle

Code Reviews Rietveld

Static Analysis FindBugs

Unit Testing JUnit

Test Coverage Cobertura

Bug Tracking Bugzilla


Used Jenkins (Hudson) ◦ Build system for 4 trees ptserver & Homer

Trunk Actor porting branch

ptdroid Trunk Actor porting branch

Rationale: stable code within trunk and code breaking core Ptolemy in the branch ◦ Merge the branch once it’s stable ◦ Risk mitigation: Didn’t want to break open source project

with our changes and impact release schedule of Berkeley team

◦ Increased configuration management complexity


less

than 20

min.

55%

20 -

60

min.

10%

60 - 120

min.

7%

120- 240

min.

7%

more

than 240

min.

21%

Duration with a broken

build (in minutes)

less than 20 min. 20 - 60 min.

60 - 120 min. 120- 240 min.

more than 240 min.

less

than 20

min.

47%

20 - 60

min.

22%

60 - 120

min.

3%

120- 240

min.

5%

more

than

240

min.

23%

Duration with failing tests

(in minutes)

less than 20 min. 20 - 60 min.

60 - 120 min. 120- 240 min.

more than 240 min.


Time to setup: ~10 hours * Based on 690 builds

JUnit tests and Cobertura for test coverage metrics ◦ Most tests were more system tests rather than unit

tests focusing on individual functionality

End to end client server simulation

Hard to write unit tests for multi threaded systems running in distributed environment

◦ Jenkins often complained about test breakage

Reason: concurrency & timing issues that were not encountered during development

Continuous integration helped catch those problems


Checkstyle ◦ Check the code base for conformance with naming and style

conventions ◦ Very important for Berkeley team ◦ Initially Berkeley team reviewed our every commit Hard to keep up with reviews while the module is in the development

◦ With Checkstyle integration we had a public venue to track and monitor style issues

FindBugs ◦ Static analysis tool ◦ Runs slowly on developer machine ◦ Jenkins allowed us to track FindBugs trends continuously and

know which commit introduced potential issues ◦ Issues were fixed organically before development on a feature was

complete – no need to file a bug


Synchronous Reviews with Bosch and Berkeley ◦ Mostly focused on knowledge sharing and

understandability of the code and comments ◦ Did not identify major bugs

Asynchronous Reviews within the team ◦ Rietveld tool Submit a change set to the website and ask team

mates to review it asynchronously

Inline comments

Saves time – no need to review code that was not changed


Don’t treat every task that we can earn value on ◦ Timeboxed tasks don’t give clear indication of

progress

Granularity of the tasks inhibited ability to see big picture

Architectural experiments and granularity of the architecture was helpful in identify issues and communicating them to stakeholders


Tailor TSP to be more agile with sprints to realize short term goals and make adjustments

Jenkins was helpful in managing technical stakeholder expectations since it acted as continuous deployment tool

Comments were useful in understanding the project given the size (350 KLOC) and we appreciated the importance of comments


“This was a very challenging project and the team did a great job meeting and exceeding my expectations.”

“I was very impressed at how quickly they were able to produce code and make changes and improvements weekly based on our feedback. This was one of the best MSE teams I've had the opportunity to work with.”


“The team did quite well meeting the functional requirements. The Ptolemy II code base is about 350K lines of non-commented code. The team needed to insert functionality at locations fairly close to the kernel, which requires understanding a large system and carefully inserting new functionality.”

“Overall, the project went quite well, the HandSimDroid team was very productive.”


“Great job all around! Very high professionalism. Nicely done!”

“It contributed immensely to our research objectives ”


Thanks!


A.k.a. slides to go into more details.


PIP Problem Suggestion

8 The team is not reviewing the metrics being collected to make better estimations and do better planning/tracking

Send out updated copy of handsimdroid stats on the weekly basis and add as agenda item with highlights what to look at in the stats - Planning manager would do that day before team meeting (24 hours)

15 With the larger number of hours, its more difficult to track where we are in the project and whether or not we will meet/miss our milestones

Individual workbooks will be updated every Tuesday, Thursday, and Saturday evenings

17 The team is having trouble tracking the members' involvement.

Upload timesheets each day after standup meetings (TL makes sure), and consolidate and validate the data right after (PM).

19 Workbooks are not updated at the end of every weekday common time

Team member in violation will pay $1 for each day that their workbook is out-of-date


0

100

200

300

400

500

600

700

800

Monday Tuesday Wednesday Thursday Friday Saturday Sunday

Earn

ed V

alu

e


Working odd hours: - Common times from 5pm-8pm - Part-timer mostly after 5pm, Friday,

and weekend - All meetings on Fridays and

weekends

0

5

10

15

20

25

30

35

40

45

50

8/29/2011 9/29/2011 10/29/2011 11/29/2011

Hou

rs

UPDATING PTOLEMY BUGZILLA

UPLOAD DOCUMENTS TO DOGBERT

UPDATING DOCUMENTATIONS

POSTER PRESENTATION

POSTMORTEM

HANDLING CHANGE REQUESTS

TESTING AND MAINTANANCE

REFLECTION PAPER


Total 632 hours

0

5

10

15

20

25

30

35

Ptdroid Ptserver Homer General

RESOLVED

IN_PROGRESS

CONFIRMED


0

2

4

6

8

10

12

14

Blocker Critical Major Normal Minor Enhancement

Design

Code


FindBugs #

BC_UNCONFIRMED_CAST 1

DC_DOUBLECHECK 1

DLS_DEAD_LOCAL_STORE 1

DMI_HARDCODED_ABSOLUTE_FILENAME 1

IS2_INCONSISTENT_SYNC 2

NP_NULL_ON_SOME_PATH_FROM_RETURN_VALUE 4

OBL_UNSATISFIED_OBLIGATION 1

SE_BAD_FIELD 2

SE_NO_SERIALVERSIONID 2

SP_SPIN_ON_FIELD 1

ST_WRITE_TO_STATIC_FROM_INSTANCE_METHOD 1

Total 17


http://msesrv5h-vm.mse.cs.cmu.edu:8080/job/ptserver/135/findbugsResult/type.-1801791658/

http://msesrv5h-vm.mse.cs.cmu.edu:8080/job/ptserver/135/findbugsResult/type.1715099639/










37

60%

20

32%

4

6%

1

2%

Test

Code Review

Design Review

Code


Inspections: 3.8% time Yield: 39% bugs


Before After

Inline comment

The team had trouble integrating modules due to everybody having a big picture; Lots of time is wasted integrating; code does not meet quality standards.

◦ Mitigation: Design reviews focusing on integration points, Checkstyle and

Cobertura for tracking quality, Bugzilla for assigning tasks focusing on quality issues (i.e. write test case)

We becoming more relaxed as we get the most functionality done; this might affect our schedule since we might not have enough time to close bugs or meet quality metrics.

◦ Mitigation: Baseline the code (implement all high and medium priority

requirements), allocate last cycle to focus on quality, friendly competition on bug bashing – free lunch and ice cream for winners

Team started developing code without following proper Ptolemy guidelines; Berkeley team might become alienated and/or we might not get needed help from them, project might not have impact.

◦ Mitigation: Integrate Checkstyle with cont. integration and publicly track it


0

20

40

60

80

100

1/19/2011

1/26/2011

2/2/2011

2/9/2011

2/16/2011

2/23/2011

3/2/2011

3/9/2011

3/16/2011

3/23/2011

3/30/2011

4/6/2011

4/13/2011

4/20/2011

4/27/2011

5/4/2011

5/11/2011

5/18/2011

5/25/2011

6/1/2011

6/8/2011

6/15/2011

6/22/2011

6/29/2011

7/6/2011

7/13/2011

7/20/2011

7/27/2011

8/3/2011

Perc

enta

ge

Normalized Planned EV Normalized Actual EV


Architecture compliance review (6/27)

Release 4: Documentations

MOSP

SRS

Architecture docs

Release 2: ptdroid

Release 3: Homer

Release 1: ptserver

EOSP

Snapshot: 8/3

Bugzilla ◦ Initial set up was hard but once set up worked well

Easy to organize bugs using different criteria

Status update emails when bug status changes

Daily reminder emails to ensure that bugs don’t get stale

◦ Added custom fields

Bug Injection Phase

Bug Detection Phase

Bug Removal Phase


Performance measure: latency 150-300ms

Extensibility: all actors were ported in less than 2 weeks. ◦ Plotter was the hardest one (8KLOC)

Reliability: latency monitoring stops the system

Usability: UI in Homer maps almost one to one to Android


Entities (actors or filters), links, and the pipe‐and-filter architecture within Ptolemy. The entities are linked and can only communicate through existing links connected with relations

79

Element Responsibility

Entity Entities are the filters in the supported pipe-and-filter architecture. They are responsible for computation within the system. An entity can contain any number of ports.

Port Port is a point of connection for the entity. It can serve both as output and input, and knows if there’s a token waiting for processing.

Relation The relation is responsible to keep track of links. It controls the creation and termination of links

Token An encapsulation of a data message. Token serves to identify the message type, size and data boundaries

Link A link defines an association between an entity's port and a relation.

A sample communication flow of the Ptolemy pipe-and-filter architecture based model. The model uses immutable objects called tokens to transfer any kind of data.

80

Element Responsibility

Entity Entities are the filters in the supported pipe-and-filter architecture. They are responsible for computation within the system. An entity can contain any number of ports.

Port Port is a point of connection for the entity. It can serve both as output and input, and knows if there’s a token waiting for processing.

Relation The relation is responsible to keep track of links. It controls the creation and termination of links

Token An encapsulation of a data message. Token serves to identify the message type, size and data boundaries

Link A link defines an association between an entity's port and a relation.

eosp summer 2011

Education

architecture team handsimdroid

defects team member

processes planning

ptolemy project

simulations of ptolemy

ui customization

opensource tool

handheld devices