no silver bullet? silver buckshot may work · 2013-08-25 · "no silver bullet? silver...
Post on 24-May-2020
2 Views
Preview:
TRANSCRIPT
KT3 Keynote 11/10/2011 12:45:00 PM
"No Silver Bullet? Silver Buckshot May Work"
Presented by:
Gregory Pope Lawrence Livermore National Laboratory
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073 888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com
Gregory Pope Lawrence Livermore National Laboratory
With more than forty years of experience developing software in the commercial and
government sectors, Gregory Pope currently works for the Lawrence Livermore
National Laboratory as a software quality engineering group leader, and verification and validation
project leader for advanced simulation. Previously, Greg founded and ran a software testing
company, patented automated software testing tools, and held management and technical positions
involving mission-critical testing of military systems and development of software code for avionics
and aerospace uses. Greg has given industry keynote addresses, written technical papers, taught on
software quality internationally, and been a consultant.
9/28/2011
1
Lawrence Livermore National Laboratory
Gregory PopeSQE Group Leader
No Silver Bullet? Silver Buckshot May Work
This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344
2LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Silver Buckshot
� The proverbial Silver Bullet
� Most common request
� Definition of Better
� Good ideas reincarnate
� Some modern challenges
� Buckshot – Common problems and solutions
9/28/2011
2
3LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
No Silver Bullet
The phrase typically appears with an expectation that some new technological development or practice will easily cure a major prevailing problem.
4LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Example Silver Bullets
High Level Language -------------------------------------- c. 1968
Top Down Structured Programming --------------------c. 1974
Waterfall Lifecycle ------------------------------------------ c. 1976
DoD Standard 2167A ------------------------------------- c. 1988
Computer Aided Software Engineering (CASE)----- c. 1990
Object Oriented Design and Programming ---------- c. 1992
CMM/CMMI -------------------------------------------------- c. 1993
Rational Unified Process --------------------------------- c. 1996
Automated Testing Tools ---------------------------------- c. 1995
Continuous Integration ------------------------------------ c. 1999
9/28/2011
3
5LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Each Helped Make things Better
Increased productivity
Less spaghetti code
In process error detection, iterative, incremental spins
More documentation is not the answer - RUP, artifacts
Frameworks and plug ins - Eclipse, Team Forge
Reusability - Java, Python, Ruby, C#
Regulated vendor minimum competencies, Outsource
Expect requirements to change - Agile
Nightly regression testing – Junit, Subversion, QTP
Defects detected earlier – CMAKE, BuildBot, Jenkins
6LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Common Question
What can we do to make our process better?
9/28/2011
4
7LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Better for Who
A. Make the software better -------------------- c. 1970
B. Make the user experience better------------c. 1990
C. Make the developer experience better-----c. 2005
Better = A & B & C
8LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Better
� If the improvements do not make the developers life better, they will probably not be easily adopted. Dah
� Examples:
• Integrated Development Environments (IDEs)
• Networking
• Static analysis
• Continuous integration
• Distributed Code repositories
9/28/2011
5
9LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Most Common Process Improvement Response….
� Good idea, but we are busy, we will do it later.
10LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
An Observation: Later
� The “land of later” is a mythical place where nothing ever happens.
� Why?
• Because if you are successful you will be whisked off to a new project.
• If you unsuccessful you will be whisked off.
9/28/2011
6
11LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Bettering Unit Testing
� Debug and try it
� Write unit test drivers and stubs manually
� Stand alone framework to write unit tests
� Automated tool (Automated Regression Tests to run
periodically)
� Automated the Automated tool, run unit tests when a code
change is detected
12LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Bettering Coding Style
� Leave coding styles up to individuals
� Ask team to follow a coding standard
� Have a plug in tool that formats the code to the correct
style (i.e. indents) in the IDE
� Have a static analyzer that checks for style violations
9/28/2011
7
13LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Bettering Peer Reviews
� No peer review of code
� Meeting to peer review code
� Meeting to discuss findings in code
� Collaborative tool to allow code review
� Collaborative tool with built-in diff and tracker interface
(to requirements and bugs) and CM tool
14LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Bettering Static Analysis
� Compiler, debugging, and testing to find bugs
� Static analysis on integrated code
� New rules added to reduce false alarms
� Static analysis on code as it is built
� Automated static analysis on check in and nightly
� Automate the automated static analysis emailing only new issues found
9/28/2011
8
15LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Is Better Cheaper and Faster?
� Cost to manually test – 4 hours per bug*
� Cost of automated test – 1 hour per bug*
� Cost of static analysis – 10 minutes per bug**
* Cost to design tests (scripts) and execute
** Cost to build and triage code
Source: William Oliver LLNL “Quantifying the Value of Static Analysis”, Starwest 2011
16LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Source: Boris Beizer Software System Testing and Quality Assurance, page 34
All bugs are not structuralLLBut some structural bugs cause system, functional, and data bugs.
Static Analysis
Structural Bugs 27%
Functional Bugs 27%
15%
System16%
Data, Code, Other30%
15%
15%
9/28/2011
9
17LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Hawthorne Effect
� Study from 1927 to 1932 at a Western Electric Company Plant in Cicero, Illinois by Harvard Researcher Elton Mayo.
� One reasonable conclusion is that the workers were pleased to receive attention from the researchers who expressed an interest in them.
� Any new tool or process can cause process improvement.
18LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Best Ideas Seem to get Reincarnated
� Built-in-test for avionics ------ c. 1972
� Design by Contract
� Assertions
9/28/2011
10
19LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Best Ideas Seem to get Reincarnated
� Mission critical software --- c. 1972
� Test Driven Design
� Pairs Programming
Missile Simulator Software
New Software
RWR Software
20LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Best Ideas Seem to get Reincarnated
� ITT 1978 “stand up” meetings
� Accomplished, Plan to Accomplish, Obstacles
� Five minutes in length, can not sit down
� All decision makers present
� Scrum meeting
� “Main purpose of SCRUM is to find problems early”.
� Assumption is if a problem is identified early it will get fixed early. Not for all however LLL
Source: Lunch with Ken Schwaber 2007
9/28/2011
11
21LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
SCRUM
The XYZ project is
ahead of schedule
and under budget.
What do you mean
the software isn't
ready for test yet?
OK. You've got to
finish. Give it to test
for a couple of days.
We have got to add
a month to this
schedule.
Admits there is
a problem.
We should have time
to test if we get the
software on schedule.
You can't use our
test schedule to keep
writing new code.
We can't even begin
to test XYZ, let alone
finish, in two days.
You can't deliver XYZ
without a minimum of
testing.
Admits the need to
automate testing.
Let's plan a picnic to
celebrate the success-
ful XYZ delivery.
Perhaps we should
delay the picnic by a
week or so.
Perhaps we should
delay the picnic by
a month.
Refers Test Manager
to Employee
Assistance Program.
Finds job for old
program manager.
Admits self to
Employee Assistance
Program.
Trust me. We just
have a few minor
functions to add.
That last function
required changing 50%
of our design. Sorry.
Tomorrow, for certain,
will be the day we
give it to test.
We didn't know you
wanted all functions
in the first version.
Admits the need to
plan based on prior
experiences.
XYZ will increase our
sales by 50% next
quarter.
XYZ will be even
better than we
first thought.
XYZ is so good, test
hasn't even found any
bugs in it.
What do you mean
it's late and doesn't
work? We'll lose
market share!
Admits the need to keep
in closer contact with
engineering.
Denial
Anger
Depression
Bargaining
Acceptance
XYZ PROGRAM
MANAGER
TEST-QA
MANAGER
HR
MANAGER
ENGINEERING
MANAGER
MARKETING
MANAGER
DEPT
STAGE
Get to acceptance faster LL
22LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Contemporary Challenges
� Prototypes become products, options:
• Start over
• Refactor
• Wrappers
−Disable features
− Input checking
−Disable inputs
• Testing in Domain of Interest
9/28/2011
12
23LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Contemporary Challenges
� Code Security
• Static Analysis
• Security Testing
• C, C++ written before there was an internet *
• Security not a function or test, it is an intrinsic value*
� Optimistic Schedules
• Estimation Tools
• Acceleration
• Release Prediction Tools
* Source: James Gosling, inventor Java, speaking at Sandia Labs 8/18/2011
24LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Contemporary Challenges
� More Code Writers
• Automated workflow
� Text Books
• Some Examples
� Understanding Requirements
• Some thoughts
� Good Design
• Some thoughts
9/28/2011
13
25LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Contemporary Challenges
� Buggy Text Book Code
Type first integer : 5Type second integer : 6Type third integer : 7
5 is in the range 30 to 199 : FALSEat least two integers you typed are equal : FALSE5 != 6 : TRUENOT (5 < 6) : FALSE5 <= 6 : TRUE7 > 6 : FALSE(7 = 5) AND (6 != 7) : FALSE(7 <= 5) XOR (6 >= 7) : FALSE(7 > 5) AND (6 <= 7) : TRUE
Polarity
26LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
More Buggy Text Book Code
Please enter January 1's starting day;
A 0 indicates January 1 is on a Monday,A 1 indicates January 1 is on a Tuesday,
etc: 5
Enter the year you want the calendergenerated: 2011
The calendar for the year 2011
January Sun Mon Tue Wed Thu Fri Sat--- --- --- --- --- --- ---
1 23 4 5 6 7 8 9
10 11 12 13 14 15 1617 18 19 20 21 22 2324 25 26 27 28 29 3031
November Sun Mon Tue Wed Thu Fri Sat--- --- --- --- --- --- ---
1 2 3 4 5 67 8 9 10 11 12 13
14 15 16 17 18 19 2021 22 23 24 25 26 2728 29 30
Change string to Sunday and Monday
Spelling
9/28/2011
14
27LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Tracking Tool: - Collabnet
28LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Requirements
� The real answers are 6 levels down
� Who
� What
� Where
� Why
� How
� Who
� Dynamic tracing to code and tests
9/28/2011
15
29LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Non-Functional Requirements
� Usability
� Maintainability
� Scalability
� Availability
� Extensibility
� Security
� Portability
� Safety
30LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Requirements
� Prototypes help elicit requirements
� Good question asking skills
� Experience
� Can be too many or too few
� Requirement writers make good test designers.
9/28/2011
16
31LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Getting Requirements
� Never and Always LLLLLLLLL...
� Talk to the help desk folks
� Testing not needed for reuse (fail)
� Sit in the user’s seat
32LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Emerging Chaos
Chaos
Perceived Order
Emerging Problem Solving
Chaos
Emerging Order
Order
Deliberate Problem Seeking
Instead of trying to make order exist initially,chaos is allowed to exist so all stakeholders can problem seek together.
Emerging chaos closely resembles the waterfallor V-model, an attempt to capture all (or most) of the software requirements at the outset (create apparent order).
Source: Moshe F. Rubinstein and Iris R. Firstenberg, The Minding Organization, Bring the Future to the Present and
Turn Creative Ideas into Business Solutions, University of California, Los Angeles, John Wiley and Sons, pg. 91
Requirement Chaos for Creativity
9/28/2011
17
33LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Software Design
� Hard to get developers to do this
� Peer Pressure
� Manager Pressure
� Director Pressure
� Owner Pressure
� Even when done the design is seldom updated
34LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Code is the Design
� The source code is an abstraction of the executing binaries.
� It is a detailed design
� Source Code is like a blueprint is to a structure
� Architecture Design is important, like a scale model is to a structure.
9/28/2011
18
35LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Safety Moment
� Most safety problems with software intensive systems did not involve the software failing.
� Most safety problems were the interaction of the software components with other system components.
� Reliability is not the same as safety in software.
� Software does not fail the same as hardware.
Source: Nancy Levenson, Safeware System Safety and Computers
36LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Security Check:
� Weak coding practices allow hackers to exploit code
� Examples:
• Array bounds overflow
• Memory Leaks
• Tainted Inputs
• Null Pointer Dereference
• Uninitialized Variables
• Use tools to find this stuff
� So Better supports more Secure
9/28/2011
19
37LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Software Management
� Graded approach to rigor
� In general: The amount of rigor used for software process is proportional to the severity of the consequence of failure of the software being built.
� Managed, Documented, Understood
38LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Deja Vu All Over Again
� Running out of memory again:
Power consumption for an exascale (1018 flops) system in 2018 even under optimistic assumptions would be 100–200 MW, $100 Million annual electric bill.
� Memory reliability again
Cosmic radiation has a much higher probability of disturbing RAM memory.
Title Source: Yogi Berra, New York Yankees
http://www.lbl.gov/CS/html/SC08ExascalePowerWorkshop/gara.pdfFlash Memory Reliability NEPP 2008 Task Final Report, Yuan Chen, Jet Propulsion Laboratory, Pasadena, California
9/28/2011
20
39LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Adoption of Better By Decree
Is Difficult
� Generational
� Cultural
� Industrial
� Geographical
� Language
40LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Generational Factors
Matures1933-1945
Boomers1946-1964
Gen X1965-1976
Gen Y1977-1998
34 Million 76 Million 41 Million 75 Million
Source: Love’Em or Lose’Em, Kay and Jordan-Evans, page 236
9/28/2011
21
41LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Generational Cheat Sheet
Matures Boomers Generation X Generation Y
Work EthicWork 'til you
drop.
Work long hours
and tell your
manager about
it.
Personal life first,
work is
important.
Lifestyle comes first.
Loyality Loyal to employer.
Loyal to
employer, with
reservations.
Career and
professional
loyalty.
Career options.
TechnologyTechnology
fascination.
Technology
challanged.
Technology
proficient.Technology immersed.
Reporting
Relationships
Strong chain of
command.
Chain of
command.
What is the
purpose of a
chain of
command?
Be respectful but
move ahead.
Source: Love’Em or Lose’Em, Kay and Jordan-Evans, page 245
42LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Cultural Differences
� Globalization makes this more critical
� Teaching in Asia
• Questions written
• Prescriptive approach
� Teaching in Israel
• Highly Interactive
• Principle based approach
� Teaching in California
9/28/2011
22
43LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
East Coast / West Coast
EAST COAST
DO IT AND DO IT NOW
YOUR PLAN STINKSWHERE IS THE SCHEDULE?
THAT'S TOTALLY INCOMPETENTSHUT UP
WE'RE DONE GET OUT OF MY OFFICE
OH SH-WHAT'S WRONG WITH YOU?
HE'S A SUBORDINATEBOZO
I'LL COVER YOUR REARSHUT UP A MINUTE
DO IT RIGHT OR YOU’RE FIREDPUNCHING HIS/HER LIGHTS OUT
IS THERE A SPEC?WE ARE UNDER BUDGET
WE ARE OVERRUNNING BUDGET WE FINISHED EARLY
UNEMPLOYED
YOU'RE FIRED
GETTING RID OF DEAD WOOD
BUGS
KNOWN TO HAVE NOT IMMEDIATELY FAILED
WEST COAST
CAN YOU GO ALONG WITH THE PROGRAM?
LET ME SHARE MY FEELINGS ON THE PLAN
WHAT IS THE GAME PLAN?
LET ME BUILD ON THAT POINT
THANK YOU FOR SHARING THAT INPUT
HOW DO YOU FEEL ABOUT THAT ?
LET’S GET A CONSENSUS ON THIS ONE
THANKS FOR BRINGING THIS PROBLEM TO MY ATTENTION I CERTAINLY UNDERSTAND WHY YOU FEEL THAT WAY
HE'S A TEAM PLAYERSUBCONTRACTOR
CONSIDER ME YOUR RESOURCE LET ME SHARE THIS WITH YOU
I HAVE CONFIDENCE IN YOUCONSTRUCTIVE CONFRONTATION
WHAT IS A SPEC?WE HAVEN'T STARTED YET
WE ARE ON SCHEDULENO TRANSLATION CONSULTING OR IN TRANSITION
WE NEED MORE DISTANCE IN THIS RELATIONSHIP PLANNED EARLY RETIREMENT
UNINTENDED FEATURESIT WORKS
44LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Industry Differences
� Commercial
� System
� Military
� MIS
� Outsourced
Source: Capers Jones, Software Assessments, Benchmarks, and Best PracticesSource: Gregory Pope, Why Software Practices Become Evil
Many SQA practices are not portableacross industries, i.e. Military to Commercial
9/28/2011
23
45LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Software Development Principles Are Portable
Know who our customers are and meet or exceed their expectations with valuable software systems that are supportable.
Understand the context in which our software systems will operate as well as the technical constraints and interfaces.
Assemble the right team of individuals, give them the environment and support they need, and trust them to get the job done.
Support the customer’s expectations throughout the software project life cycle.
Pursue technical excellence by continuously preventing, detecting, and removing defects from the software system.
Collaborate and coordinate with partners frequently throughout the software project.
Create, integrate, and manage the software codes focusing on continuous improvement of the process and products.
Demonstrate frequently during development that the software system supports the customer’s expectations and is fully supportable.
Source: Gregory Pope, Why Software Practices Become Evil
46LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
My Code Does Not Need Testing
But you can run all the experiments you want.
9/28/2011
24
47LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Language Issue:
When Was the First Time You Heard the word “Test”?
48LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
A B C
1. Rapid 1. Quality 1. Assurance
2. Unified 2. Verification (and) 2. Validation
3. Agile 3. Experimental 3. Trails
4. Meta 4. Examination 4. Study
5. Flexible 5. Observational 5. Demonstration
6. Tailored 6. Conceptual 6. Prediction
7. Scalable 7. Acceptance 7. Proof
8. Integrated 8. Criterion 8. Scoring
9. Independent 9. Requirement
10. Observed 10. Satisfaction
11. Customer Based
Don’t Call It Testing Table
9/28/2011
25
49LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
A B
1. Potential 1. Anomaly
2. Suspect 2. Correctness
3. Tentative 3. Believability
4. Pseudo 4. Certainty
5. Unresolved 5. Convergence
6. Unstable 6. Correlation
7. Irregular 7. Correctitude
8. Arbitrary 8. Correspondence
9. Random 9. Censure
10. Fuzzy 10. Result
11. Biased 11. Presentation
Don’t Call It a Bug Table
50LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
“The software was so good that the
developers felt it to be without bugs and not
necessary to test. We did, however, perform
some Rapid Requirement Proofs and found
a number of cases of Irregular Convergence
and Biased Believability. These findings
were handled by the developers as trivial
enhancements, which have now been fully
implemented, and we are ready to ship after
performing the mandatory Independent
Observational Scoring.”
Release Notes:
Source: Gregory Pope, Test Is a Four Letter Word
9/28/2011
26
51LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Solutions
� Harley Davidson
� Cold Stone
� Mongolian Barbeque
� Oakley
� Risk Based Graded Approach
� Crystal
� Let Developers Pick Solutions
52LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Buckshot and Birdshot
9/28/2011
27
53LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Shotgun Analogy
Shot
Size
Pellet
Diameter
(inches)
# of
Pellets
in a 1oz
Shell
000 Buck .36 6
00 Buck .33 8
0 Buck .32 9
1 Buck .30 11
2 Buck .27 15
3 Buck .25 19
4 Buck .24 21
BB .18 50
2 Bird .148 90
4 Bird .129 135
5 Bird .12 170
6 Bird .109 225
54LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Agile Is a Bunch of Things
� Scrum
� Feature List
� Acceleration
� Sprints
� User Stories
� Pairs Programming
� Test Driven Design
� Automated Unit / System Testing
� Continuous Integration
9/28/2011
28
55LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
How to Load the Buckshot
� Determine problems, communicate with:
• Developers
• Testers
• Stakeholders, Users
• Marketing
• Help Desk
� Determine problems
� Look up problems to determine solutions
� Load your “shells” with these solutions
56LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
How to make Software Better
Neuron Connections
9/28/2011
29
57LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Silver Buckshot Demonstration
58LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Silver Buckshot Demonstration – Problem List
9/28/2011
30
59LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Silver Buckshot Demonstration – Solution List
60LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Problems that Solutions May Solve
9/28/2011
31
61LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Solutions that Prevent Defects
51 52
64
28
1117
22
64
13
28
15
30
16
0
10
20
30
40
50
60
70
Prevention Techniques
Prevention
62LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Solutions that Detect Defects
7
41
9
33
41
19 1813
31
41
13
6 7 6 7
20
0
24
7
41
4
33
17
7
05
1015202530354045
Detection Techniques
Detection
9/28/2011
32
63LLNL-PRES-493892 - Draft Lawrence Livermore National Laboratory
Silver Buckshot Conclusion
� The proverbial Silver Bullet
• No cure alls, but each has contributed
� Most common request
• How to make things better, cheaper, faster
� Definition of Better
• Better for everyone involved
� Good ideas reincarnate
• New names, new groupings
� Some modern challenges
• Security, Complexity, Scale, Safety
� Silver Buckshot – Common problems and solutions
• Buckshot Wiki, expert system on a Wiki
top related