2 u.s. dod contractor 24 years of experience b.s. computer science m.s. software engineering d.m....
DESCRIPTION
3 The purpose of this study is to determine if the use of agile methods improves the quality of ecommerce websites The goal of this study is to develop an instrument to measure the use of agile methods and ecommerce website quality The objective of this study is to examine the links between the factors of agile methods and ecommerce website qualityTRANSCRIPT
2
• U.S. DoD contractor• 24 years of experience• B.S. Computer Science• M.S. Software Engineering• D.M. Information Tech. (2007)• Four books and numerous articles• Worked in Japan, U.S., and Europe• Some experiences with agile methods
3
• The purpose of this study is to determine if the use of agile methods improves the quality of ecommerce websites
• The goal of this study is to develop an instrument to measure the use of agile methods and ecommerce website quality
• The objective of this study is to examine the links between the factors of agile methods and ecommerce website quality
4
• A software method is an approach to the analysis, design, construction, and implementation of an information system
• An agile method is a process of injecting customer feedback into a stream of working software versions to converge on a solution
• A website’s quality is the extent to which it facilitates efficient and effective shopping, purchasing, and delivery of its products
5
• The Internet is a powerful communication medium for free market-style business transactions
• The Internet shifted the balance of power away from industrial age organizations in favor of Internet firms
• The Internet presents formidable challenges for the development of Internet products and services
• Some use principles of flexible manufacturing, lean development, and adaptation to turbulent markets
• Other firms continue to use methods based on principles pioneered at the turn of the 20th century
• Our challenge is to help determine if the use of agile methods is more effective than traditional ones
6
• IT is 2nd major contributor to U.S. economy• There are 5 million IT workers in the U.S.• Today, there are over 100 million websites• Top 500 U.S. firms spend $186B on IT• U.S. ecommerce revenues are about $2T
–$1.82T is B2B–$130B is B2C
• There are about 250,000 U.S. IT projects–180,000 U.S. IT projects are failing each year–165,000 use agile methods to help reduce failures
7
• 1928 - Organismic Biology (Bertalanffy)
• 1949 - Cybernetics (Wiener)
• 1966 - Learning by doing (Levhari)
• 1969 - Involvement (Fitch)
• 1971 - Participatory design (Milne)
• 1975 - Iterative (Basili)
• 1976 - Evolutionary (Bauer)
• 1977 - Double-loop learning (Argyris)
• 1978 - Incremental (Cave)
• 1982 - Adaptive organization (Anderson)
• 1982 - Prototyping (Naumann)
• 1986 - Joint application design (IBM)
• 1987 - Rapid systems design (Gane)
• 1986 - Spiral (Boehm)
• 1991 - Cooperative design (Greenbaum)
• 1995 - Sense and response (Randall)
• 1996 - Ecosystems (Moore)
• 1997 - Probes (Brown)
• 1998 - Experimentation (Thomke)
• 1998 - Internet time (Cusumano)
8
• Cheap personal computers• Standard operating systems• Easy-to-use Internet browsers• Powerful programming languages• Explosion of computer programmers• Real-time software development cycles• Internet shopping (electronic commerce)
9
Phase System
Requirements Analysis
System Architectural
Design
Software Requirements
Analysis
Software Architectural
Design
Software Detailed Design
Software Coding
and Testing Software
Integration Software
Qualification Testing
System Integration
System Qualification
Testing Software
Installation Software
Acceptance Support
Product SRS SARAD SRD
SAD SIDD (t) DDD (t) UDD (p) TVPL (si)
SDD SIDD (d) DDD (d) UDD (u) TVPL (su) TVPL (siu)
Software TVPR (su) TVRR (su) UDD (u) TVPL (siu)
SOIP TVRR (si) UDD (u) TVPR (sq)
TVRR (sq) UDD (u) SIAR (sfc) SIAR (spc)
TVRR (yi) TVPR (yq)
TVRR (yq) SIAR (yfc) SIAR (ypc)
SIP TVRR (sa) Training
Evaluation Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Walkthru Inspection
Record SY RER SYAER SORER SOAER DDER EOCR SCTRER SCR
SIER DER SCR SQTER
SCR SER SQTARR
SIRR SCR
Audit PCA FCA PCA
FCA
Review System
Requirements Review
System Design Review
Software Specification
Review
Preliminary Design Review
Critical Design Review
Software
Test Readiness
Review
Software Formal
Qualification Review
System Test
Readiness Review
System Formal
Qualification Review
Baseline Functional Baseline Allocated
Baseline Developmental Configuration
Software Test
Baseline
Software Product Baseline
System Test
Baseline
System Product Baseline
12 phases — 35 documents — 62 evaluations — 17 records — 4 audits — 9 reviews — 9 baselines — 28,978 hours (10 KLOC)
10
11
Y ear Source Findings Responses
1998 Harvard (Thomke et al., 1998)
50% reduction in engineering effort 55% improvement in time to market 925% improvement in number of changes allowed
391
1998 Harvard (MacCormack, 1998)
48% productivity increase over traditional methods 38% higher quality associated with more design effort 50% higher quality associated with iterative development
29
1999 Boston College (Fichman et al., 1999)
38% reduction in time to produce working software 50% time to market improvement 50% more capabilities delivered to customers
28
2003 Reifer Consultants (Reifer, 2003)
20% reported productivity gains 10% reported cost reductions 53% reported time-to-market improvements
78
2003 Shine Technologies (Johnson, 2003)
49% experienced cost reductions 93% experienced productivity increases 88% experienced customer satisfaction improvements
131
2004 CIO Magazine (Prewitt, 2004)
28% had been using agile methods since 2001 85% initiated enterprise-wide agile methods initiatives 43% used agile methods to improve growth and marketshare
100
2006 Digital Focus (Digital Focus, 2006)
27% of software projects used agile methods 23% had enterprise-wide agile methods initiatives 51% used agile methods to speed-up development
136
2006 Version One (Version One, 2006)
86% reported time-to-market improvements 87% reported productivity improvements 92% reported ability to dynamically change priorities
722
2006 AmbySoft (Ambler, 2006)
41% of organizations used agile methods 44% reported improved productivity, quality, and costs 38% reported improvements in customer satisfaction levels
4,232
12
• Early studies based on primitive models• Agile methods matured after early studies• Later studies were either
– Too narrow (e.g., pair programming)– Too specific (e.g., extreme programming)– Too broad (e.g., every conceivable practice)
• Gaps in literature and need for new study– Theory covering essential factors– General-purpose measurement model– Original measurement data on agile methods
13
Method Major Factors New development
rhythm Iterations 1, involvement 2, empowered teams 3, modularity 4, synchronization, configuration control, dependency management, performance reviews, metrics, testing, reviews
Scrum Iterative development 1, stakeholder feedback 2, self managing teams 3, prioritized requirements 3, daily team meetings 3, early architectural design 4
Dynamic systems development
Iterative development 1, frequent delivery 1, user involvement 2, stakeholder cooperation 2, empowered teams 3, simple flexible designs 4, change control, high-level requirements, tests
Synch-n-stabilize Iterations 1, daily builds 1, releases 1, customer feedback 2, small teams 3, vision statements 3, prioritized features 3, milestones 3, evolving specifications 4, parallel development
Internet time Rapid prototyping and early beta releases 1, daily incorporation of rapid market feedback 2, experienced teams 3, large investments in software architecture and design 4
J udo strategy Beta testing 1, market feedback 2, small teams 3, cross platform design 4, modular designs 4, reuse 4, flexible priorities 4, evolving features 4, parallel development, testing
Extreme programming
Releases 1, on-site customer 2, pair programming 3, simplicity 4, planning, metaphors, tests, refactoring, continuous integration, collective owners, 40-hours, open workspace, just rules
Feature driven development
Regular builds 1, domain experts 2, feature teams 3, technical architecture 4, object modeling, design by feature, class (code) ownership, inspections, configuration management, reporting
Open source software
Rapid releases 1, increased user involvement 2, prompt feedback 2, international community 3, highly-talented developers 3, evolutionary designs 4, parallel development, peer reviews
Agile manifesto Working software 1, Customer collaboration 2, individuals and interactions 3, responding to change 4
1 Iterative development — 2 Customer feedback — 3 Well-structured teams — 4 Flexibility
14
Product Dev. Institute Harvard I Harvard I I National Research Council NASA LeanTEC Customer focused Build in market-focused actions Market needs/customer value Field work by core project team Begin early Broaden the base
Front-end loaded Preliminary market assessment Technical assessment Supplier assessment Market research Concept testing Customer value assessment Business/financial assessment
Spiral development User needs and wants study Full proposition concept test Rapid prototype and test Field trial and beta test
Cross functional teams Multi-disciplinary team End-to-end project
responsibility Clearly defined team leader Excellent communications Empowerment over resources
Measure and improve Put performance metrics in
place Establish success criteria Hold teams responsible Continuously improve
Maximize portfolio productivity Strategic buckets Product roadmaps Project selection/prioritization Resource allocations Portfolio reviews
Product innovation Institutionalize stage-gates Scalable/adaptable stage-gates Automate stage-gates Alliance-enabled stage-gates Improve stage-gates
Development strategy Prioritize projects Functional integration Maximize efficiency Create/improve capabilities
Functional mapping Marketing mapping Engineering mapping Manufacturing mapping Integrative mapping
Aggregate project planning Research projects Alliance-based projects Incremental projects Breakthrough projects Next-generation projects
Development funnels Research-driven Entrepreneurial-driven Innovation-driven
Development framework Customer-focused Disciplined Coherence and detailed Fit with mission Standardized pattern
Cross-functional integration Cross-communication Relationship management Organization commitment Management commitment Incentives to integrate
Cross-functional leadership Functional teams Lightweight teams Heavyweight teams Autonomous teams
Tools and methods Prototype test cycles Organizational learning Capability development
Superior performance in time, productivity, and quality Lead time focus Productivity focus Total product quality focus
Integration in the development process Overlapping and
communication Small team sizes and
specialization Simple, flatter organizations
Integrating customer and product Heavyweight product managers Customer access and orientation Leadership by concept
Manufacturing for design Manufacturing principles focus Rapid prototyping Rapid tools development
Acceptance of risk Flexible environment Open communication Organization-wide
commitment Value innovation Focus on end-user needs Reduced lead times Consistent funding Use of COTS Iterative development Simple procurement and
acquisition processes Flexible standards and
testing procedures Decentralized decision-
making
Schedule/budget constraints Collocated personnel Flat organization Concurrent engineering Contract outsourcing Streamlined acquisition COTS components Design reuse Flexible designs Front-loaded funding Simplified reviews Minimal redundancy Extensive testing Public relations
Technology transition process
Enabling environment Technology transition
portfolio Project resources Project charters Project plans Project contracts Communication protocols Collaboration protocols Shared team experiences Formal reviews
Common Factors Early market feedback (e.g., on each increment). Iterative development (e.g., 30-60-90 day increments). Small, highly-qualified cross-functional project teams
with influential leaders (e.g., mostly PhDs). Flexible processes and products (e.g., streamlined
processes and rapid-prototyping technologies). (What this boils down to is integrating and streamlining
research, development, and operations into a single, smaller, and more efficient organization.)
15
Iterative Development(Software Management)
Basili & Turner (1975) Madden & Rone (1984)
Boehm (1985)
AGILE METHODS(Manufacturing, Internet)
Anderson et al. (1998) Beck (1999)
Highsmith (2002)
Well-Structured Teams(Sociology, Psychology, & Org. Behavior)
Babchuk & Goode (1951) Trow (1957) Herbst (1962)
Customer Feedback(Marketing, Psychology, & Sociology)
Dunn (1966) Milne (1971)
Von Hippel (1978)
Flexibility(Manufacturing, Innovation, & Internet)
Carlin (1958) Williams & Williams (1964) Stecke & Solberg (1981)
Website Quality(Manufacturing, Marketing, & Software)
Rubey & Hartwick (1968) Boehm, Brown, Kaspar, & Lipow (1973)
Gilb (1977)
16
Agile Methods Website Effectiveness Electronic Commerce
Customer Feedback
Feedback solicited Feedback received Feedback frequency Feedback quality Feedback incorporated
Iterative Development
Time boxed releases Operational releases Small releases Frequent releases Numerous releases
Well-Structured Teams
Team leader Vision and strategy Goals and objectives Schedules and timelines Small team size
Flexibility
Small size Simple design Modular design Portable design Extensible design
H 1 (+)
H 2 (+)
H 3 (+)
H 4 (+)
Website Quality
Website design Privacy and security Fulfillment and reliability Customer service
Business-to-Consumer (B2C)
Shopping Retail Services
17
Factor Variable Item Time-boxed releases We develop software using time-based iterations, increments, or demonstrations
Operational releases We develop software using operational iterations, increments, or demonstrations (working code)
Small releases We develop software using small iterations, increments, or demonstrations
Frequent releases We develop software using daily, weekly, bi-weekly, or monthly iterations, increments, or demonstrations
Iterative development
Numerous releases We develop software using multiple (several) iterations, increments, or demonstrations
Feedback solicited We seek customer feedback on our software iterations, increments, or demonstrations
Feedback received We receive customer feedback on our software iterations, increments, or demonstrations
Feedback frequency We receive timely customer feedback on our software iterations, increments, or demonstrations
Feedback quality We receive a lot of (detailed) customer feedback on our software iterations, increments, or demonstrations
Customer feedback
Feedback incorporated We incorporate customer feedback into our software iterations, increments, or demonstrations
Team leader Our software teams have clear administrative or technical leaders
Vision and strategy Our software teams have clear visions, missions, or strategies
Goals and objectives Our software teams have clear goals or objectives
Schedules and timelines Our software teams have clear schedules or timelines
Well-structured
teams
Small team size Our software teams have a small size with no more than 10 people
Small size Our software is designed to be as small as possible
Simple design Our software is designed to be as simple as possible
Modular design Our software is designed to be modular or object-oriented
Portable design Our software is designed to work on multiple operating systems
Customer service
Extensible design Our software is designed to be changed, modified, or maintained
18
• Consulted with numerous experts• Analyzed 100s of papers/books (3 yrs)• Used a lot of suggestions and changes• Presented research at major conference• Solicited expert feedback on final models• Conducted cognitive interviews w/experts• Numerous refinements to survey instrument
– Prototype surveys– Multiple pilot surveys– Multiple-phased surveys
19
• Type of service– Online service (e.g., freeonlinesurveys.com)– Free trial-period, inexpensive, and easy-to-use
• Source of data– Clients of big consulting firm (e.g., QAI)– Patrons of popular journal (e.g., Dr. Dobbs)– Assisted by Scott Ambler and Jon Erickson (editor)
• Method of administration– 1st round from email survey (50 respondents)– 2nd round from editor’s blog (100 respondents)– 3rd round from editor’s newsletter (100 respondents)
• Type of survey and data collection– Respondents mostly self-selected– Respondents were provided incentives
20
ANNUAL REVENUES
22%
19%
11%9%
7%
5%
19%
5%
3%
0-$999,000
$1,000,000-$9,999,999
$10,000,000-$24,999,999
$25,000,000-$49,999,999
$50,000,000-$99,999,999
$100,000,000-$249,999,999
$250,000,000-$499,999,999
$500,000,000-$999,999,999
$1,000,000,000-More
JOB FUNCTION
6%6%
11%
2%
33% 15%
4%
5%5%2%1%
10%Executive
Director
Sr. Manager
Project Manager
Chief Engineer
Systems Engineer
Webmaster
Database Admin
Program Analyst
Software Engineer
System Admin
Other
NUMBER OF EMPLOYEES
17%
10%
9%
4%
5%
10%10%
7%
4%
9%
7%
6%2%
1-1920-4950-99100-249250-499500-9991,000-2,499
2,500-4,9995,000-9,99910,000-24,99925,000-49,99950,000-99,999100,000-More
INDUSTRY SECTOR3%
7%0%
3%
3%
1% 9%27%
21%
1%
3%
2%
6%
1%12%
1%
Utilities ManufacturingWholesale RetailTransportation InformationFinance Real EstateProfessional ManagementEducational Health CareArts AccommodationOther Public Admin
YEARS OF EXPERIENCE
13%
18%
22%16%
15%
7%
5%4%
0-56-1011-1516-2021-2526-3031-3536-More
21
1 2 3 4 5 6 7 Factor Variable Strongly
Disagree Disagree Somewhat Disagree Neutral Somewhat
Agree Agree Strongly Agree
Responses Score
Time-boxed releases 14 (6%) 23 (9%) 22 (9%) 13 (5%) 38 (15%) 59 (24%) 80 (32%) 249 5.15 / 7 (73.55%)
Operational releases 10 (4%) 18 (7%) 19 (8%) 17 (7%) 50 (20%) 84 (34%) 51 (20%) 249 5.15 / 7 (73.55%)
Small releases 10 (4%) 24 (10%) 18 (7%) 25 (10%) 34 (14%) 64 (26%) 71 (29%) 246 5.13 / 7 (73.34%)
Frequent releases 27 (11%) 30 (12%) 17 (7%) 23 (9%) 34 (14%) 50 (20%) 66 (27%) 247 4.7 / 7 (67.21%)
ITERATIVE DEVELOPMENT
Numerous releases 9 (4%) 18 (7%) 11 (4%) 26 (10%) 35 (14%) 67 (27%) 82 (33%) 248 5.38 / 7 (76.79%)
Feedback solicited 6 (2%) 14 (6%) 12 (5%) 20 (8%) 38 (15%) 72 (29%) 87 (35%) 249 5.55 / 7 (79.23%)
Feedback received 4 (2%) 16 (6%) 24 (10%) 17 (7%) 55 (22%) 77 (31%) 56 (22%) 249 5.24 / 7 (74.87%)
Feedback frequency 7 (3%) 34 (14%) 35 (14%) 37 (15%) 60 (24%) 44 (18%) 29 (12%) 246 4.45 / 7 (63.59%)
Feedback quality 25 (10%) 35 (14%) 39 (16%) 37 (15%) 44 (18%) 45 (18%) 21 (9%) 246 4.05 / 7 (57.9%)
CUSTOMER FEEDBACK
Feedback incorporated 6 (2%) 8 (3%) 12 (5%) 13 (5%) 44 (18%) 80 (32%) 86 (35%) 249 5.67 / 7 (81.01%)
Team leader 9 (4%) 11 (4%) 20 (8%) 27 (11%) 43 (17%) 84 (33%) 57 (23%) 251 5.25 / 7 (74.96%)
Vision and strategy 11 (4%) 20 (8%) 23 (9%) 29 (12%) 60 (24%) 79 (32%) 28 (11%) 250 4.82 / 7 (68.91%)
Goals and objectives 3 (1%) 12 (5%) 22 (9%) 24 (10%) 57 (23%) 95 (38%) 36 (14%) 249 5.2 / 7 (74.35%)
Schedules and timelines 6 (2%) 15 (6%) 26 (11%) 25 (10%) 59 (24%) 67 (27%) 48 (20%) 246 5.07 / 7 (72.42%)
WELL-STRUCTURED
TEAMS
Small team size 6 (2%) 10 (4%) 10 (4%) 13 (5%) 16 (6%) 63 (25%) 131 (53%) 249 5.96 / 7 (85.08%)
Small size 13 (5%) 44 (18%) 18 (7%) 61 (24%) 51 (20%) 38 (15%) 24 (10%) 249 4.22 / 7 (60.24%)
Simple design 8 (3%) 28 (11%) 21 (8%) 37 (15%) 47 (19%) 67 (27%) 41 (16%) 249 4.82 / 7 (68.79%)
Modular design 6 (2%) 11 (4%) 14 (6%) 23 (9%) 49 (20%) 87 (35%) 58 (23%) 248 5.38 / 7 (76.9%)
Portable design 38 (15%) 41 (17%) 20 (8%) 32 (13%) 36 (15%) 43 (17%) 38 (15%) 248 4.08 / 7 (58.29%)
FLEXIBILITY
Extensible design 4 (2%) 16 (6%) 23 (9%) 23 (9%) 51 (20%) 75 (30%) 59 (24%) 251 5.24 / 7 (74.84%)
Survey has good reliability and validity – Low feedback frequency and quality is systemic issue – Poor wording affected small size and portable design
22
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Time-boxed releases
Operational releases
Small releases
Frequent releases
Numerous releases
Feedback solicited
Feedback received
Feedback frequency
Feedback quality
Feedback incorporated
Team leader
Vision and strategy
Goals and objectives
Schedules and timelines
Small team size
Small size
Simple design
Modular design
Portable design
Extensible design
ITER
ATI
VED
EVEL
OPM
ENT
CU
STO
MER
FEED
BA
CK
WEL
L-ST
RU
CTU
RED
TEA
MS
FLEX
IBIL
ITY
Strongly Agree
Agree
Somewhat Agree
Neutral
Somewhat Disagree
Disagree
Strongly Disagree
Blues and greens indicate reliability and validity – Feedback frequency and quality systemic issue – Small size and portable design misinterpreted
23
PRODUCTIVITY
3%11%
13%
19%
17%
19%
18%0 to 10%
11% to 25%
26% to 50%
51% to 75%
76% to 100%
101% to 200%
201% or more
CYCLE TIME
8%
11%
14%
18% 13%
17%
19%
0 to 10%
11% to 25%
26% to 50%
51% to 75%
76% to 100%
101% to 200%
201% or more
COST
26%
19%
19%
15%
7%
11%
3%
0 to 10%
11% to 25%
26% to 50%
51% to 75%
76% to 100%
101% to 200%
201% or more
QUALITY
5%
15%
13%
13%19%
15%
20%0 to 10%
11% to 25%
26% to 50%
51% to 75%
76% to 100%
101% to 200%
201% or more
CUSTOMER SATISFACTION
18%
10%
18%
10%
25%
11%
8%0 to 10%
11% to 25%
26% to 50%
51% to 75%
76% to 100%
101% to 200%
201% or more
24
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
In-depth information
Processing efficiency
Processing speed
Personalization
Product selection
Protection of privacy
Feelings of safety
Adequate security
Order received
On time delivery
Order accurate
Willingness to respond
Desire to fix issues
Promptness of service
WEB
SITE
DES
IGN
PRIV
AC
Y A
ND
SEC
UR
ITY
FULF
ILLM
ENT
AN
DR
ELIA
BIL
ITY
CU
STO
MER
SER
VIC
E
Strongly Agree
Agree
Somewhat Agree
Neutral
Somewhat Disagree
Disagree
Strongly Disagree
Results of independent assessment of website quality using eTailQ based on complete transactions (e.g., order through delivery)
25
IterativeDevelopment
CustomerFeedback
Well-StructuredTeams
Flexibility
WebsiteDesign
Privacyand Security
Fulfillmentand Reliability
CustomerService
Agile Methods Website QualityH 0 (+) = + 0.541 (0.094)
H 1 (+) = + 0.758 (0.039)
H 2 (+) = + 1.395 (0.095)
H 3 (+) = – 1.333 (0.051)
H 4 (+) = – 1.031 (0.030)
First three hypotheses are true and last two hypotheses are not true
26
• Agile survey questions exhibit reliability• Customer feedback data indicate issues• Flexibility variables need some refinement• “All” 20 agile/benefit variables correlated• Aggregate agile/quality factors correlated• Individual agile/quality factors not correlated
– Population building websites is too small– Should have used generalized quality model– Simpler research design could improve results
27
• Choose large/slow industry to study• Use cognitive interviews/pilot surveys• Use online survey ‘sites to collect data• Use web blogs instead of email surveys• Develop small/simple theory and survey• Utilize very simple single-phase surveys• Present minor incentives to respondents• Ask for a lot of help from industry leaders
28
• Published three items– Conference paper– Journal article– Textbook chapter
• Designed a unique general-purpose– Conceptual theory of agile methods– Measurement instrument for agile methods
• Collected original data and measurements– Collected original data using main instrument– Identified/used best-of-breed web quality model– Established a good foundation for future research