understanding performance in operating systems
DESCRIPTION
Understanding Performance in Operating Systems. Andy Wang COP 5611 Advanced Operating Systems. Outline. Importance of operating systems performance Major issues in understanding operating systems performance Issues in experiment design. Importance of OS Performance. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/1.jpg)
Understanding Performance in Operating Systems
Andy Wang
COP 5611
Advanced Operating Systems
![Page 2: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/2.jpg)
Outline
Importance of operating systems performance
Major issues in understanding operating systems performance
Issues in experiment design
![Page 3: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/3.jpg)
Importance of OS Performance
Performance is almost always a key issue in operating systems File system research OS tools for multimedia Practically any OS area
Since everyone uses the OS (sometimes heavily), everyone is impacted by its performance
A solution that doesn’t perform well isn’t a solution at all
![Page 4: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/4.jpg)
Importance of Understanding OS Performance Great, so we work on improving OS
performance How do we tell if we succeeded? Successful research must prove its
performance characteristics to a skeptical community
![Page 5: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/5.jpg)
So What?
Proper performance evaluation is difficult Knowing what to study is tricky Performance evaluations take a lot of careful
work Understanding the results is hard Presenting them effectively is challenging
![Page 6: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/6.jpg)
For Example,
An idea - save power from a portable computer’s battery by using its wireless card to execute tasks remotely
Maybe that’s a good idea, maybe it isn’t How do we tell? Performance experiments to validate concept
![Page 7: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/7.jpg)
But What Experiments?
What tasks should we check? What should be the conditions of the portable
computer? What should be the conditions of the
network? What should be the conditions of the server? How do I tell if my result is statistically valid?
![Page 8: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/8.jpg)
Issues in Understanding OS Performance Techniques for understanding OS
performance Elements of performance evaluation Common mistakes in performance evaluation Choosing proper performance metrics Workload design/selection Monitors Software measurement tools
![Page 9: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/9.jpg)
Techniques for Understanding OS Performance Analytic modeling Simulation Measurement Which technique is right for a given situation?
![Page 10: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/10.jpg)
Analytic Modeling
+ Sometimes relatively quick+ Within limitations of model, testing
alternatives usually easy– Mathematical tractability may require
simplifications– Not everything models well– Question of validity of model
![Page 11: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/11.jpg)
Simulation
+ Great flexibility+ Can capture an arbitrary level of detail– Often a tremendous amount of work to write
and run– Testing a new alternative often requires
repeating a lot of work– Question of validity of simulation
![Page 12: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/12.jpg)
Experimentation
+ Lesser problems of validity+ Sometimes easy to get started– Can be very labor-intensive– Often hard to perform measurement– Sometimes hard to separate out effects you
want to study– Sometimes impossible to generate cases you
need to study
![Page 13: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/13.jpg)
Elements of Performance Evaluation Performance metrics Workloads Proper measurement technique Proper statistical techniques Minimization of effort Proper data presentation techniques
![Page 14: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/14.jpg)
Performance Metrics
The criteria used to evaluate the performance of a system
E.g., response time, cache hit ratio, bandwidth delivered, etc.
Choosing the proper metrics is key to a real understanding of system performance
![Page 15: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/15.jpg)
Workloads
The requests users make on a system If you don’t evaluate with a proper workload,
you aren’t measuring what real users will experience
Typical workloads - Stream of file system requests Set of jobs performed by users List of URLs submitted to a Web server
![Page 16: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/16.jpg)
Proper Performance Measurement Techniques You need at least two components to
measure performance
1. A load generator
To apply a workload to the system
2. A monitor
To find out what happened
![Page 17: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/17.jpg)
Proper Statistical Techniques
Computer performance measurements generally not purely deterministic
Most performance evaluations weigh the effects of different alternatives
How to separate meaningless variations from vital data in measurements?
Requires proper statistical techniques
![Page 18: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/18.jpg)
Minimizing Your Work
Unless you design carefully, you’ll measure a lot more than you need to
A careful design can save you from doing lots of measurements
Should identify critical factors And determine the smallest number of
experiments that gives a sufficiently accurate answer
![Page 19: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/19.jpg)
Proper Data Presentation Techniques You’ve got pertinent, statistically accurate
data that describes your system Now what? How to present it -
Honestly Clearly Convincingly
![Page 20: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/20.jpg)
Why Is Performance Analysis Difficult? Because it’s an art - it’s not mechanical
You can’t just apply a handful of principles and expect good results
You’ve got to understand your system You’ve got to select your measurement
techniques and tools properly You’ve got to be careful and honest
![Page 21: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/21.jpg)
Some Common Mistakes in Performance Evaluation No goals Biased goals Unsystematic approach Analysis without understanding Incorrect performance metrics Unrepresentative workload Wrong evaluation technique
![Page 22: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/22.jpg)
More Common Performance Evaluation Mistakes Overlooking important parameters Ignoring significant factors Inappropriate experiment design No analysis Erroneous analysis No sensitivity analysis
![Page 23: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/23.jpg)
Yet More Common Mistakes
Ignoring input errors Improper treatment of outliers Assuming static systems Ignoring variability Too complex analysis Improper presentation of results Ignoring social aspects Omitting assumptions/limitations
![Page 24: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/24.jpg)
Choosing Proper Performance Metrics Three types of common metrics:
Time (responsiveness) Processing rate (productivity) Resource consumption (utilization)
Can also measure various error parameters
![Page 25: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/25.jpg)
Response Time
How quickly does system produce results? Critical for applications such as:
Time sharing/interactive systems Real-time systems Parallel computing
![Page 26: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/26.jpg)
Processing Rate
How much work is done per unit time? Important for:
Determining feasibility of hardware Comparing different configurations Multimedia
![Page 27: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/27.jpg)
Resource Consumption
How much does the work cost? Used in:
Capacity planning Identifying bottlenecks
Also helps to identify the “next” bottleneck
![Page 28: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/28.jpg)
Typical Error Metrics
Successful service (speed) Incorrect service (reliability) No service (availability)
![Page 29: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/29.jpg)
Characterizing Metrics
Usually necessary to summarize Sometimes means are enough Variability is usually critical
![Page 30: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/30.jpg)
Essentials ofStatistical Evaluation Choose an appropriate summary
Mean, median, and/or mode Report measures of variation
Standard deviation, range, etc. Provide confidence intervals (95%) Use confidence intervals to compare means
![Page 31: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/31.jpg)
Choosing What to Measure
Pick metrics based on: Completeness (Non-)redundancy Variability
![Page 32: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/32.jpg)
Designing Workloads
What is a workload? Synthetic workloads Real-World benchmarks Application benchmarks “Standard” benchmarks Exercisers and drivers
![Page 33: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/33.jpg)
What is a Workload?
A workload is anything a computer is asked to do
Test workload: any workload used to analyze performance
Real workload: any workload observed during normal operations
Synthetic workload: any workload created for controlled testing
![Page 34: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/34.jpg)
Real Workloads
+ They represent reality– Uncontrolled
Can’t be repeated Can’t be described simply Difficult to analyze
Nevertheless, often useful for “final analysis” papers
![Page 35: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/35.jpg)
Synthetic Workloads
+ Controllable+ Repeatable+ Portable to other systems+ Easily modified– Can never be sure real world will be the same
![Page 36: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/36.jpg)
What Are Synthetic Workloads?
Complete programs designed specifically for measurement May do real or “fake” work May be adjustable (parameterized)
Two major classes: Benchmarks Exercisers
![Page 37: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/37.jpg)
Real-World Benchmarks
Pick a representative application and sample data
Run it on system to be tested Modified Andrew Benchmark, MAB, is a real-
world benchmark+ Easy to do, accurate for that sample
application and data– Doesn’t consider other applications and data
![Page 38: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/38.jpg)
Application Benchmarks
Variation on real-world benchmarks Choose most important subset of functions Write benchmark to test those functions+ Tests what computer will be used for– Need to be sure it captures all important
characteristics
![Page 39: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/39.jpg)
“Standard” Benchmarks
Often need to compare general-purpose systems for general-purpose use Should I buy a Compaq or a Dell PC? Tougher: Mac or PC?
Need an easy, comprehensive answer People writing articles often need to compare
tens of machines
![Page 40: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/40.jpg)
“Standard” Benchmarks (cont’d)
Often need comparisons over time How much faster is this year’s Pentium Pro
than last year’s Pentium? Writing new benchmark undesirable
Could be buggy or not representative Want to compare many people’s results
![Page 41: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/41.jpg)
Exercisers and Drivers
For I/O, network, non-CPU measurements Generate a workload, feed to internal or
external measured system I/O on local OS Network
Sometimes uses dedicated system, interface hardware
![Page 42: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/42.jpg)
Advantages and Disadvantages of Exercisers+ Easy to develop, port+ Incorporates measurement+ Easy to parameterize, adjust– High cost if external– Often too small compared to real workloads
![Page 43: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/43.jpg)
Workload Selection
Services exercised Completeness Level of detail Representativeness Timeliness Other considerations
![Page 44: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/44.jpg)
Services Exercised
What services does system actually use? Speeding up response to keystrokes won’t
help a file server What metrics measure these services?
![Page 45: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/45.jpg)
Completeness
Computer systems are complex Effect of interactions hard to predict So must be sure to test entire system
Important to understand balance between components
![Page 46: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/46.jpg)
Level of Detail
Detail trades off accuracy vs. cost Highest detail is complete trace Lowest is one request, usually most the
common request Intermediate approach: weight by frequency
![Page 47: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/47.jpg)
Representativeness
Obviously, workload should represent desired application
Again, accuracy and cost trade off Need to understand whether detail matters
![Page 48: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/48.jpg)
Timeliness
Usage patterns change over time File size grows to match disk size
If using “old” workloads, must be sure user behavior hasn’t changed
Even worse, behavior may change after test, as result of installing new system “Latent demand” phenomenon
![Page 49: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/49.jpg)
Other Considerations
Loading levels Full capacity Beyond capacity Actual usage
Repeatability of workload
![Page 50: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/50.jpg)
Monitors
A monitor is a tool used to observe system activity
Proper use of monitors is key to performance analysis
Also useful for other system observation purposes
![Page 51: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/51.jpg)
Event-Driven Vs. Sampling Monitors Event-driven monitors notice every time a
particular type of event occurs Ideal for rare events Require low per-invocation overheads
Sampling monitors check the state of the system periodically Good for frequent events Can afford higher overheads
![Page 52: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/52.jpg)
On-Line Vs. Batch Monitors
On-line monitors can display their information continuously Or, at least, frequently
Batch monitors save it for later Usually using separate analysis procedures
![Page 53: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/53.jpg)
Issues in Monitor Design
Activation mechanism Buffer issues Data compression/analysis Priority issues Abnormal events monitoring Distributed systems
![Page 54: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/54.jpg)
Activation Mechanism
When do you collect the data? Several possibilities:
When an interesting event occurs, trap to data collection routine
Analyze every step taken by system Go to data collection routine when timer
expires
![Page 55: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/55.jpg)
Buffer Issues
Buffer size should be big enough to avoid frequent disk writes But small enough to make disk writes cheap
Use at least two buffers, typically One to fill up, one to record
Must think about buffer overflow
![Page 56: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/56.jpg)
Data Compression or Analysis
Data can be literally compressed Or can be reduced to a summary form Both methods save space
But at the cost of extra overhead Sometimes can use idle time for this But idle time might be better spent dumping
data to disk
![Page 57: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/57.jpg)
Priority of Monitor
How high a priority should the monitor’s operations have?
Again, trading off performance impact against timely and complete data gathering
Not always a simple question
![Page 58: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/58.jpg)
Monitoring Abnormal Events
Often, knowing about failures and errors more important than knowing about normal operation
Sometimes requires special attention System may not be operating very well at the
time of the failure
![Page 59: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/59.jpg)
Monitoring Distributed Systems
Monitoring a distributed system is not dissimilar to designing a distributed system
Must deal with: Distributed state Unsynchronized clocks Partial failures
![Page 60: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/60.jpg)
Tools For Software Measurement
Code instrumentation Tracing packages System-provided metrics and utilities Profiling
![Page 61: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/61.jpg)
Code Instrumentation
Adding monitoring code to the system under study
+ Usually most direct way to gather data+ Complete flexibility + Strong control over costs of monitoring– Requires access to the source– Requires strong knowledge of code– Strong potential to affect performance
![Page 62: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/62.jpg)
Typical Types of Instrumentation
Counters+ Cheap and fast But low level of detail
Logs+ More detail But more costly Require occasional dumping or digesting
Timers
![Page 63: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/63.jpg)
Tracing Packages
Allow dynamic monitoring of code that doesn’t have built-in monitors
Akin to debuggers+ Allows arbitrary insertion of code+ No recompilation required+ Tremendous flexibility + No overhead when you’re not using it– Somewhat higher overheads– Effective use requires access to source
![Page 64: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/64.jpg)
System-Provided Metrics and Utilities Many operating systems provide users
access to some metrics Most operating systems also keep some form
of accounting logs Lots of information can be gathered this way
![Page 65: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/65.jpg)
Profiling
Many compilers provide easy facilities for profiling code
+ Easy to use+ Low impact on system– Requires recompilation– Provides very limited information
![Page 66: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/66.jpg)
Introduction To Experiment Design You know your metrics You know your factors You’ve got your instrumentation and test
loads Now what?
![Page 67: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/67.jpg)
Goals in Experiment Design
Obtain maximum information with minimum work Typically meaning minimum number of
experiments More experiments aren’t better if you have to
perform them Well-designed experiments are also easier to
analyze
![Page 68: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/68.jpg)
Experimental Replications
A run of the experiment with a particular set of levels and other inputs is a replication
Often, you need to do multiple replications with a single set of levels and other inputs For statistical validation
![Page 69: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/69.jpg)
Interacting Factors
Some factors have effects completely independent of each other Double the factor’s level, halve the response,
regardless of other factors But the effects of some factors depends on
the values of other factors Interacting factors
Presence of interacting factors complicates experimental design
![Page 70: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/70.jpg)
Basic Problem in Designing Experiments Your chosen factors may or may not interact How can you design an experiment that
captures the full range of the levels? With minimum amount of work
![Page 71: Understanding Performance in Operating Systems](https://reader035.vdocuments.mx/reader035/viewer/2022062517/56813074550346895d9651e7/html5/thumbnails/71.jpg)
Common Mistakes in Experimentation Ignoring experimental error Uncontrolled parameters Not isolating effects of different factors One-factor-at-a-time experiment designs Interactions ignored Designs require too many experiments