lec05 product rule and bayes' rule

Probability and Statistics for Teachers, Math 507, Lecture 5

1

USES OF CONDITIONAL PROBABILITY

The Product Rule, Bayes’ Rule, and Extended Independence


2

The Product Rule

• Notation: In advanced probability courses it is common to denote the intersection of events by concatenating them rather than writing an intersection symbol between them. That is, if A and B are events we write AB instead of . Our book does not use this notation, but I will use it in this lecture to simplify what I have to type (i.e., writing the letters adjacent does not make me use the equation editor!).

BA


3

The Product Rule

• Example: Suppose a bag contains four green beads and seven red ones. If I pull two out (without replacement), what is the probability that both are red?– Intuitively my probability of getting red on the first draw is

7/11, my probability of then getting another red on the second draw is 6/10=3/5, and my total probability of getting both red is (7/11)*(3/5)=21/55, which is about 0.38.

– Formally we can solve the problem by counting. There are C(11,2)=55 2-subsets of the 11 beads, and C(7,2)=21 of them contain 2 red beads. Assuming all outcomes are equally likely, the probability of 2 reds is 21/55. We see that our intuitive procedure yields the correct answer, but can we justify it formally?


4

The Product Rule

• Theorem (The Product Rule for Probabilities): Suppose A and B are events in a sample space S with probability measure P. We already know that P(B|A)=P(AB)/P(A). Clearing denominators we see P(AB)=P(A)P(B|A). In other words, the probability that A and B both happen equals the probability that A happens times the probability that B happens if A does.


5

The Product Rule

• Example Revisited: Again suppose we have four green and seven red beads in a bag and we choose two of them without replacement. Let F be the event that the first bead chosen is red and S be the event that the second bead chosen is red. Then the even that both beads are red is FS. We can find its probability from the product rule as follows: P(FS)=P(F)P(S|F)=(7/11)*(6/10)=21/55. Our intuition is now justified.


6

The Product Rule• The product rule generalizes to more events. For instance,

suppose A, B, C, and D are events in a sample space S with probability measure P. Then P(ABCD)=P(A)P(B|A)P(C|AB)P(D|ABC). This same pattern works with any number of events.

• Example Re-revisited: Now suppose we have four green and seven red beads and we want the probability that four beads chosen (without replacement) from the bag are all red. Let A, B, C, and D, be the events that the first, second, third, and fourth beads are red, respectively. Then the probability that all four beads are red is P(ABCD)=(7/11)*(6/10)*(5/9)*(4/8) =(7/11)*(3/5)*(5/9)*(1/2)=7/66, which is about 0.11. Note that, for instance, the fourth factor 4/8=1/2 is the probability that the fourth bead is red after we have already removed three red beads.


7

Independence of three events

• If A, B, and C are events in some sample space S with a probability measure P, then we say the events are independent if each pair is independent (i.e., P(AB)=P(A)P(B), P(AC)=P(A)P(C), and P(BC)=P(B)P(C)) and in addition P(ABC)=P(A)P(B)P(C).

• Intuitively this is what is needed to guarantee that all the events and their complements are proportionally represented in each other in all relevant ways. It is possible to satisfy some of these equations while failing to satisfy others


8


• Contrary Example: Roll a red die and a clear die. Let A be the event “the red die is 1,” B be the event “the clear die is 3,” and C be the event “both dice have the same number.” The P(A)=P(B)=P(C)=6/36=1/6. Clearly P(AB)=1/36=P(A)P(B), P(AC)=1/36=P(A)P(C), and P(BC)=1/36=P(B)P(C). The event ABC, however, is empty (since you cannot have the red die 1, the clear die 3, and both dice the same), so P(ABC)=0. But P(A)P(B)P(C)=(1/6)(1/6)(1/6)=1/216. Thus A, B, and C are not independent.


9


• Example: Roll red, green, and clear dice. Let A be the event “the red dice is 1,” B be the events “the green die is 1,” and C be the event “the clear die is 1.” It is easy to test that events A, B, and C are independent under the uniform model. In particular P(ABC)=1/216=(1/6)(1/6)(1/6)=P(A)P(B)P(C).


10

Independence of more events.

• A collection of events is independent if the probability of every subset of them equals the product of the probabilities of the events in the subsets. For example, events A, B, C, and D are independent if every pair is independent, every triple is independent, and P(ABCD)=P(A)P(B)P(C)P(D). (Consider rolling four dice and getting all 1’s, flipping four coins and getting HTTH).


11

Bayes’ Rule

• Bayes’ Rule is a simple formula relating the values of P(A|B) and P(B|A). It has several forms and interesting consequences.


12

Bayes’ Rule

• Theorem 2.16 (Bayes’ Rule)– Given events H and E in a sample space S with probability

measure P it holds that P(H|E)=P(H)P(E|H)/P(E)

– Proof: By definition of conditional probability, P(H|E)=P(HE)/P(E). By the product rule P(HE)=P(H)P(E|H). Therefore P(H|E)=P(H)P(E|H)/P(E).

– Here we use H and E to stand for Hypothesis and Evidence. We sometimes conceive of Bayes’ Rule as telling us how to revise the probability of a hypothesis based on the observation of some particular piece of evidence.


13

Bayes’ Rule

• Example: You are living in a dorm. One night the fire alarm goes off. How likely is it that there is a fire? Here H is the event “there is a fire” and E is the event “the fire alarm goes off.” You want to know P(H|E). You estimate that all things being equal a fire is unlikely on a given night, setting P(H)=0.001 (roughly one fire in three years). You know that in a typical semester of about 100 days there are about 3 fire alarms (typically false alarms), so you estimate P(E)=0.03. Finally you guess that it is nearly certain someone would set off the alarm if there really were a fire, so you estimate P(E|H)=0.98. By Bayes’ Rule, P(H|E)=P(H)P(E|H)/P(E)=(0.001)(0.98)/(0.03)=0.033.


14

Bayes’ Rule

• Notes to the example: From one point of view the alarm is almost meaningless. There is only 3.3% chance of a fire. Why is it so low? Your probabilities say that in 100 days you should expect 30 alarms but only one fire. Thus your chance of having fire with the alarm is 1/30. From another point of view the alarm carries a lot of weight: The alarm raises the likelihood of a fire thirtyfold, from 0.1% to 3.3% (that is 1/1000 to 1/30). This is how the evidence (alarm) causes you to revise your estimate of the hypothesis (fire). In any case the difference between P(H|E) and P(E|H) is large: 0.033 to 0.98, a clear example of how these quantities need not be equal.


15

Bayes’ Rule

• Theorem 2.17 (Bayes’ Rule, extended form)– Under the same circumstances as before .

)|()()|()(

)|()()|(

HEPHPHEPHP

HEPHPEHP


16

Bayes’ Rule• Proof: This is the same equation as

in the simpler statement except that in that case the denominator was simply P(E). It is easy to see that the sets EH and partition E. The Venn Diagram makes it clear: The event E is partitioned into the yellow section EH and the orange section . Since the sets are disjoint and have union E, we have By the product rule the righthand side becomes So we have the same equation as before, but with a fancy expansion of P(E) in the denominator.

S

H E

HE

HE

)()()( HEPEHPEP

)|()()|()( HEPHPHEPHP


17

Bayes’ Rule

• Example (medical testing)– A drug company has designed a test for a disease. Through

extensive testing, the company reports that the test produces only 1% false positive results (i.e., a healthy person tests positive) and only 2% false negative results (i.e., a person with the disease tests negative). Let P be the event “someone tests positive,” N be the event “someone tests negative,” H be the event “someone is healthy,” and D be the event “someone has the disease.” Then the company is reporting P(P|H)=0.01 (or equivalently P(N|H)=0.99) and P(N|D)=0.02 (or equivalently P(P|D)=0.98).


18

Bayes’ Rule

• Example (medical testing)– Suppose you test positive for the disease. How likely is it that

you in fact have the disease? It is tempting but incorrect to say 98% since P(P|D)=0.98. But you want to know P(D|P), which may be quite different. It turns out you do not have enough information yet. Oddly enough you must also know P(D), the prevalence of the disease in your population. Why? As in the case of the fires and fire alarms, if the disease is rare, then false positives will dominate true ones. If the disease is common, true positives will dominate false ones.


19

Bayes’ Rule

• Example (medical testing)– Suppose the disease is rare, occurring in only 0.05% of the

population. Then applying the second form of Bayes’ we get

%7.4047.001.0*9995.098.0*0005.0

98.0*0005.0

)|()()|()(

)|()()|(

HPPHPDPPDP

DPPDPPDP


20

Bayes’ Rule

• Thus with a positive test your chance of having the disease is still just below 5%. Why? Roughly speaking among 2000 randomly chosen people you expect to have 20 positive tests but only 1 person with the disease. Thus about 95% of your positives are false. Still this is a dramatic increase in the probability of having the disease, from 0.05% to 4.7%, almost a hundredfold increase.


21

Bayes’ Rule

• On the other hand, suppose the disease is common. Suppose 10% of people in your “population” have the disease. Then

%6.91916.001.0*9.098.0*1.0

98.0*1.0

)|()()|()(

)|()()|(

HPPHPDPPDP

DPPDPPDP


22

Bayes’ Rule

• Now the positive test gives you over a 90% chance of having the disease. How can this be? Now among 1000 people, 100 will have the disease and 98 of them will test positive. Similarly 900 will not have the disease and 9 of them will test positive. Now fewer than 10% of your positives are false.


23

Revision of Probabilities by Bayes

• Is this all nonsense? Do not the test results speak for themselves? Do they not give the same information regardless of who takes them and how many people have the disease? No! The subtle but crucial point is that Bayes’ Rule lets us revise a probability based on new evidence. Revision implies the existence of a prior probability to be revised. The new probability depends not only on the evidence but also on our prior estimate of the probability. If we already know an event is likely, then evidence in its favor may make it nearly certain. The same evidence will be less compelling, however, if we know the event to be inherently unlikely. The less likely the event, the stronger the evidence must be in order to make it probable.


24


• For example, suppose your neighbor goes to play in the U.S. Open chess tournament (a big tournament drawing hundreds or thousands of players). Later you overhear on the radio that someone from your town won the tournament. Do you excitedly call up your neighbor to find out if he won? It depends on your original estimate of his likelihood of winning. If he is a weak player with no realistic chance of winning, then you probably call him up to find out who did win. If he is one of the strongest players in the county, one with serious chances of winning the tournament, you call up excitedly to see if he is in fact the winner.


25


• More simply, a positive test for AIDS is more worrisome for a promiscuous drug addict than a chaste person who avoid drugs. The evidence is more compelling when revising a probability you know to be likely than when revising one you know to be unlikely.


26

Bayes Rule Extended

• Theorem 2.19 (Bayes’ Rule for multiple hypotheses) Suppose you have n events (hypotheses) that partition the sample space S and you have an event (evidence) E in S. Then for i between 1 and n,

nHH ,,1

)|()()|()(

)|()()|(

11 nn

iii HEPHPHEPHP

HEPHPEHP


27

Bayes Rule Extended

• Proof: This is identical to the proof of the extended form of Bayes’ Rule except that you partition E into n blocks by dividing it among the H’s. Then you use this partition to expand the denominator P(E) in the simple form of Bayes’ Rule.


28

Bayes Rule Extended

• Example– At a college, 40% of the students are freshmen, 25%

sophomores, 20% juniors, and 15% seniors (These are your H’s, partitioning the whole population of students). Among students on the honor roll (your evidence E), 5% are freshmen (the percentage of E among freshmen), 10% are sophomores, 18% are juniors, and 22% are seniors. What percentage of honor roll students are sophomores? By Bayes’ Rule the probability is .

%9.21219.022.0*15.018.0*2.01.0*25.005.0*4.0

1.0*25.0


29

Bayes Rule Conclusion

• Note: Theorem 2.18 and remark 2 on p. 50 are interesting but not essential to our work. In particular you may want to look at remark 3 before Theorem 2.18 if you have ever wondered what “odds” are and how they relate to probability.


30

Statistical Interlude

• Recall that one of the crucial jobs of statistics is to help people summarize data to look for the real information and patterns to be found in it. A couple lectures ago we talked about using a single number to summarize a collection of numbers. In particular we looked at various numbers that might do this job (mean, median, mode), and we looked at data resistant to summary (e.g., phone books). The key is to make sure the summary is clear and accurate, just as you would in writing a summary of a book.


31

Statistical Interlude

• God, in His wisdom, has designed the human eye to see certain relationships easily. Statisticians take advantage of this by drawing pictures of data that the eye can easily scan.

• One of the most powerful such tools is the histogram. – To make a histogram one must have a collection of “interval”

or “ratio” data. Typically this means that the data represent counts or measurements.


32

Histograms

– One takes the range of the data and partitions it into a convenient number of subranges, usually of equal width. Then one counts the number of data values falling into each of these ranges (frequency classes or bins). Next one draws a horizontal axis containing the range of data values and a vertical axis labeled frequency with values going as high as that of the most frequent (modal) class. Finally one draws over every subrange of values a bar whose height (or area) represents the frequency with which data falls into that subrange. Usually the bars touch without overlap to indicate that they partition the data values (i.e., nothing falls in the cracks).


33

Histograms

– The histogram quickly yields much information about the data, indicating what values are typical, what values are uncommon, and whether any values are extremely different from the main body of values. This information is obvious to the eye which easily compares heights and areas of rectangles.

– To convert a frequency histogram into a relative frequency histogram, simply divide every frequency by the total number of data values. Now the heights of the bars represent the proportion of data falling into each subrange. This is useful in making histograms from populations of different sizes visually comparable.


34

Histograms

• Example: The website http://www.amstat.org/publications/jse/jse_data_archive.html has a file poverty.dat (described in poverty.txt) with information from 1990 on birth, death, and infant mortality rates, and life expectancies of men and women in 97 countries around the world. Let us begin by looking at the data on women’s life expectancies.


35

Histograms

• One challenge in constructing a histogram is to decide how many bars to have or, equivalently, to decide how wide to make the subranges. Too few bars makes the histogram coarse and blocky, obscuring patterns and crucial details. Too many bars produce a snaggletoothed histogram with lots of little gaps and bars of height one. The ideal normally lies somewhere in between. Here are attempts that divide women’s life expectancy into 1-year, 10-year, and 2.5 year subranges. The last seems to work best.


36

Women’s Life Expectancies Life Expectancy of Women in 97 Countries

0

1

2

3

4

5

6

7

8

40 43 46 49 52 55 58 61 64 67 70 73 76 79 82

Age

Fre

qu

ency

Female Life Expectancy


37


0

5

10

15

20

25

30

35

40

45

40 50 60 70 80 90

Age

Freq

uen

cy



38


0

2

4

6

8

10

12

14

Age

Freq

uen

cy



39

Women’s Life Expectancies

• All three histograms are valid and somewhat useful, but the last seems to give us the clearest sense of the “shape” of the data. What do we see? All the values fall between about 42 and 83. Values in the upper 70’s and low 80’s are most common. Higher values drop off very quickly. Perhaps this suggests a large number of nations share about equally in the benefits of modern medicine and public health policy. On the other hand the frequencies decrease rather uniformly from 75 down to 42, with perhaps a small gap in the 60’s. It would be interesting to look for anything the nations with life expectancies below 60 have in common.


40

Comparing Two Sets of Data

• It is also interesting to compare the life expectancy data for men and women. This involves trying to show two histograms at once in such a way as to make their data visually comparable. One of the most straightforward approaches is simply to make separate histograms for men and women using the same horizontal and vertical scales and then display the two histograms, one above the other. Here are some other approaches:


41

Men & Women’s Life ExpectanciesLife Expectancy in 97 Countries by Sex

0

2

4

6

8

10

12

14

37.5 40 42.5 45 47.5 50 52.5 55 57.5 60 62.5 65 67.5 70 72.5 75 77.5 80 82.5

Age

Fre

qu

en

cy

Male Life Expectancy Female Life Expectancy


42


0

5

10

15

20

25

37.5 40 42.5 45 47.5 50 52.5 55 57.5 60 62.5 65 67.5 70 72.5 75 77.5 80 82.5

Age

Fre

qu

en

cy



43

Men & Women’s Life Expectancies

0

2

4

6

8

10

12

14

Fre

qu

en

cy

37.5 42.5 47.5 52.5 57.5 62.5 67.5 72.5 77.5 82.5

Male Life Expectancy


Age

Life Expectancy in 97 Countries by Sex



44


0

2

4

6

8

10

12

14

37.5 40 42.5 45 47.5 50 52.5 55 57.5 60 62.5 65 67.5 70 72.5 75 77.5 80 82.5

Age

Fre

qu

en

cy



45


• The point of all these displays of the same data is to show that one can take many creative approaches to studying the same data and communicating what is there.


46


• One interesting apparent pattern is a tendency of women to live longer than men. How can we investigate this further?– A natural approach is to construct a histogram of the

difference between women’s and men’s life expectancies in each country. That produces the following histogram.


47

Men & Women’s Life ExpectanciesFemale Minus Male Life Expectancy in 97 Countries

0

5

10

15

20

25

30

-3 -2 -1 0 1 2 3 4 5 6 7 8 9 10

Years

Fre

qu

en

cy


48


• From this histogram we see one nation standing out in having men live about two years longer than women. It would be interesting to find out what the story is there. Otherwise women live longer than men in almost every nation, with the bulk of the differences being between three and seven years. This has some interesting implications in terms of how societies will deal with large numbers of widows. It also suggests (not proves) that death from childbirth is far less common worldwide than it once was.


49


• A different approach to this matter calls for a scatterplot, a graph in which the average life expectancies for men and women in each country are treated as an ordered pair and graphed. This is a powerful tool for spotting relationships between sets of data, as the following graph shows.


50

Men & Women’s Life ExpectanciesMale vs. Female Life Expectancy in 97 Countries,

1990

30

40

50

60

70

80

90

30 40 50 60 70 80 90

Male Life Expectancy

Fem

ale

Lif

e E

xp

ectan

cy


51


• Here we see a strong, positive "linear relationship”. The data roughly fit along a line (not the line shown). Generally if men live longer in a country, then so do women. The line is not a regression line. It is simply the line y=x, showing those countries in which men and women have equal life expectancies. Dots above the line indicate nations where women live longer than men. Only a very few dots lie below the line, confirming the pattern we saw in the previous graph.

lec05 product rule and bayes' rule

Documents