probability and statistics for engineering
DESCRIPTION
PROBABILITY AND STATISTICS FOR ENGINEERING. Hossein Sameti Department of Computer Engineering Sharif University of Technology. The Weak Law and the Strong Law of Large Numbers. Timeline…. 1700 1800 1866 1909 1926. Bernoulli, weak law of large numbers (WLLN) - PowerPoint PPT PresentationTRANSCRIPT
PROBABILITY AND STATISTICS FOR ENGINEERING
Hossein Sameti
Department of Computer EngineeringSharif University of Technology
The Weak Law and the Strong Law of Large Numbers
Timeline…
Bernoulli, weak law of large numbers (WLLN)
Poisson, generalized Bernoulli’s theorem
Tchebychev, discovered his method
Markov, used Tchebychev’s reasoning to extend Bernoulli’s theorem to dependent random variables as well.
Borel, the strong law of large numbers that further generalizes Bernoulli’s theorem.
Kolmogorov, necessary and sufficient conditions for a set of mutually independent r.vs.
1700
1800
1866
1909
1926
Weak Law of Large Numbers
s: i.i.d Bernoulli r.vs such that
: the number of “successes”in n trials.
Then the weak law due to Bernoulli states that
i.e., the ratio “total number of successes to the total number of trials”
tends to p in probability as n increases.
iX
,1)0( ,)( qpXPpXP ii
nXXXk 21
.2
npq
pP nk
Strong Law of Large Numbers
Borel and Cantelli:
this ratio k/n tends to p not only in probability, but with probability 1.
This is the strong law of large numbers (SLLN).
What is the difference?
SLLN states that if is a sequence of positive numbers converging to zero, then
From Borel-Cantelli lemma, when this formula is satisfied the events
can occur only for a finite number of indices n in an infinite sequence,
or equivalently, the events occur infinitely often, i.e., the
event k/n converges to p almost-surely.
}{ n
. 1
nnpP n
k
n
pnk
= n nknA p
Proof
Since
we have
and hence
where
pPpPnnkpnpk nk
nk
n
n
k
)()( 4444
0
4
440
4 )()(
n
kpnpkpP
n
kn
nk
knkn
iin qpkXPkp
k
n
1
)(
444 nnpkpn
k
Proof – continued
since
So we obtain
,3
)]1(3[))(1(3)(
)()()1(3)()()1(4)(
)(}){(
2
233
2
1 1
2
1 1
3
1
4
1 1 1 1
4
1
pqn
pqnnnpqnnpqqpn
YEYEnnYEYEnnYE
YYYYEYE
j
n
i
n
jij
n
i
n
ji
n
ii
n
i
n
k
n
j
n
lljki
n
ii
can coincide withj, k or l, and the second variabletakes (n-1) values
ni 1
12/1 ,133)( 22333 pqpqqpqpqp
2 4
3kn
pqP p
n
0
}{}{1
44
10
4 )( )()()()(
n
ii
n
ii
n
kn pXEnpXEkpnpk
Proof – continued
Let so that the above integral reads
and hence
thus proving the strong law by exhibiting a sequence of positive numbers
that converges to zero and satisfies
3/ 21/8 3/ 2 1
1 1
1 13 3 1
3 (1 2) 9 ,
( )n n
knP p pq pq x dx
n n
pq pq
1/81
n
. 1
nnpP n
k 1/81/n n
What is the difference?
The weak law states that for every n that is large enough, the
ratio is likely to be near p with certain probability that
tends to 1 as n increases.
It does not say that k/n is bound to stay near p if the number of trials is increased.
Suppose is satisfied for a given in a certain number of trials
If additional trials are conducted beyond the weak law does not
guarantee that the new k/n is bound to stay near p for such trials.
There can be events for which for in some regular manner.
.2
npq
pP nk
1( ) / /
n
iX n k ni
.0n
,0n
,/ pnk 0nn
What is the difference?
The probability for such an event is the sum of a large number of very small probabilities
The weak law is unable to say anything specific about the convergence of that sum.
However, the strong law states that not only all such sums converge, but the total number of all such events:
where is in fact finite!
. 1
nnpP n
k
pnk /
Bernstein’s inequality
This implies that the probability of the events as n increases becomes and remains small,
since with probability 1 only finite violations to the above inequality takes place as
It is possible to arrive at the same conclusion using a powerful bound known as Bernstein’s inequality that is based on the WLLN.
}{ pnk
.n
Bernstein’s inequality
Note that
and for any this gives
Thus
)( pnkpnk
,0 .1))(( pnke
n
k
knkknpnk
knkn
pnkknpnk
n
pnk
knkkn
qpe
qpe
qppP nk
0
))((
)(
))((
)(
}{
Bernstein’s inequality
Since for any real x,
We can obtain
2xx exe
.
)()(22222
2222
eqepe
epqeqpqepepq
pqpq
.}{2 nnepP n
k
.)(
)()(}{0
npqn
knpkqn
k
n
qepee
qepeepP kn
nk
Bernstein’s inequality
But is minimum for and hence
and hence we obtain Bernstein’s inequality
This is more powerful than Tchebyshev’s inequality as it states that the
chances for the relative frequency k /n exceeding its probability p tends to zero exponentially fast as
nn 2 2/
0. ,}{ 4/2
nepP nk
4/2
}{ nepP nk Similarly,
2 / 4 { } 2 .nk
nP p e
.n
Chebyshev’s inequality gives the probability of k /n to lie between
and for a specific n.
We can use Bernstein’s inequality to estimate the probability for k /n to
lie between and for all large n.
Towards this, let
so that
To compute the probability of note that its complement is given by
p p
pp
}{ ppy nk
n
2 / 4 ( ) { } 2c n
nnkP y P p e
,n
mny
cn
mn
cn
mnyy
)(
We have
This gives
or,
Thus k /n is bound to stay near p for all large enough n, in probability, as stated by the SLLN.
2
2
/ 4
/ 4
2( ) {1 ( )} 1 1 as
1
m
n nn m n m
eP y P y m
e
. as 1} ,{ mmnall forppP nk
2
2
2
/ 4/ 4
/ 4
2( ) ( ) 2 .
1
mc c nn n
n m n m n m
eP y P y e
e
Let Thus if we toss a fair coin 1,000 times, from the weak law
Thus on the average 39 out of 40 such events each with 1000 or more trials will satisfy the inequality
It is quite possible that one out of 40 such events may not satisfy it.
Continuing the experiment for 1000 more trials, with k successes out of
n, for it is quite possible that for few such n the above inequality may be violated.
}{ 1.021 n
k
,20001000n
Discussion
.40
11.0
21
nkP
.1.0
.2
npq
pP nk
This is still consistent with the weak law
but according to the strong law such violations can occur only a finite number of times each with a finite probability in an infinite sequence of trials,
hence almost always the above inequality will be satisfied, i.e., the
sample space of k /n coincides with that of p as
Discussion - continued
.n
2n red cards and 2n black cards (all distinct) are shuffled together to form a single deck, and then split into half.
What is the probability that each half will contain n red and n black cards?
From a deck of 4n cards, 2n cards can be chosen in different ways.
Consider the unique draw consisting of 2n red cards and 2n black cards in each half.
Example
Solution
n
n
2
4
Among those 2n red cards, n of them can be chosen in different ways;
Similarly for each such draw there are ways of choosing n black cards.
Thus the total number of favorable draws containing n red and n black cards in each half are among a total of draws.
This gives the desired probability to be
Example – continued
n
n2
n
n2
n
n
2
4
n
n2
n
n2
np
.)! ()!4(
)! 2(4
4
2
4
22
nnn
p
n
n
n
n
n
n
n
For large n, using Stirgling’s formula we get
For a full deck of 52 cards, we have which gives
For a partial deck of 20 cards, we have and
nennennenn
pnnnn
nn
n 2
] 2[)4( )4(2])2( )2(2[
444
422
,13n 221.0np
5n .3568.0n
p
Example – continued
An Experiment
20 cards were given to a 5 year old child to split them into two equal halves
the outcome was declared a success if each half contained exactly 5 red and 5 black cards.
With adult supervision (in terms of shuffling) the experiment was repeated 100 times that very same afternoon. The results are tabulated below in next slides.
Expt Number of successes
Expt Number of successes
Expt Number of successes
Expt Number of successes
Expt Number of successes
1 0 21 8 41 14 61 23 81 29
2 0 22 8 42 14 62 23 82 29
3 1 23 8 43 14 63 23 83 30
4 1 24 8 44 14 64 24 84 30
5 2 25 8 45 15 65 25 85 30
6 2 26 8 46 16 66 25 86 31
7 3 27 9 47 17 67 25 87 31
8 4 28 10 48 17 68 25 88 32
9 5 29 10 49 17 69 26 89 32
10 5 30 10 50 18 70 26 90 32
11 5 31 10 51 19 71 26 91 33
12 5 32 10 52 20 72 26 92 33
13 5 33 10 53 20 73 26 93 33
14 5 34 10 54 21 74 26 94 34
15 6 35 11 55 21 75 27 95 34
16 6 36 12 56 22 76 27 96 34
17 6 37 12 57 22 77 28 97 34
18 7 38 13 58 22 78 29 98 34
19 7 39 14 59 22 79 29 99 34
20 8 40 14 60 22 80 29 100 35
0.3437182
Results of an experiment of 100 trialsThis figure shows the convergence of k/n to p.
n
np