Illustration of the central limit theorem

This article gives two concrete illustrations of the central limit theorem. Both involve the sum of independent and identically-distributed random variables and show how the probability distribution of the sum approaches the normal distribution as the number of terms in the sum increases.

The first illustration involves a continuous probability distribution, for which the random variables have a probability density function.

The second illustration, for which most of the computation can be done by hand, involves a discrete probability distribution, which is characterized by a probability mass function.

A free full-featured interactive simulation that allows the user to set up various distributions and adjust the sampling parameters is available through the External links section at the bottom of this page.

Contents

Illustration of the continuous case

The density of the sum of two independent real-valued random variables equals the convolution of the density functions of the original variables.

Thus, the density of the sum of m+n terms of a sequence of independent identically distributed variables equals the convolution of the densities of the sums of m terms and of n term. In particular, the density of the sum of n+1 terms equals the convolution of the density of the sum of n terms with the original density (the "sum" of 1 term).

A probability density function is shown in the first figure below. Then the densities of the sums of two, three, and four independent identically distributed variables, each having the original density, are shown in the following figures. If the original density is a piecewise polynomial, as it is in the example, then so are the sum densities, of increasingly higher degree. Although the original density is far from normal, the density of the sum of just a few variables with that density is much smoother and has some of the qualitative features of the normal density.

The convolutions were computed via the discrete Fourier transform. A list of values y = f(x0 + k Δx) was constructed, where f is the original density function, and Δx is approximately equal to 0.002, and k is equal to 0 through 1000. The discrete Fourier transform Y of y was computed. Then the convolution of f with itself is proportional to the inverse discrete Fourier transform of the pointwise product of Y with itself.


A probability density function.

Original probability density function

We start with a probability density function. This function, although discontinuous, is far from the most pathological example that could be created. It is a piecewise polynomial, with pieces of degrees 0 and 1. The mean of this distribution is 0 and its standard deviation is 1. 4545

Density of a sum of two variables.

Probability density function of the sum of two terms

Next we compute the density of the sum of two independent variables, each having the above density. The density of the sum is the convolution of the above density with itself.

The sum of two variables has mean 0. The density shown in the figure at right has been rescaled by √2, so that its standard deviation is 1.

This density is already smoother than the original. There are obvious lumps, which correspond to the intervals on which the original density was defined.


Density of a sum of three variables.

Probability density function of the sum of three terms

We then compute the density of the sum of three independent variables, each having the above density. The density of the sum is the convolution of the first density with the second.

The sum of three variables has mean 0. The density shown in the figure at right has been rescaled by √3, so that its standard deviation is 1.

This density is even smoother than the preceding one. The lumps can hardly be detected in this figure.


Density of a sum of four variables

Probability density function of the sum of four terms

Finally, we compute the density of the sum of four independent variables, each having the above density. The density of the sum is the convolution of the first density with the third.

The sum of four variables has mean 0. The density shown in the figure at right has been rescaled by √4, so that its standard deviation is 1.

This density appears qualitatively very similar to a normal density. No lumps can be distinguished by the eye.

Illustration of the discrete case

This section illustrates the central limit theorem via an example for which the computation can be done quickly by hand on paper, unlike the more computing-intensive example of the previous section.

Original probability mass function

Suppose the probability distribution of a discrete random variable X puts equal weights on 1, 2, and 3:

X=\left\{\begin{matrix} 1 & \mbox{with}\ \mbox{probability}\ 1/3, \\
2 & \mbox{with}\ \mbox{probability}\ 1/3, \\
3 & \mbox{with}\ \mbox{probability}\ 1/3.
\end{matrix}\right.

The probability mass function of the random variable X may be depicted by the following bar graph:

    o    o    o
   -------------
    1    2    3

Clearly this looks nothing like the bell-shaped curve of the normal distribution. Contrast the above with the depictions below.

Probability mass function of the sum of two terms

Now consider the sum of two independent copies of X:

\left\{\begin{matrix}
1+1 & = & 2 \\
1+2 & = & 3 \\
1+3 & = & 4 \\
2+1 & = & 3 \\
2+2 & = & 4 \\
2+3 & = & 5 \\
3+1 & = & 4 \\
3+2 & = & 5 \\
3+3 & = & 6
\end{matrix}\right\}
=\left\{\begin{matrix}
2 & \mbox{with}\ \mbox{probability}\ 1/9 \\
3 & \mbox{with}\ \mbox{probability}\ 2/9 \\
4 & \mbox{with}\ \mbox{probability}\ 3/9 \\
5 & \mbox{with}\ \mbox{probability}\ 2/9 \\
6 & \mbox{with}\ \mbox{probability}\ 1/9
\end{matrix}\right\}

The probability mass function of this sum may be depicted thus:

              o
         o    o    o
    o    o    o    o    o
   ----------------------------
    2    3    4    5    6

This still does not look very much like the bell-shaped curve, but, like the bell-shaped curve and unlike the probability mass function of X itself, it is higher in the middle than in the two tails.

Probability mass function of the sum of three terms

Now consider the sum of three independent copies of this random variable:

\left\{\begin{matrix}
1+1+1 & = & 3 \\
1+1+2 & = & 4 \\
1+1+3 & = & 5 \\
1+2+1 & = & 4 \\
1+2+2 & = & 5 \\
1+2+3 & = & 6 \\
1+3+1 & = & 5 \\
1+3+2 & = & 6 \\
1+3+3 & = & 7 \\
2+1+1 & = & 4 \\
2+1+2 & = & 5 \\
2+1+3 & = & 6 \\
2+2+1 & = & 5 \\
2+2+2 & = & 6 \\
2+2+3 & = & 7 \\
2+3+1 & = & 6 \\
2+3+2 & = & 7 \\
2+3+3 & = & 8 \\
3+1+1 & = & 5 \\
3+1+2 & = & 6 \\
3+1+3 & = & 7 \\
3+2+1 & = & 6 \\
3+2+2 & = & 7 \\
3+2+3 & = & 8 \\
3+3+1 & = & 7 \\
3+3+2 & = & 8 \\
3+3+3 & = & 9 
\end{matrix}\right\}
=\left\{\begin{matrix}
3 & \mbox{with}\ \mbox{probability}\ 1/27 \\
4 & \mbox{with}\ \mbox{probability}\ 3/27 \\
5 & \mbox{with}\ \mbox{probability}\ 6/27 \\
6 & \mbox{with}\ \mbox{probability}\ 7/27 \\
7 & \mbox{with}\ \mbox{probability}\ 6/27 \\
8 & \mbox{with}\ \mbox{probability}\ 3/27 \\
9 & \mbox{with}\ \mbox{probability}\ 1/27
\end{matrix}\right\}

The probability mass function of this sum may be depicted thus:

                   o
              o    o    o
              o    o    o
              o    o    o
         o    o    o    o    o
         o    o    o    o    o
    o    o    o    o    o    o    o
   ---------------------------------
    3    4    5    6    7    8    9

Not only is this bigger at the center than it is at the tails, but as one moves toward the center from either tail, the slope first increases and then decreases, just as with the bell-shaped curve.

The degree of its resemblance to the bell-shaped curve can be quantified as follows. Consider

Pr(X1 + X2 + X3 ≤ 7) = 1/27 + 3/27 + 6/27 + 7/27 + 6/27 = 23/27 = 0.85185... .

How close is this to what a normal approximation would give? It can readily be seen that the expected value of Y = X1 + X2 + X3 is 6 and the standard deviation of Y is the square root of 2. Since Y ≤ 7 (weak inequality) if and only if Y < 8 (strict inequality), we use a continuity correction and seek

\mbox{Pr}(Y\leq 7.5)
=\mbox{P}\left({Y-6 \over \sqrt{2}}\leq{7.5-6 \over \sqrt{2}}\right)
=\mbox{Pr}(Z\leq 1.0606602\dots) = 0.85558\dots

where Z has a standard normal distribution. The difference between 0.85185... and 0.85558... seems remarkably small when it is considered that the number of independent random variables that were added was only three.

Probability mass function of the sum of 1,000 terms

Central theorem 2.png

The following image shows the result of a simulation based on the example presented in this page. The extraction from the uniform distribution is repeated 1,000 times, and the results are summed.

Since the simulation is based on the Monte Carlo method, the process is repeated 10,000 times. The results shows that the distribution of the sum of 1,000 uniform extractions resembles the bell-shaped curve very well.

External links


Wikimedia Foundation. 2010.

Look at other dictionaries:

  • Central limit theorem — This figure demonstrates the central limit theorem. The sample means are generated using a random number generator, which draws numbers between 1 and 100 from a uniform probability distribution. It illustrates that increasing sample sizes result… …   Wikipedia

  • Newton's theorem of revolving orbits — Figure 1: An attractive force F(r) causes the blue planet to move on the cyan circle. The green planet moves three times faster and thus requires a stronger centripetal force, which is supplied by adding an attractive inverse cube force. The …   Wikipedia

  • Classical central-force problem — In classical mechanics, the central force problem is to determine the motion of a particle under the influence of a single central force. A central force is a force that points from the particle directly towards (or directly away from) a fixed… …   Wikipedia

  • Financial history of the Dutch Republic — describes the history of the interrelated development of financial institutions in the Dutch Republic. The rapid economic development of the country after the Dutch Revolt in the years 1585 1620, described in Economic History of the Netherlands… …   Wikipedia

  • List of statistics topics — Please add any Wikipedia articles related to statistics that are not already on this list.The Related changes link in the margin of this page (below search) leads to a list of the most recent changes to the articles listed below. To see the most… …   Wikipedia

  • List of probability topics — This is a list of probability topics, by Wikipedia page. It overlaps with the (alphabetical) list of statistical topics. There are also the list of probabilists and list of statisticians.General aspects*Probability *Randomness, Pseudorandomness,… …   Wikipedia

  • Outline of probability — Probability is the likelihood or chance that something is the case or will happen. Probability theory is used extensively in statistics, mathematics, science and philosophy to draw conclusions about the likelihood of potential events and the… …   Wikipedia

  • List of mathematical examples — This page will attempt to list examples in mathematics. To qualify for inclusion, an article should be about a mathematical object with a fair amount of concreteness. Usually a definition of an abstract concept, a theorem, or a proof would not be …   Wikipedia

  • List of mathematics articles (I) — NOTOC Ia IA automorphism ICER Icosagon Icosahedral 120 cell Icosahedral prism Icosahedral symmetry Icosahedron Icosian Calculus Icosian game Icosidodecadodecahedron Icosidodecahedron Icositetrachoric honeycomb Icositruncated dodecadodecahedron… …   Wikipedia

  • probability theory — Math., Statistics. the theory of analyzing and making statements concerning the probability of the occurrence of uncertain events. Cf. probability (def. 4). [1830 40] * * * Branch of mathematics that deals with analysis of random events.… …   Universalium


Share the article and excerpts

Direct link
Do a right-click on the link above
and select “Copy Link”

We are using cookies for the best presentation of our site. Continuing to use this site, you agree with this.