University of the Punjab, College of Information Technology

CS-501 (Fall 2013): COMPUTATIONAL MODELLING

by S. W. Jaffry Tuesday, 12 November 2013

Lecture 05: Input Modelling, Random Numbers and Distributions

This lecture presents some basic definitions and concepts about random number variates
generation.

1.1 Input Modelling

 Independent and identically distributed Samples (IID), distribution should be simple

(unimodel) and stationary.
 Direct use of Sample Data: Collect empirical data and use it for simulation purposes as
input data
 Create an Empirical Distribution and use it for model input
 Map Empirical Distribution on a theoretical distribution and use theoretical distribution for
model input
 Use of theoretical distribution for model input (parametric: exponential-1, normal-2 and
triangular-3)

1.2 Pseudo Random Number:

 No real random number exists

 Real Random Number (depends on natural phenomena, like radioactive decay, thermal noise
from a semiconductor, human computer interaction process keyboard and mouse activity)
 Pseudo Random Number (based on computer algorithm, deterministic, seed value, can be
reproduced, statistical test confirms randomness)
 Three types of random numbers, Boolean (Bernoulli Trial), Discrete and Continuous

Random Number Generator must possess following properties

 uniformly distributed
 statistically independent
 reproducible, and
 non-repeating for any desired length.

Von Neuman’s mid-square method: 5772156649, 33317792380594909201, 7923805949,

62786700717407790601 relatively slow and statistically unsatisfactory.

1.2.1 Linear Congruential Generator (LCG) or residue method (Severence 2009)

X(0)= seed
Mixed congruential method, X(i+1) = ( a * X(i) + c ) mod m, R(i) = X(i)/m

multiplicative congruential X(i+1) = a*X(i) mod m where c = 0

additive congruential X(i+1) = (X(i) + c ) mod m where a = 1

multiplicative is better than additive and mixed is not better than multiplicative so X(i+1) = a*X(i) mod m

X0 = 0, a = c = 7 and m = 10.

7, 6, 9, 0, 7, 6, 9, 0

it has period 4, if period is equal to m then generator has full period.

X0 = 7, a = 5, c = 3 and m = 16.

X(i) = {a^i * X(0) + c * (a^i - 1)/(a-1) } mod m, X(1) = {7*0 + 7 (6)/6} = 7

Random numbers are from 0 to m – 1.

Hull Dobell Theorem: Condition for full period:

 a and c have no common divisor (a and c are relatively prime)

 if r is prime factor of m and it also divides a–1 (all prime numbers that divides m also divide a – 1)
 if 4 divides m then 4 also divide a–1

a = 314, 159, 269 c = 453, 806, 245 and m=2^32

x(i+1) = f(x(i), x(i-1), …) (mod m)

e.g. x(i+1) = a(1) * x(i)^2 + a(2) * x(i-1) + c mod m

here if a(1) = a(2) = 1, c = 0 and m being a power of 2

additive congruential methods f(x(i), x(i-1), …) = a(1) * x(i) + a(2) * x(i-1) + . . . + a(k) * x(i-k)

1.2.2 Statistical Test for Pseudo Random Number Generator

[1-3] check randomness of a sequence of bits
[4-5] check goodness of fit

1. equidistribution or frequency test (fundamental test, number of zeros and one in bit
concatenates)

2. serial test (k bits can be combined in 2^k different ways, whether the number of times each of
these combinations occurs in uniform)

The equidistribution test examines whether an individual element in a sequence comes up

more often than it should, but we would also like pairs of successive numbers to be uniformly
and independently distributed (for example, in a binary sequence, (0,0), (0,1), (1,0) and (1,1)
should all be equally likely). As Knuth eloquently puts it, “the sun comes up just about as
often as it goes down, in the long run, but this doesn't make its motion random". To do this,
 count the number of times that the pair (Y2j; Y2j+1) = (q; r) occurs for 0 <= j < n. This should
be done for each pair of integers (q; r) with
2 2 2
0 <= q; r < d. We then have d categories for a χ test, with a probability of 1/d assigned to
each category.

3. autocorrelation test (shift bits and compare how does it relates with new bit set, if it is similar
then sequence is not random)
4. runs test (test the assumption that the pseudo-random numbers are independent of each
other)
5. chi-square test for goodness of fit (check whether an empirical distribution follows a specific
theoretical distribution).
2
The chi-square (χ ) test was initially published by Karl Pearson in 1900. Pearson's original
2
notation in explaining the theory behind the test included use of the χ symbol; hence the
name. This test can be used in many situations and basically, when given an outcome of an
experiment, can give an approximate probability as to how likely that outcome is [Knuth 39,
52, 54].

Suppose we had n independent observations (in our case, perhaps elements of a sequence
generated by a possible RNG), each falling into one of k categories. Let Ys be the number of
observations falling into the sth category and ps be the probability that an observation falls
into category s. Then we would expect that

Ys = nps

for large values of n. We'd like a way to measure 'how far off' we are from these expected
values, so define the reasonable statistic Vi (the i here stands for 'inadequate,' as we'll shortly
see) as the following
2 2 2
Vi = (Y1 - np1) + (Y2 - np2) + - - - + (Yk - npk)

(the squaring is to make each term positive; without this, discrepancies could 'balance out').
This will give some measure as to how close the actual results are to the expected.

Note that Vi gives equal weight to each category. If not every ps is the same, then Vi can be
misleading because it could 'overemphasize' some discrepancies and 'hide' others. We
modify the statistic to the one that's actually used, V , which gives the correct weight to each
term. V (called the chi-square statistic of Y1,...,Yk [Knuth 40]) is defined as
2 2 2 2
V = (Y1 - np1) / np1 + (Y2 - np2) / np2 + - - - + (Yk - npk) / npk = Σ (Ys - nps) / nps

This solves the weight problem (if only it were so easy for humans). V can be written another
way, which is often easier to compute. We take advantage of two fairly obvious facts:

Y1 + Y2 + ::: + Yk = n and p1 + p2 + ::: + pk = 1

2
V = 1/n Σ (Ys / ps) – n

1.2.3 Hypothesis testing

Various statistical tests can be applied to a sequence to attempt to compare and evaluate the sequence
to a truly random sequence. Randomness is a probabilistic property; that is, the properties of a random
sequence can be characterized and described in terms of probability. The likely outcome of statistical
tests, when applied to a truly random sequence, is known a priori and can be described in probabilistic
terms. There are an infinite number of possible statistical tests, each assessing the presence or absence
of a “pattern” which, if detected, would indicate that the sequence is nonrandom. Because there are so
many tests for judging whether a sequence is random or not, no specific finite set of tests is deemed
“complete.” In addition, the results of statistical testing must be interpreted with some care and caution to
avoid incorrect conclusions about a specific generator (see Section 4).

A statistical test is formulated to test a specific null hypothesis (H0). For the purpose of this document,
the null hypothesis under test is that the sequence being tested is random. Associated with this null
hypothesis is the alternative hypothesis (Ha), which, for this document, is that the sequence is not
random. For each applied test, a decision or conclusion is derived that accepts or rejects the null
hypothesis, i.e., whether the generator is (or is not) producing random values, based on the sequence
that was produced.

For each test, a relevant randomness statistic must be chosen and used to determine the acceptance or
rejection of the null hypothesis. Under an assumption of randomness, such a statistic has a distribution of
possible values. A theoretical reference distribution of this statistic under the null hypothesis is
determined by mathematical methods. From this reference distribution, a critical value is determined
(typically, this value is "far out" in the tails of the distribution, say out at the 99 % point). During a test, a
test statistic value is computed on the data (the sequence being tested). This test statistic value is
compared to the critical value. If the test statistic value exceeds the critical value, the null hypothesis for
randomness is rejected. Otherwise, the null hypothesis (the randomness hypothesis) is not rejected (i.e.,
the hypothesis is accepted).

In practice, the reason that statistical hypothesis testing works is that the reference distribution and the
critical value are dependent on and generated under a tentative assumption of randomness. If the
randomness assumption is, in fact, true for the data at hand, then the resulting calculated test statistic
value on the data will have a very low probability (e.g., 0.01 %) of exceeding the critical value. On the
other hand, if the calculated test statistic value does exceed the critical value (i.e., if the low probability
event does in fact occur), then from a statistical hypothesis testing point of view, the low probability
event should not occur naturally. Therefore, when the calculated test statistic value exceeds the critical
value, the conclusion is made that the original assumption of randomness is suspect or faulty. In this
case, statistical hypothesis testing yields the following conclusions: reject H0 (randomness) and accept
Ha (non-randomness).

Statistical hypothesis testing is a conclusion-generation procedure that has two possible outcomes, either
accept H0 (the data is random) or accept Ha (the data is non-random). The following 2 by 2 table relates
the true (unknown) status of the data at hand to the conclusion arrived at using the testing procedure.

Decision
Real Situation H0 is not rejected H0 is rejected
H0 is true Valid Type I error (false negative)
Ho is not true Type II Error (false positive) Valid

Null hypothesis (H0) and alternative hypothesis (Ha)

Type I error (false negative, measured by alpha, also known as level of significance usually 0.01 or 0.05),
Type II error (false positive)

If the data is, in truth, random, then a conclusion to reject the null hypothesis (i.e., conclude that the data
is non-random) will occur a small percentage of the time. This conclusion is called a Type I error. If the
data is, in truth, non-random, then a conclusion to accept the null hypothesis (i.e., conclude that the data
is actually random) is called a Type II error. The conclusions to accept H0 when the data is really random,
and to reject H0 when the data is non-random, are correct.

The probability of a Type I error is often called the level of significance of the test. This probability can
be set prior to a test and is denoted as α. For the test, α is the probability that the test will indicate that the
sequence is not random when it really is random. That is, a sequence appears to have non-random
properties even when a “good” generator produced the sequence. Common values of α in cryptography
are about 0.01.

The probability of a Type II error is denoted as β. For the test, β is the probability that the test will
indicate that the sequence is random when it is not; that is, a “bad” generator produced a sequence that
appears to have random properties. Unlike α, β is not a fixed value. β can take on many different values
because there are an infinite number of ways that a data stream can be non-random, and each different
way yields a different β. The calculation of the Type II error β is more difficult than the calculation of α
because of the many possible types of non-randomness.

One of the primary goals of the following tests is to minimize the probability of a Type II error, i.e., to
minimize the probability of accepting a sequence being produced by a generator as good when the
generator was actually bad. The probabilities α and β are related to each other and to the size n of the
tested sequence in such a way that if two of them are specified, the third value is automatically
determined. Practitioners usually select a sample size n and a value for α (the probability of a Type I error
– the level of significance). Then a critical point for a given statistic is selected that will produce the
smallest β (the probability of a Type II error). That is, a suitable sample size is selected along with an
acceptable probability of deciding that a bad generator has produced the sequence when it really is
random. Then the cutoff point for acceptability is chosen such that the probability of falsely accepting a
sequence as random has the smallest possible value.

Each test is based on a calculated test statistic value, which is a function of the data. If the test statistic
value is S and the critical value is t, then the Type I error probability is P(S > t || Ho is true) = P(reject Ho |
H0 is true), and the Type II error probability is P(S ≤ t || H0 is false) = P(accept H0 | H0 is false). The test
statistic is used to calculate a P-value that summarizes the strength of the evidence against the null
hypothesis. For these tests, each P-value is the probability that a perfect random number generator would
have produced a sequence less random than the sequence that was tested, given the kind of non-
randomness assessed by the test. If a P-value for a test is determined to be equal to 1, then the
sequence appears to have perfect randomness. A P-value of zero indicates that the sequence appears to
be completely non-random. A significance level (α) can be chosen for the tests. If P-value ≥α, then the
null hypothesis is accepted; i.e., the sequence appears to be random. If P-value <α, then the null
hypothesis is rejected; i.e., the sequence appears to be non-random. The parameter αdenotes the
probability of the Type I error. Typically, α is chosen in the range [0.001, 0.01].

 An α of 0.001 indicates that one would expect one sequence in 1000 sequences to be rejected by
the test if the sequence was random. For a P-value ≥ 0.001, a sequence would be considered to
be random with a confidence of 99.9%. For a P-value < 0.001, a sequence would be considered
to be nonrandom with a confidence of 99.9%.
 An α of 0.01 indicates that one would expect 1 sequence in 100 sequences to be rejected. A P-
value ≥ 0.01 would mean that the sequence would be considered to be random with a confidence
of 99%. A P-value < 0.01 would mean that the conclusion was that the sequence is non-random
with a confidence of 99%.
For the examples in this document, α has been chosen to be 0.01. Note that, in many cases, the
parameters in the examples do not conform to the recommended values; the examples are for illustrative
purposes only.

What is the difference between an alpha level and a p-value?

Science sets a conservative standard to meet for a researcher to claim that s/he has made a discovery of a real
phenomenon. The standard is the alpha level, usually set of 0.05.
Assuming that the null hypothesis is true, this means we may reject the null only if the observed data are so
unusual that they would have occurred by chance at most 5% of the time. The smaller the alpha, the more
stringent the test (the more unlikely it is to find a statistically significant result).

Once the alpha level has been set, a statistic (like r) is computed. Each statistic has an associated probability
value called a p-value, or the likelihood of an observed statistic occurring due to chance, given the sampling
distribution.

Alpha sets the standard for how extreme the data must be before we can reject the null hypothesis. The p-value
indicates how extreme the data are. We compare the p-value with the alpha to determine whether the observed
data are statistically significantly different from the null hypothesis:
If the p-value is less than or equal to the alpha (p< .05), then we reject the null hypothesis, and we say the result
is statistically significant.

If the p-value is greater than alpha (p > .05), then we fail to reject the null hypothesis, and we say that the result is
statistically nonsignificant (n.s.).

1.3 Generating random variates

1.3.1 Continuous time distributions

1.3.1.1 Uniform Distribution

X = a + (a – b) * r

1.3.1.2 Exponential Distribution

exponential distribution (a.k.a. radical exponential distribution) is the probability distribution that
describes the time between events in a Poisson process, i.e. a process in which events occur
continuously and independently at a constant average rate.

X = - 1/a ln (r)

1.3.1.3 Normal Distribution

z = (x – meu) / sigma
meu +- 1sigma = 68%, meu +- 2sigma = 95%, meu +- 3sigma = 99.7%

if grads of a class form a normal distribution with mean 70 and standard deviation 6 what is probability to
have grad higher than 78 in this class? z = (78-70)/6 = 1.33,  0.0918

X = sigma * SQRT (12/k) * (summation r(i) – k/2) + meu

K should be large minimum value might be 10, but it has good computational efficiency for k=12

Box Muller Method:

X1 = SQRT (-2 ln (r1) ) * cos ( 2 * pi * r2)

X2 = SQRT (-2 ln (r1) ) * sin ( 2 * pi * r2)

1.3.2 Discrete Time Distributions:

1.3.3 Geometric Distribution

It’s the probability that the first occurrence of success require k number of independent trials, each
with success probability p. If the probability of success on each trial is p, then the probability that the kth
trial (out of k trials) is the first success is

for k = 1, 2, 3, ....
The above form of geometric distribution is used for modeling the number of trials until the first success.
By contrast, the following form of geometric distribution is used for modeling number of failures until the
first success:

for k = 0, 1, 2, 3, ....

Number of failures to occur before a success will occur

n = ln (r) / ln (1 - p)

where 1 – p is probability of a single failure

1.3.4 Binomial Distribution

In probability theory and statistics, the binomial distribution is the discrete probability distribution of the
number of successes in a sequence of n independent yes/no experiments, each of which yields success
with probability p.
Toss a coin three times TTT, TTH, THT, HTT, THH, HTH, HHT, HHH probability of three time head and
tail is 1/8 for each and two time head or tail is 3/8 for each. If we throw a die three times the probability for
two to come 0, 1, 2 and 3 times is as follows

P(0) = 125/216, P(1) = 75/216, P(2) = 15/216, P(3) = 1/216

If p is the probability of two and q for probability not having two then
0 1 2 3
P(x) q^3 2*q^2*p 3*q*p^2 p^3
Which forms binomial distribution: P(x) = nCx * p^x * q^(n-x)
Where x is the number of occurrences of an event and n is total number of trial for example
P(0) = 3C0 * (1/6)^0 * (5/6)^3
= 3!/0!(3-0)! * 1 * 125/216
= 1*125/216
= 125/216
What is the probability of showing single 1 in three tosses of a die? 25/72
If a die is tossed five times find the probabilities for a number to appear exactly 0, 1, 2, 3, 4, 5 in an
experiment?
Probability of at least four three showing? Hint { x >= 4 }
If a die is tossed eight times what is the probability the four will show exactly twice?
If a die is tossed five times what is the probability the three will show at most three times? {x<=3}

Properties for binomial nature problems:

a) number of trials must be fixed,
b) each trial must result either in success or failure,
c) probability of success must be identified,
d) all of the trial must be independent.

p is probability of success and q is probability of failure. Let X be number of success in n trials (Bernoulli
trials).

k(i) = k(i-1) + 1 if ri < p

k(i-1) if ri > p

this method of generating variates is called rejection method as well.

1.3.5 Poisson Distribution:

The Poisson distribution models the occurrence of a particular event over a time period. Let λ be the
average number of occurrences during a unit time period. Then, the number of occurrence x during a unit
-λ n
period has the following probability density function p(n) = e (λ /n!), n = 0, 1, 2, . . .

When more than two outcomes are possible (not just HT, TF) and when number of trail are very large and
probability of success in a trial is very small.
P(x) = e^(-m) * m^x / x! Where x is 0, 1, 2, 3, . . .
X is number of occurrences per unit and m is average of mean of occurrences per unit.

Requirements: An occurrences in one unit does not affect occurrences in another unit, and mean must
remain constant.
Example: if there are two defective resistors in a box on average what is the probability that there will be
no defective resistor in a give box. x = 0, m = 2, e = 2.7
P(0) = 2.7^(-2) * 2^0 / 0! = 0.137
Probability of a box having three defective transistors: 0.18
If on average one person enters a store every 5 sec. and person enters on random, what is the probability
that in a selected 5 sec-period of time three person will enter the store? 0.06

Standard error in sample = sigma / sqrt (N)

1.4 Assigned Readings

Chapter 3 from (Severence 2009)

For more about distributions see following link:
http://www.itl.nist.gov/div898/handbook/eda/section3/eda366.htm

1.5 Exercise:

E01: This is a more involved exercise! Implement the random number generators described
in this lecture, and then test them using different statistical tests. Measure the execution time for each
generator for the same output size. Present your results in a table that gives the execution speed of each
generator and whether it has failed or passed each statistical test. Discuss your results. What is your
favorite generator for your simulation assignments?

1.6 References

Severence, F. L. (2009). System Modeling And Simulation: An Introduction, Wiley India Pvt. Limited.