The reference distributions of Maurer's universal statistical test and its improved tests

Maurer's universal statistical test can widely detect non-randomness of given sequences. Coron proposed an improved test, and further Yamamoto and Liu proposed a new test based on Coron's test. These tests use normal distributions as their reference distributions, but the soundness has not been theoretically discussed so far. Additionally, Yamamoto and Liu's test uses an experimental value as the variance of its reference distribution. In this paper, we theoretically derive the variance of the reference distribution of Yamamoto and Liu's test and prove that the true reference distribution of Coron's test converges to a normal distribution in some sense. We can apply the proof to the other tests with small changes.


Introduction
Random number sequences are used in many fields such as the Monte Carlo method and information security, including cryptography. In particular, for cryptography, high randomness is required. Therefore, we have to evaluate the randomness of a given sequence or property of its generator. Randomness test is one of such evaluating method. It is a hypothesis test, and the null is that the given sequence is truly random. It is just an experimental method and cannot give proof of randomness. On the other hand, the randomness test has a merit that we can adapt the test for any sequences regardless of their generators. For instance, Tamura and Shikano used randomness tests to inspect the properties of a quantum computer developed by IBM [1].
Since we can consider numberless alternative hypotheses against the null, many randomness tests have been proposed. We do not use a single randomness test for evaluation in practice, but a set of randomness tests called a test suite. There are many test suites such as TestU01, Diehard and Dieharder [2,3,4], and NIST SP800-22 [5] is one of the most widely used test suites. SP800-22 consists of 15 types of randomness tests. Some of the tests were pointed out that are problematic, and improvements have been proposed [6,7,8,9,10,11]. This paper focuses on Maurer's universal statistical test [12], which is included in SP800-22. Almost all proposed randomness tests aim to detect non-randomness which the developers explicitly specify, but universal tests that aim to detect a wide range of non-randomness are exceptions. Maurer's universal statistical test was also developed to have such property. Coron proposed an improved test based on Maurer's test [14], but the improved test has not been adopted for SP800-22 so far. Further, Yamamoto and Liu added an optional improvement to Coron's test and state that their test can detect non-randomness more sensitively [15]. These three tests use normal distributions as their reference distributions. As a common problem of the three tests, it has not been proven that their true reference distributions are normal distributions at least approximately. In other words, these tests use normal distributions as their reference distributions without theoretical evidence, although there is no report that using normal distribution is improper as far as the authors know. This is one of the problems we address in this paper. Additionally, the expected values and variances of the normal distributions are needed to perform these tests. Deriving the theoretical values of the variances is harder than deriving the expected values. As the variance of the Maurer's test, an experimental value was at first used, and later, Coron and Naccache derived the theoretical value [13]. The method in Ref. [13] can be applied to Coron's test and derived the theoretical value of the variance of Coron's test. However, the theoretical value of the variance of Yamamoto and Liu's test has not been derived, and an experimental value is used. The lack of the theoretical value is the other problem we address in this paper.
This paper is organized as follows. In section 2, we introduce Maurer's test, Coron's test, and Yamamoto and Liu's test. In section 3, we theoretically derive the variance of the reference distribution of Yamamoto and Liu's test. Section 3 is a reconstruction of Ref. [16] and [17]. In section 4, we prove that the distribution of the test statistic of Coron's test converges to a normal distribution in some sense. The proof can be applied for Maurer's test and Yamamoto and Liu's test with minor change. Finally, we offer a conclusion.

Maurer's universal statistical test
Let us consider a case that we test a given N -bit sequence X. First, we divide X into L-bit blocks. We use the first Q blocks for initialization, and the remained K blocks for testing. Here, Q and K are required to be sufficiently large comparing with 2 L . In practical, Q ≥ 10 · 2 L and K ≥ 1000 · 2 L are suggested. For simplicity, we assume that N = L × (Q + K). Let b n (X) be the n-th block and define A n (X) as In other words, A n (X) implies the distance between the n-th block and the nearest former block which has the same value of the n-th block if exists. Using A n (X), the test statistic is described as Finally, using a normal distribution as the reference distribution (i.e., assuming that f M R (0.5) follows a normal distribution where R (p) is a random variable sequence and each element independently takes one with probability p and zero with probability 1 − p), we compute the corresponding p-value. In practical use of Maurer's test, we need the expected value and variance of the reference distribution. For sufficiently large Q, we can regard that the distributions of A n R (0.5) (n ≥ Q + 1) do not depend on n and their support is N. Then, the expected value is given as As an experimentally derived approximation of the variance, Maurer proposed that where by the same reason of deriving (4), Later, Coron and Naccache derived the theoretical formula of the variance of the reference distribution for Q → ∞ [13]. The test statistic of Maurer's test relates to entropy of the tested sequence and its universality is stated based on the fact. Maurer proved that where C is a constant and H(p) := −p ln p − (1 − p) ln(1 − p) is per bit entropy of R (p) . It was also proven that where U S is a binary sequence generated by an ergodic stationary source S which has finite memory and H S is per bit entropy S generates. Maurer gave a conjecture that but it was shown that the conjecture is false [13].

Coron's universal statistical test
Coron proposed a test which replaces the test statistic f M by where This test uses a normal distribution as its reference distribution, too. For sufficiently large Q, the expected value of f C R (p) is approximated as for any p ∈ (0, 1). Then, the expected value of the reference distribution is given as LH(0.5). For p = 1 2 , by the same way in Ref. [13], we can derive the theoretical formula of Var f C R (0.5) as Q → ∞. Based on the theoretical formula, Coron proposed an approximation for sufficient large Q, and d(L) and e(L) are given in Ref. [14].
As seen in (13), the test statistic of Coron's test relates to entropy of the tested sequence, too. Unlike the case of Maurer's test, (13) holds for finite L and this is an advantage of Coron's test.

Yamamoto and Liu's test
Yamamoto and Liu proposed to flip a part of bits in a given sequence before performing Coron's test [15]. The flipping is stochastic, and each bit x k is independently converted tox k following the rule as They stated that the flipping made Coron's test more sensitive, and showed some experimental results to detect non-randomness sensitively. It is obvious that the exact reference distribution is the distribution of f C R (p) for p = α 2 . Approximately, this test also uses a normal distribution as its reference distribution, but the expected value and variance are not equal to those of Coron's test. As we mentioned in the former subsection, (13) holds for any p ∈ (0, 1) and we can use (13) with p = α 2 as the expected value for sufficiently large Q. On the other hand, an experimentally derived value is used as the variance. Yamamoto and Liu pointed out that A n R (p) are not i.i.d, and thus to derive the theoretical variance is difficult. However, the derivation process for the variances of the reference distributions of Maurer's test and Coron's test do not need independent but just identity. Even in Yamamoto and Liu's test, identity is preserved. Thus, we expect that we can theoretically derive the variance of the reference distribution of Yamamoto and Liu's test by a similar way of the derivation process.

Derivation of the variance of the reference distribution of Yamamoto and Liu's test
In this section, we derive the theoretical formula of the variance of the reference distribution that Yamamoto and Liu's test uses and numerically evaluate the formula. To get the formula, we need the distributions of A n R (p) and the joint distributions of A n R (p) , A n+k R (p) . For simplicity, we omit to write argument R (p) for every variable. In the following, we replace the index of A n as illustrated in Figure 1 and consider the case that Q → ∞. Then, the sequence of {A n } K n=1 exactly follows a stationary ergodic process, that is, the joint distribution of {A n } k+m n=k depends only on m, and its support fully covers N m .

The distribution of A n
We consider the event of M := A n = i for i ≥ 1. As illustrated in Figure 2, M is equivalent to the n-th block coincides (n − i)-th block and do not coincide other blocks between n-th and (n − i)-th blocks, i.e., Let ℓ(b) be the number of "1" included in a block b ∈ {0, 1} L . We have where w r = p r (1 − p) L−r and L r = L! r!(L−r)! is a binomial coefficient. Then, we obtain

The joint distribution of A n and A n+k
We derive the probability of the event A n = i, A n+k = j for i ≥ 1 and j ≥ 1. Depending on k, i and j, we consider five cases.
there is no overlapping between (n − i)-th to n-th blocks and (n + k − j)-th to (n + k)-th blocks as illustrated in Figure 3. Then, we obtain   Figure 4. The event e 2 (b) can be written as Then, we have where r = l(b). If b = b ′ , then it is obvious that Then, since A n = i, A n+k = j = b∈B L e 2 (b) and the r.h.s. of (25) depends only on l(b), we obtain where  Figure 5. The event e 3 (b, b ′ ) can be written as where r 1 = l(b) and r 2 = l(b ′ ). Let h 3 (r 1 , r 2 ) be the r.h.s. of (34). Since 3.2.4 Case 4: j = k + i As illustrated in Figure 6, it is obvious that 3.2.5 Case 5: j ≥ k + i + 1 We fix b, b ′ ∈ B L , and consider the event Figure 7. The event e 5 (b, b ′ ) is written as If b = b ′ , then it is obvious that where r 1 = l(b) and r 2 = l(b ′ ). Let h 5 (r 1 , r 2 ) be the r.h.s. of (34). Since , by the same way in case 3, we obtain Here, by (13), we have Substituting (21)

Numerical analysis
We numerically analyzed Var[f C ] based on the above theoretical analysis.

Experiment 1
We computed Var[f C ] for p = 0.33, 0.4 and 0.5, where L = 4. Note that p = 0.33 is suggested in Ref. [15]. Figure 8 shows the results. We can say that Var[f C ] is approximately proportional to 1 K . Table 1 shows D K (p) described as Var[f C ] = D K (p) K . Figure 9 shows comparison between the theoretical values of Var[f C ] and the corresponding experimentally derived values. The detail of the experiment is as follows: 1. Generate 1000 sequences using Mersenne twister [18].
2. Stochastically flip the bits of the sequences following (15) and (16)  Here, we set L = 4 and Q = 10 × 2 L . As a result, we can confirm that the numerical value based on our theory is valid with high accuracy.
where ζ and η are constants. We estimated the values of ζ and η using the values of Var[f C ] for K = 40000, 45000, and obtained ζ = 3.112098237555 and η = 897.7504381251. Figure 10 shows for the suggested K. Figure 11 shows a comparison between (51) and the experimental results using Mersenne twister. In the experiment, we used 4 × 10 6 sequences for each trial and set L = 8, Q = 10 × 2 8 and K = 1000 × 2 8 . We can confirm that (51) is consistent with the experimental values and more appropriate than the value used in Ref. [15].

Form of the reference distribution
In this section, we prove that the distribution of the test statistic of Coron's test converges to a normal distribution. More exactly, the test statistic converges to zero as K → ∞ and we prove that the distribution of √ K × f C R (0.5) converges to a normal distribution in some sense. This proof can also be applied for Maurer's test and Yamamoto and Liu's test, with minor changes. In the following, we omit to write argument R (0.5) for every variables. Note that the omitted argument is different from that in the previous section.

Stationary case
First, we consider the case that Q → ∞, i.e., the random variable sequence {A n } n∈N satisfies ergodic stationary condition. We use the following lemmas repeatedly: Lemma 4.1. Let M be a real number satisfying 0 < M < 1. Then, there exist real numbers C > 0 and 0 < δ < 1 such that Proof. Since g(i) = O(log i), the lemma is obvious.
There exist positive real numbers C and δ < 1 such that Proof. In Ref. [13], it is shown that (We can get the same results by substituting p = 0.5 into the results of the former section.) Then, the lemma holds.
Let X n be g(A n ). Proof. Since {X n } n∈N is ergodic stationary, it is obvious that E [X n ] does not depend on n if it exists. The existence is directly proven by lemma 4.1 and (55).
We use the notation µ X as E [X n ].
Lemma 4.4. The variance Var 1 √ K K n=1 (X n − µ X ) exists for all K ∈ N and the limit lim K→∞ Var 1 Proof. By lemmas 4.1 and 4.2, there exist positive real numbers C 1 , C 2 and δ < 1 such that Since {X n } n∈N is ergodic stationary, Var [X n − µ X ] does not depend on n and Cov [X n − µ X , X n+k − µ X ] depends only on k. Then, we obtain We have From the above, lemma 4.4 holds.
We use the notation τ X as lim K→∞ Var 1 n and M J as Lemma 4.6. For arbitrary J ∈ R and K ∈ N, Var 1 Proof. For all J, the support of Y Y is a finite set. Then, for all J and K, the support of is also a finite set and Var 1 exists. By (66) and (69), we have Then, by lemma 4.4, Var 1 exists for all J and K.
because the support of Y is a finite set. Then, by the theorem 2 in Ref. [19], the lemma holds.
Lemma 4.9. The following equation is true: Proof. Since Z (J) n n∈N is ergodic stationary, we have By lemma 4.1, there exist positive real numbers C 1 and δ 1 < 1 such that Then, by lemma 4.7, By (54), we have for j ≤ n − 2. Then, by lemmas 4.1 and 4.2, for n ≥ M J + 1, there exist positive real numbers C 2 and δ 2 < 1 such that and for n ≤ M J , there exist positive real numbers C 3 and δ 3 < 1 such that Then, we obtain From the above, lemma 4.9 is holds.

Nonstationary case
In real situation, we set Q sufficiently large but finite, and we cannot assume that {A n } n∈N is ergodic stationary. However, even in such situation, {A n } n≥T approximately satisfies ergodic stationary condition as T → ∞ and we can show that using a normal distribution is proper. Let X (Q) n be a random variable described as X (Q) n = X n (X n ≤ Q + n − 1) g(Q + n − 1) (otherwise) .
We show that theorem 4.1 holds even if we replace {X n } n∈N with X (Q) n n∈N . Concretely, we prove the following.
Proof. By lemma 4.1 and (55), there exist positive real number C 1 and δ 1 < 1 such that By lemmas 4.1 and 4.2, there exist positive real number C 2 and δ 2 < 1 such that

Proof for Maurer's test and Yamamoto and Liu's test
The discussion in this section holds even if we replace g with log. Using the result of the previous section, we can prove lemmas 4.1 even if we replace R (0.5) with R (p) where p ∈ (0, 1). In addition, for arbitrary n, Pr A n (R (p) ) = i exponentially reduce when i become larger, which corresponds to (55). Then, the discussion in this section also holds for Maurer's test and Yamamoto and Liu's test, and we can confirm the properness of using normal distributions as their reference distributions.

Conclusion
We showed the properness that we use normal distributions as the reference distributions of Maurer's test, Coron's test, and Yamamoto and Liu's test. Additionally, we derived the variance of the reference distribution of Yamamoto and Liu's test, and all of the parameters fixing the normal distributions have been now theoretically derived. These results let us use these tests properly and evaluate randomness more precisely.