Practical Correlation-Matrix Approaches for Standardized Testing of Wireless Devices in Reverberation Chambers

We extend the autocorrelation-based approaches currently used in standards to full correlation-matrix-based approaches in order to identify correlation between both spatially adjacent and non-adjacent samples in reverberation-chamber measurements. We employ a scalar metric that allows users to identify the number of effectively uncorrelated samples in new types of stirring sequences. To make these approaches practical and enhance their accuracy, we implement a thresholding technique that retains correlation related to important aspects of chamber configuration such as loading conditions. We develop a method to propagate uncertainty in the complex correlation coefficients through to the number of effective samples for a given reverberation-chamber set-up by use of a bootstrap technique that is accurate even for highly skewed distributions of correlation coefficients. We further apply this method in a sensitivity study regarding the choice of threshold value. Agreement with existing approaches in determining the number of effectively uncorrelated samples is presented for a measurement example where spatially adjacent samples are utilized. Examples are then illustrated for non-spatially-adjacent correlated samples at microwave and millimeter-wave frequencies.


I. INTRODUCTION
E VALUATING the spatial correlation between stirringsequence samples in reverberation-chamber measurements has been a subject of study for many years [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14]. In reverberation-chamber measurements, the number of stirring sequence samples N is often determined by the required level of uncertainty for a specific application. In [15], the component of relative uncertainty related to the stirring sequence was computed as 1/ √ N, where it was assumed that all samples were effectively uncorrelated. However, there will often be some spatial correlation between samples in real reverberation-chamber measurements. Thus, identifying significant correlation between samples and relating it to the uncertainty in the final quantity of interest is a key aspect of creating effective stirring sequences and verifying chamber performance. For example, IEC 61000-4-21 [16] requires that the field homogeneity within the characterized test volume of the reverberation chamber exceed a specified threshold, as determined with field measurements performed at the edges of the test volume using only "independent" stirrer samples. In a second example, the CTIA reverberation-chamber test plan [17], [18] requires that the number of effectively uncorrelated samples exceeds 100 ("preferably 200 or 400") in order to meet an overall uncertainty budget for measurements of total radiated power (TRP) or total isotropic sensitivity (TIS). Thus, estimating the number of effectively correlated samples is critical to meet the requirements of various standards, as is the uncertainty in that estimate. Autocorrelation approaches are often used to identify spatial correlation between adjacent or near-adjacent samples in the data record [4], [6], [16], [19], providing N ind , the number of (independent) samples. Autocorrelation methods are adequate for identifying correlation between spatially adjacent or near-adjacent samples (as created with a rotating paddles or translational stages). The work presented here extends the standardized autocorrelation approaches of [16] and [19,Sec. 7.8], eliminating the current prerequisite for sequential-sample stirring approaches. To differentiate between the single-row autocorrelation approach and the full-correlation-matrix based approaches presented here, we use the term N ind to denote the number of effectively uncorrelated stirring-sequence samples for the former and the term N eff for the latter.
Prior work describes an approach that forms a full Pearson correlation matrix from a single measurement of the stirring sequence by circularly shifting the data [4], [7], [8], [9], [10], [13]. Such methods have typically been implemented frequency point by frequency point with a new correlation matrix formed for each frequency. In [8], [9], [10], [13], correlation matrices of stirring-sequence samples were formed from field-strength measurements at the physical locations corresponding to each corner of the working volume of the chamber. A threshold was applied to the terms in each correlation matrix prior to computing the number of effectively uncorrelated samples N eff . The variance of the field values obtained with the stirring sequence that utilized the N eff samples was then used to determine field homogeneity, as required by the IEC standard [16].
We will utilize a similar approach, but extend it to the case where position stirring is an integral part of the stirring sequence, as for over-the-air (OTA) testing of wireless devices. In these applications, field inhomogeneity is a given because RF absorber is often inserted to broaden the chamber's coherence bandwidth (CBW) [20], [21] allowing demodulation of the communication signal. Note that measurement methods for CBW are discussed in [18] and [22,Ch. 5]. We will refer to this approach as a Circular-Shift Correlation Matrix approach. The term "circular-shift" is used because, as pointed out in [4], the matrix is formed from a single, circularly shifted set of data. If correlation exists only between spatially adjacent or near-adjacent stirringsequence samples, a single row may be used to assess spatial correlation in the stirring sequence, as in the current, standardized approaches. The Circular-Shift Correlation Matrix approach provides correlation between non-spatially-adjacent samples such as those obtained by an antenna switch. Note that such circular shift of data for the IEC approach requires that the stirring mechanisms start and end at the same physical location. No such requirement is necessary for the Circular-Shift Correlation Matrix approach.
Unlike the autocorrelation approaches, a full correlation matrix approach requires a number of independent measurements of the stirring sequence (typically called "observations" of a variable). This enables the determination of both the correlation coefficients and their distribution, with the latter allowing determination of uncertainty in the correlation coefficients themselves. Prior work [7], [8], [9], [11], [14], [23] has obtained these observations from measurements conducted over a limited range of frequencies with the assumption of a slowly varying chamber frequency response.
As examples, Pirkl et al. [7] first explored using a full complex correlation matrix based on correlations of scattering parameters and an entropy metric for the number of effective samples without using a threshold. The groups Becker et al. [11] and Verwer et al. [14] have also applied this technique to assess the correlation between samples for various stirring sequences at microwave and millimeter-wave frequencies, respectively. Gradoni et al. [9], [23] utilized a similar approach in simulation studies, exploring the impact of applying a threshold in the latter work. Pfennig and Krauthäuser [8] later applied the entropy approach of [7] to field-probe measurements with a threshold and refined the algorithm to address small measurement sets.
We also utilize the approach of [7], with the application of a threshold in which correlation coefficients below the threshold value are set to zero to correlation matrices formed from complex S-parameter measurements. We denote this method the Full Correlation Matrix approach, and illustrate its utility with several measurement examples. We also provide an analysis of the components of uncertainty related to estimating the number of effectively uncorrelated samples N eff . Knowledge of the uncertainties in N eff , when propagated through to TRP, TIS, and other quantities obtained from reverberation-chamber measurements, provides greater confidence that N eff is large enough to obtain results of a specified accuracy.
The original work in [7] required that the frequencies of observation be spaced farther than the CBW of the chamber set-up in order to obtain independent observations. In studies not reported here, our work does not support this requirement, finding that the representation of the chamber's spatial correlation is improved by use of all available observations within the correlation-matrix calculation bandwidth. Although, as stated in prior work, we do require that a sufficient number of independent observation frequencies be utilized.
In both the Circular-Shift Correlation Matrix and Full Correlation Matrix approaches described here, we compute the complex Pearson's correlation coefficients, r ij , for every pair of samples i, j in the correlation matrix. After thresholding, we then apply the scalar, entropy-based metric developed in [7] to the remaining set of |r ij | 2 values to VOLUME 4, 2023 409 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.
obtain the effective number of uncorrelated samples. Note that we apply the threshold to the magnitude-squared of the complex correlation coefficients because our analysis utilizes complex S-parameter measurements. This is discussed in detail below. Unlike most prior publications, our work is focused on optimizing stirring sequences for OTA measurements of wireless devices, where position stirring is a necessary part of the stirring sequence, as opposed to verifying a specified field uniformity at specific locations in the chamber. We also present a useful visualization tool based on the distribution of correlation coefficients that can identify the effects of chamber loading, which is also a requirement for wireless device measurements. Our methods are based on measurements of complex transmission parameters S 21 , which are commonly utilized for OTA chamber-precharacterization tests [18]. S-parameter measurements are straightforward to acquire and, for the Full Correlation Matrix approach, retain the frequency dependent phase relationships between the correlation coefficients, which can provide additional insight into the spatial correlation between samples as we will show. Finally, we take advantage of the multiple observations provided by the Full Correlation Matrix approach to quantify components of uncertainty related to the estimate of the number of effectively uncorrelated samples for this approach.
Both of the correlation-matrix based approaches discussed here are easily implemented in a few lines of code in commonly available computational packages. They can be readily standardized as they are based on straightforward mathematical functions. In the following sections, we provide additional detail on the approaches, including a discussion of components of the uncertainty in N eff . We then compare them to the IEC/3GPP approach, and illustrate their utility on measured data from reverberation chambers of different sizes and utilizing various stirring sequences at both microwave and millimeter-wave frequencies. We conclude with a discussion of the relative merits of each approach.

A. SPATIAL CORRELATION IN A STIRRING SEQUENCE
To explain the issues with identifying non-adjacent spatial correlation, consider a reverberation chamber with a single stirring mechanism such as a paddle traversing a linear rail or rotating around an axis. Each position of the paddle represents a unique geometrical state of the chamber, forming a continuous sequence of states as the paddle moves through the chamber. The electric-field distribution in the chamber is linked to the geometry of the chamber and changes as the geometry changes. In general, a small movement of the paddle gives a small change in field distribution, while a large movement makes a large change in the field. For reverberation-chamber measurements, it is desirable that the change in field between stirring positions be large enough that samples become (effectively) uncorrelated. For chambers with a single stirrer, the stirring sequence is a one-dimensional geometrical state space and, thus, correlated samples will be adjacent to each other in the sequence if samples are recorded sequentially. If the chamber has two stirrers the state space becomes a two dimensional region of geometrical stirrer states, where each stirrer creates one axis. This makes possible many variants of trajectories of stirring sequence traversing the two dimensional space. Samples that are close in sequence are correlated, but it is also possible for samples that are far apart in the stirring sequence to be correlated if the trajectory crosses or approaches itself in the two dimensional space. This is illustrated in Fig. 1.
The same also applies to chambers with more than two stirrers, creating a corresponding multi-dimensional space of geometrical states. As discussed below, when switching between source antennas is used as a stirring mechanism, this also adds an axis to the multidimensional space. However, such a switch can only take on a discrete number of states. Thus, identifying the number of spatially uncorrelated samples can be complex, and is the motivation of this work.
As mentioned in the introduction, current standardized methods for determining the number of spatially uncorrelated samples, N ind , used in IEC [16] and 3GPP [19] (Section 7.8) standards, are based on the application of a threshold to the computed autocorrelation or autocovariance of N electricfield or power samples acquired using the chamber's stirring sequence. In practice, an autocovariance approach is typically used in which the mean is removed prior to the calculation of the correlation function. Implicitly, these techniques assume that the stirrer-dependent fluctuations of stirring samples may be modelled as a wide-sense stationary process, having a time-independent expected value, with correlation based only on the difference in time between elements (that is, a fixed autocorrelation function) and finite second moments for all observation times.
While samples are often acquired at frequencies covering the measurement bandwidth, the autocorrelation function is computed at a single frequency. N ind is then obtained from the spatial step size k of the stirring mechanism that provides a value of correlation between the measured power-based samples satisfying r(k) ≤ r lim , where r is the computed autocorrelation and r lim is the threshold below which correlation is deemed insignificant.
An example of the application of the current standardized method using this threshold for determining the stirring-sample step size k is given in Fig. 2(a), where the autocorrelation of measured power samples is plotted and interpolated with the thin lines for a stirring sequence consisting of only mechanical mode stirring (no position stirring) at three different frequencies. The x-axis consists of the lags in the autocorrelation function, formed from lag-shifted vectors of all of the stirring-sequence samples. In Fig. 2(a) each lag corresponds to different, consecutive positions of the mechanical mode-stirrers. In contrast to the results presented in later sections, the magnitude of the correlation coefficients is not taken here, so both negative and positive values are present in the plots. A threshold of r lim = 0.3255 is plotted with a horizontal dashed line, below which samples are usually considered to be uncorrelated. The choice of threshold r lim will be discussed further in Section II-B and the uncertainty related to this choice in Section II-D.
For all three frequencies, the spatial correlation becomes successively smaller when comparing samples with increasing separation (that is, for increasing values of offset). For the highest frequency of 2200 MHz, the correlation goes below the threshold for a step size of k = 1. This means that the spatial step used in the stirring sequence is sufficiently large at this frequency so that N ind = N. The autocorrelation from the measurements taken at 1800 MHz and 700 MHz shown in Fig. 2(a) indicate that a step of k = 1 is not always sufficient to reach correlation values below the threshold. In the case of the measurement at 1800 MHz, a value below r lim is reached for k = 2. In the case of the measurement at 700 MHz, a value below r lim is reached for k = 4.
In chambers that employ multiple stirring mechanisms, it may further be the case that correlation is not monotonically decreasing with increasing sample separation k. For instance, the curve corresponding to a frequency of 1800 MHz shown in Fig. 2(a) illustrates a dip, rise, and then decay for the autocorrelation function with increasing offset between shifted samples. This type of non-monotonic decrease is treated in Section A.3 of IEC 61000-4-21 [16]: "· · · when the magnitude of the correlation coefficient for increasing shifts drops below and remains less than the value 1/e ≈ 0.37." For this case, the correlation is still due to the proximity of mechanical stirrers to their prior location.
Non-monotonicity in the autocorrelation function may also occur when non-sequential position stirring is used. For this case, samples that are not adjacent to each other in the data record may be spatially correlated. An example would be the use of "source stirring" as part of the stirring sequence, where physically separated antennas are used to randomize the measurement of the fields in the chamber. An example of the autocorrelation for this type of stirring sequence is shown in Fig. 2(b), where the stirring sequence consists of four spatially independent antennas that each successively acquire samples for a given chamber state by use of a switch. For these types of stirring sequences, where correlation exists between samples in the data record that are quite distant from each other, the current definition of correlation may not VOLUME 4, 2023 411 Authorized licensed use limited to the terms of the applicable license agreement with IEEE. Restrictions apply.
provide an accurate estimate of the number of uncorrelated samples. For such cases, a full correlation matrix approach should be used and the Pearson's correlation coefficients computed for all sample pairs. The need for such an approach is highlighted in Note 3 of Section A.3 of IEC 61000-4-21 [16], where the limitation of the autocorrelation approach to equispaced samples from a single tuner is discussed. After a brief overview of the current IEC/3GPP approach, we describe in more detail two approaches for obtaining such matrices and identifying the number of effectively uncorrelated samples.

B. IEC/3GPP APPROACH AND THRESHOLDING
In the IEC [16] and 3GPP [19] standards, the spatial correlation between stirring-sequence samples is estimated from an autocorrelation coefficient computed from received power measurements obtained from a single observation of the stirring sequence at a single frequency. As discussed in the previous subsection, this method is valid for spatially adjacent or near-adjacent samples.
In these standards, a threshold is specified for the desired maximum value of correlation coefficient below which samples are considered to be uncorrelated as ρ 0 = 1/e ≈ 0.37. The intent is to apply this criterion to the autocorrelation function computed over all shifted samples of the stirring sequence. However, as stated in these standards and discussed in [4], [6], [10], [24], the specified maximum correlation ρ 0 of 0.37 would only be obtained if this value were used to threshold correlation coefficients that were computed from an infinite number of samples. Thus, a reduced threshold r lim is chosen that is only applied to the measured correlation coefficients if the probability is less than 5% that the true value of the correlation coefficient is greater or equal to ρ 0 = 0.37. In the IEC/3GPP approach, this threshold is used to determine the width of the autocorrelation curve, whereas values of correlation coefficients below this threshold are set to zero in the correlation-matrix-based approaches discussed in the next section. Thresholding compensates for the use of a finite number of measured samples by suppressing noise and providing a more conservative (reduced) estimate for N ind . In the IEC and 3GPP standards, the threshold r lim is defined as where N is the number of stirring-sequence samples. Use of this threshold gives 95% confidence that there will be at least N ind independent samples in the chamber measurement. Equation (1) nonlinearly reduces the threshold r lim based on the number of measurement samples acquired. This function is plotted in Fig. 3 for N ranging from 100 to 1000.
As an example, if the stirring sequence uses N = 200 samples, then r lim ≈ 0.28, reducing the computed number of effectively uncorrelated samples. Note that the IEC/3GPP standards specify a minimum value for N of 100 in order to avoid the "correlation noise floor" whereby a small number of samples will have a higher probability of correlation than a larger number of samples. As well, (1) applies only for ρ 0 = 0.37 as it is based on curve-fitting to numerical data from [4].

C. CORRELATION-MATRIX-BASED APPROACHES
For the two correlation-matrix-based approaches presented here, we first compute an N × N matrix of complex correlation coefficients representing the pairwise correlation between all measured samples. With σ the covariance matrix, the Pearson correlation coefficients are calculated as The work of [7] derived an estimate of the true number of effectively uncorrelated stirring-sequence samples from the complex correlation matrix. We will refer to this estimate as N eff, unthresholded . In the present work, we estimate the lower bound on N eff , as is done in the IEC/3GPP approach, such that correlation will not exceed the specified threshold ρ 0 . To do this, each element, r ij , in the N ×N matrix of complex correlation coefficients is thresholded prior to calculating the number of effectively uncorrelated samples as: Note that if all samples are effectively uncorrelated, the elements r ii will still be 1, and N eff is thus limited to be less than or equal to the number of samples, N. We set the values of r ij that fall below the threshold equal to zero for backwards compatibility with the IEC/3GPP approach. A small value of r ij could be chosen as well.
Thresholding eliminates the effects of small correlation coefficients with low significance on our estimate of N eff . Such low correlation can occur, for example, with multiple bounces in the reverberation chamber, creating partial or low-level correlation that does not significantly affect our estimate of N eff or a quantity of interest such as total radiated power. As well, the use of a non-infinite number of samples may introduce low levels of correlation, sometimes referred to as the correlation noise floor. This is because the chances of a limited number of samples having the same or similar value are higher than they would be if an infinite number of samples could be used. The two approaches are explained in more detail in the following two subsections.

1) CIRCULAR-SHIFT CORRELATION MATRIX APPROACH
In this approach, we form the correlation matrix with a circular shift of the measurement samples as in [4], [7], [8], [10], [12], [13]. However, rather than calculating the circular autocorrelation, instead we form a complex correlation matrix. For a measurement sequence x i with i = 1, 2, . . . , N samples, the circular-shifted data are used to create an N ×N matrix at each frequency as: Note that, in this formulation, x n is a complex value consisting of measured transmission parameters S 21 and that this evaluation is performed at each frequency f (there is no averaging over a frequency bandwidth). For each pair of observations, i, j, corresponding to rows in the matrix X| f the covariance is calculated as where <·> indicates the mean over the stirring-sequence samples and · * is the complex conjugate. The Pearson product-moment correlation coefficients are then calculated as in (2), with r circ ij being the pairwise correlation coefficient obtained with this approach. Superscript "circ" denotes the Circular-Shift Correlation Matrix approach. Note that r circ ij can be simplified to the current definition of r(k) in [16], [19] We then apply the threshold r lim of (1) to each |r circ ij | 2 , setting values less than the threshold to 0. This reduces the noise in the result and compensates for the finite number of samples in the measurement. The number of effective samples is then evaluated by applying the scalar metric, here termed N circ eff , from (3). Steps to implement these operations using computational software are provided in the Appendix.

2) FULL CORRELATION-MATRIX APPROACH
The Full Correlation Matrix approach requires the measurement of several sets or "observations" of the stirring sequence. Due to the slowly varying nature of the frequency response of the reverberation chamber, in this approach, our observations are obtained by use of measurements performed over a band of frequencies, here termed the "correlationmatrix calculation bandwidth." See, for example, [7], [8], [11], [12], [14]. For a measurement sequence with rows of observations (f = f 1 , . . . , f F ) of stirring-sequence samples (n = 1, . . . , N), an F × N matrix is formed as: . . .
Note that, in this formulation, x f ,n is again a complex value consisting of measured transmission parameters S 21 . But here, unlike the Circular-Shift Correlation Matrix approach of Section II-C1, the evaluation is performed over a band of frequencies.
Using (2), Pearson's complex correlation coefficients, here termed r corr ij , are calculated pairwise between the F observations of all N samples in the stirring sequence. In this case, the superscript "corr" is used to denote the frequency dependent Full Correlation Matrix approach. N must again exceed 100 as in IEC61000-4-21 [16], [19].
The threshold of (1) is applied, setting r corr ij to zero for |r corr ij | 2 ≤ r lim . The effective number of statistically uncorrelated samples, here termed N corr eff , is then calculated as in (3).
In many computational packages, the set of N×N complex correlation coefficients may be readily obtained using built-in functions. For a matrix of measured complex S 21 parameters with F rows of frequency vectors and N columns of stirring samples, such built-in functions may be used to obtain the N × N matrix of normalized correlation coefficients directly. An example of the implementation steps is given in the Appendix.

3) THRESHOLD FOR CORRELATION-MATRIX-BASED APPROACHES
For both of the correlation-matrix-based approaches described above, the threshold is applied to the magnitudesquared of the correlation coefficients by setting r ij to zero if |r ij | 2 ≤ r lim . The application of the threshold to the magnitude-squared of our complex S-parameter data is analogous to the application of the same threshold value to IEC/3GPP power measurements, as discussed in [7]. Specifically, consider a complex normally distributed random variable X with independent real and imaginary components having a mean of 0 and variance of σ 2 X , and an exponentially distributed random variable P = |X| 2 . For this case, the covariance σ ij,P may be related to the covariance σ ij,X as [see [7, eq. (14)] and [25]] We will illustrate the equivalence of the terms in (5) in Section III-A by comparing the IEC/3GPP power-based VOLUME 4, 2023 413 approach to the two complex S-parameter-based correlationmatrix-based approaches. Note that the requirement for a mean of zero on X(= S 21 ) may not be satisfied for a chamber configuration with a fixed receive antenna and a stirring sequence based solely on the movement of mechanical paddles. However, OTA measurements of wireless devices often require the use of loaded chambers to broaden the coherence bandwidth [18], [20], [21], [26]. In this case, position stirring of the receive or transmit antenna generally must be used, which shifts the mean to approximately zero, as discussed in [22,Ch. 5].
The importance of thresholding for the correlation-matrixbased approaches is illustrated in Fig. 4. In this figure, the unthresholded values of (a) N circ eff and (b) N corr eff have been computed as a function of frequency for a large, unloaded chamber (to be described in more detail in Sections II-D1 and III-A). The value of N eff = 360 corresponds to the maximum number of stirring samples. The rapid change in N eff between 2 GHz and 2.5 GHz illustrates the frequency dependent nature of the chamber, which supports a higher number of modes at higher frequencies.
In Fig. 4(a) and (b), when no threshold is applied, the value of N eff is underestimated due to the large number of low-level, insignificant correlations between samples present in the measured data. Fig. 4(a) illustrates both the need for applying a threshold and for frequency averaging for the Circular-Shift Correlation Matrix approach. Fig. 4 (b) shows results for the Full Correlation Matrix approach for three different correlation-matrix calculation bandwidths. Even when the correlation-matrix calculation bandwidth is 800 MHz (corresponding to 800 observations), the values of N corr eff are significantly underestimated as compared to those computed with the threshold. The effect of insignificant correlation will be discussed further in Section II-D2.

D. UNCERTAINTY
The probability distribution function of N eff provided by the Full Correlation Matrix approach of Section II-C2 allows us to investigate the components of uncertainty related to the estimate of N eff for a given chamber configuration and stirring sequence. One such component includes the uncertainty in N eff arising from uncertainty in our estimates of the correlation coefficients from a limited number of imperfect measurements. A second component arises from the sensitivity of N eff to the choice of threshold, which affects our level of confidence in the estimated value of N eff .

1) UNCERTAINTY IN N EFF
We first evaluate the uncertainty in our estimate of N eff for the Full Correlation Matrix approach due to the limited number of physical, nonideal measurements used to estimate the correlation coefficients. These measurements are nonideal due to instrumentation noise, minor positioner errors, operator errors, and other real-world effects. This analysis extends the work of [7]. Here we seek to propagate the uncertainty in our estimates of the true value of the correlation coefficients through to N eff .
Fisher's seminal work in [27], [28], [29] has contributed greatly to current understanding of the statistics of correlation coefficients [30]. In [28], Fisher writes that the distribution of correlation coefficients around the true value of the correlation coefficient in the neighborhood of ±1 "become extremely skew, even for large samples, and change their form so rapidly that the ordinary statement of the 'probable error' is practically valueless." This can be understood, at least in part, by considering the finite support [−1, +1] over which the correlation coefficients, which cannot have a magnitude greater than 1, are defined. Indeed, we found that an analytic description based on the standard uncertainty of the correlation coefficients and the application of the law of propagation of error significantly underestimated the variance of N eff .
With this in mind, here we apply a non-parametric bootstrap method to evaluate the uncertainty of N eff . The first bootstrap methods were developed by Efron in [31], with more recent work described in [32]. Bootstrapping mimics the sampling process by resampling from the original set of observations. Multiple bootstrap resamples are used to form distributions that may be used to evaluate the statistics of quantities derived from the samples themselves. Bootstrap methods do not require a priori knowledge of the statistics of the sampled measurements or intermediate results, such as the correlation coefficients used here, and are applicable to complex transforms, such as that used to calculate N eff from (3), with the caveat that variances remain finite [33].
The most common approach to bootstrapping uses sampling with replacement [34]. Sampling with replacement is often preferred because it draws samples from the original set of measured samples and, thus, does not impact the expected results by changing the sample size (e.g., drawing samples from a wider frequency bandwidth). This can be especially important for reverberation-chamber measurements due to the change in the chamber's response with frequency.
Here, we use bootstrap resamples of the measured values of the complex S 21 stirring-sequence samples at randomly selected frequencies within the desired correlation-matrix calculation bandwidth to obtain the bootstrap mean, bootstrap distribution, bootstrap variance and 95% confidence intervals of N corr eff . We identified the top and bottom 2.5% of the data to compute the bootstrap 95% confidence interval.
To obtain our bootstrap resamples, we form many (here 10 000) data matrices from the S 21 measurements of the stirring sequence, retaining the order of the stirring-sequence samples but using randomly ordered frequencies from within the correlation-matrix calculation bandwidth (here 400 MHz) extracted from the full set of measured S-parameters. We then compute the thresholded correlation matrix from each resampled data matrix. The bootstrap values of N eff are computed from the 10 000 bootstrap resamples. These bootstrap values of N eff are then used to evaluate the bootstrap statistics of N eff . Fig. 5 shows the bootstrap distributions and statistics of N eff computed from measured data for two loading cases. We plot only the five-and eleven-absorber cases because, for the unloaded chamber at 3.55 GHz, the bootstrap distribution consists of a single value corresponding to the maximum number of 360 acquired samples. While the chamber configuration corresponding to these data will be discussed in detail in the following section, briefly, the measurements consist of complex S 21 -parameters of a 360-sample stirring sequence acquired over a 400 MHz bandwidth centered at 3.55 GHz with a frequency spacing of 1 MHz. A 360 × 360 correlation matrix was formed from the 400 × 360 data matrix. The threshold corresponding to |r ij | 2 ≤ r lim = 0.3082 was applied, and N corr eff was computed using (3). Measurements were performed in an unloaded chamber (CBW ≈ 613 kHz), as well as with significant (CBW ≈ 3.3 MHz) and heavy (CBW ≈ 6.7 MHz) loading with five and eleven absorber blocks, respectively. Fig. 5(a) and (b) illustrate the bootstrap statistics for these latter two cases. The nominal value of N eff , shown by the vertical dashed line in Fig. 5, was computed from the original data.
In Fig. 5, the smooth distribution indicates that a sufficient number of bootstrap resamples have been used. We see that the bootstrap standard deviation σ is less than one sample, distributed symmetrically above and below the bootstrap mean. The 95% confidence interval, which may be asymmetrically distributed, is less than four samples for both five-and eleven-absorber cases. Thus, this component of uncertainty would not significantly affect the estimate of N eff for this case. However, as will be illustrated in Sections III-A and III-C, the bootstrap uncertainty in N eff is a function of the measurement configuration, including the stirringsequence step size, chamber loading, frequency and number of acquired samples. Thus, there could be configurations for which this component of uncertainty could be significant.
We also note that there are small offsets in Fig. 5 between the nominal values of N eff , which are computed from all of the frequencies in the correlation-matrix calculation bandwidth (400 MHz) over all of the stirring-sequence samples, and the mean of N eff calculated from each bootstrap resample formed by randomly selecting the measurement frequencies in the 400 MHz calculation bandwidth. Offsets in the nominal value of N eff and the bootstrap mean are common when investigating the uncertainty of results generated by nonlinear functions of its inputs. In our case, N eff is clearly not a linear function of frequency (see Fig. 9, for example). Thus, we expect to see the offsets shown in Fig. 5 between the nominal solution for N eff based on all of the mode-stirring samples frequencies and mean of N eff calculated from each bootstrap resample with measurement frequencies randomly selected from the complete set of frequencies used to determine the nominal value of N eff . Finally, note that the variance was zero for the unloaded case where the value of N eff was 360 for all calculations at this frequency, as expected from the graphs in Fig. 4.

2) SENSITIVITY TO CHOICE OF THRESHOLD
The choice of ρ 0 = 1/e ≈ 0.37 for the IEC/3GPP approach was discussed above, with a reduced value of threshold calculated according to (1) implemented to account for the finite number of samples in the stirring sequence. The application of a threshold in the correlation-matrix-based approaches eliminates the vast number of low-value correlation coefficients that do not significantly affect the estimate of a quantity of interest such as TRP or TIS. Such samples are considered to be below the "critical value" (see, for example, [4], [10] and the references therein) and their elimination can result in a significant improvement in the estimate of N eff .
The potentially large number of insignificantly correlated samples from a typical reverberation-chamber measurement configuration is illustrated in Fig. 6, where the distribution of |r corr ij | 2 is plotted in histogram form on the left axis, and the value of N corr eff for various threshold values |r ij | 2 is plotted on the right axis. Aside from the insignificant low-level correlation, these plots clearly illustrate the significant levels of spatial correlation that can be introduced by the chamber setup. For example, the only difference between the unloaded case in Fig. 6(a) and Figs. 6(b) and (c) is that the chamber is loaded with RF absorbers. Thus, the humps toward the middle of the plots correspond directly to these loading conditions. The data correspond to the same chamber configuration and S-parameter measurements as above, with a We have capped the smaller histogram counts in Figs. 6(a)-(d) at 100 on the y axis in order to visualize the higher values of correlation that relate to chamber loading. For Figs. 6(a) -(c), the maximum count is approximately 125 000, 88 000, and 60 000, respectively. These large numbers of insignificantly correlated samples is the reason the threshold is effective at improving the estimate of N eff .
The red dashed curve in these plots show N eff computed with various threshold values applied to |r corr ij | 2 . As can be seen quite clearly in Figs. 6(b) and (c) and as we discuss next, the value of N eff can be quite sensitive to the choice of threshold.
Note that this type of histogram representation of the correlation coefficients can be used to optimize the stirring sequence. For example, choosing a spatial step to reduce higher correlation values is illustrated in Fig. 6(d). When the spatial step is doubled for the five-absorber case, the second hump is virtually eliminated. This choice of increased step size aligns with the estimate of N eff = 180 for the five-absorber case, as shown in the next section.
The effect of the choice of threshold value ρ 0 on the estimate of N ind was discussed in [4], [8], [10] and many other papers. The selection of ρ 0 = 1/e was standardized in [16] and appears to be based on [35] (see [24]). IEC 61000-4-21 [16] refers to this value as a "convention." Lundén and Bäckström point out in [24] that, for the test engineer, ρ 0 is "a trade-off between the cost of using more stirrer positions and the merit of the decreased measurement uncertainty in the test." Krautháuser et al. [4] conclude that ρ 0 must be "established by the community" and investigated values of ρ 0 as high as 0.43.
To study the sensitivity of N eff to the choice of threshold, we again used a bootstrapping approach. We note that the CTIA Proximity Effect Test, a reverberationchamber precharacterization test for ensuring adequate spacing between antennas and absorbing objects, sets r lim = 0.30 [18]. Fixing r lim = 0.30 while N varies from 200 to 1500 corresponds to choices of ρ 0 ranging from approximately ρ 0 = 0.34 to ρ 0 = 0.40. Based on this usage, we elected to use in our bootstrap algorithm a range of threshold values with a uniform distribution centered around 0.37, extending to 0.40 on the upper end and 0.34 on the lower end.
To determine the new values of r lim we integrated the probability density function of the correlation coefficients as was done in [4, eq. (3)], using the numerical tool from [36]. We again created 10 000 bootstrap resamples, this time including both randomly selected threshold values along with the random frequency vector selection outlined in the previous subsection. Fig. 7 plots the resulting bootstrap probability distribution, mean, variance and 95% confidence intervals of N eff from our five-and eleven-absorber examples at a center In contrast to Fig. 5, for these bootstrap analyses we see a much wider bootstrap standard deviation, greater asymmetry in the 95% confidence interval, and, most striking, the potential for greatly asymmetric bootstrap distributions depending on the frequency studied. These results indicate that the estimate of N eff can be quite sensitive to the choice of threshold. Therefore, for standardization purposes it is desirable to decide upon a universally adopted threshold value or values. Because the IEC and 3GPP standards have deemed correlation below the target value of ρ 0 = 0.37 [ensured in practice with the use of r lim from (1)] as insignificant for the EMC and cellular OTA test communities, we select this value as well, setting |r ij | 2 ≤ r lim . An uncertainty of five or even 10 -20 effective samples (see Sections III-A and III-C) is often not an issue in estimates of metrics such as TIS or TRP because, in practice, a large number of extra stirring-sequence samples are typically acquired. However, the bootstrap analysis presented here (or similar analyses) may be used if greater knowledge about the uncertainty and sensitivity of a particular reverberation-chamber set-up to the choice of r lim is desired.

III. APPLICATION OF METHODS
We next illustrate the utility of two correlation-matrix-based approaches, first with a validation example where we expect the three approaches (IEC/3GPP, Circular-Shift Correlation Matrix and Full Correlation Matrix) to provide a similar number of effectively uncorrelated samples (N ind , N circ eff and N corr eff , respectively). We illustrate the uncertainty in N corr eff for three loading cases. In Section III-B, we then present examples for which the IEC/3GPP method overestimates the value of N eff because the acquired samples were not spatially adjacent in the acquired data record due to the use of an antenna switch. Finally, in Section III-C, we illustrate results of the three approaches in the 28 GHz -40 GHz frequency band.

A. STIRRING SEQUENCES WITH SPATIALLY ADJACENT CORRELATED SAMPLES
For measurements in which mechanical rotational or translational stirrers are used exclusively, spatial correlation is expected to exist primarily between adjacent and nearadjacent samples. For such cases, the original IEC/3GPP autocorrelation-based methods of [16] and [19] should provide a good estimate of the number of uncorrelated samples in the stirring sequence, especially for unloaded chambers. Thus, we first compare our correlation-matrixbased approaches to the IEC/3GPP approach to confirm that the three methods agree in this case. Such agreement also demonstrates the equivalence between thresholding a power-like correlation coefficient and the thresholding the magnitude squared of a complex S-parameter-based correlation coefficient, as given in (5).
To study agreement between the methods, we used a dataset consisting of VNA measurements of complex S 21 values ranging from 700 MHz to 6 GHz acquired in 1 MHz steps. These data were initially discussed in Section II-D. The measurements were conducted in a large reverberation chamber of dimensions 4.27 m × 3.65 m × 2.90 m utilizing three rotational stirring mechanisms: a vertical paddle, a horizontal paddle and a rotating turntable. The measurements were performed under static conditions in unloaded conditions, as well as moderately loaded (5 RF absorbers, CBW ≈ 3.3 MHz) and heavily loaded (11 RF absorbers, CBW ≈ 6.7 MHz) conditions. The coherence bandwidths were determined with 0.5 threshold per [18]. Note that current 3GPP test procedures for cellular base stations are performed in chambers with no loading other than the base station itself (or an equivalently sized block of absorber), although CTIA procedures do require loading for testing of cellular-enabled IoT user equipment. Each stirring mechanism was stepped For the Full Correlation Matrix approach, we need several observations of the stirring sequence, ideally at least as many as the number of stirring samples. Thus, given the 1 MHz increment for the S-parameter measurements, we formed the 360 × 360 correlation matrix from a 400 MHz correlationmatrix calculation bandwidth, providing 400 observations of the stirring sequence at each selected center frequency. Center frequencies ranged between 950 MHz and 5.75 GHz with a 100 MHz increment. The IEC/3GPP and Circular-Shift Correlation Matrix approaches were computed for each frequency and the results averaged over the same 400 MHz increments. Fig. 8 plots correlation matrices computed over the 400 MHz correlation-matrix calculation bandwidths corresponding to (a) the lowest measured frequencies (center frequency f c = 950 MHz) and (b) highest measured frequencies (f c = 5.75 GHz). The increased width of the diagonal line in Fig. 8(a) shows that there is significant correlation between samples adjacent to the diagonal, that is, spatially adjacent stirring positions, at lower frequencies, while little correlation is seen in Fig. 8(b). Because the correlated samples are spatially adjacent to each other, the correlation is expected to be detected well by all three approaches. Fig. 9 shows that the application of the threshold r lim from (1) to |r circ ij | 2 and |r corr ij | 2 provides results that agree to within a worst-case of 20 samples for all loading conditions for all three methods. All three methods provide N eff = N for the unloaded chamber above the transition region. Fig. 10 plots N corr eff for the three loading cases along with the 95% confidence intervals computed on a 500 MHz grid by use of the bootstrapping approach described above. The confidence intervals are less than 10 samples except for the rapid transition in the value of N eff between 2 GHz and 3 GHz in the unloaded case. The increase in uncertainty in that frequency range is due to the 400 MHz computation bandwidth utilized for the Full Correlation Matrix approach. As the value of N eff changes rapidly as a function of frequency, the uncertainty in the estimate increases. The IEC/3GPP method computes the correlation in one column (or row) of the correlation matrix a single frequency at a time. The discrete number of stirring-sample steps [visible on either side of the diagonal in Fig. 8(a)] results in discrete steps for N ind as a function of frequency. This discrete step can cause the value of N ind to vary significantly in the transition regions as a function of frequency. This is illustrated in Fig. 11, where values of N ind take on the limited set of values of N/3 = 120, N/2 = 180, or N = 360, depending on the frequency. Frequency averaging is clearly necessary for these situations and gives the smoother plot in Fig. 9. Similarly, the necessity of frequency averaging for the Circular-Shift Correlation Matrix approach was illustrated in Fig. 4(a). Whether a finer frequency step can improve the estimate is the subject of future work.

B. STIRRING SEQUENCES WITH NON-ADJACENT CORRELATED SAMPLES
We next consider a case involving data from a chamber that, in addition to mechanical stirring mechanisms, uses a switch to select between multiple source antennas. The stirring sequence is shown in Table 1. This example will highlight the need for the correlation-matrix-based approaches to estimate the number of uncorrelated samples.
Because the switch toggles between measurement antennas for each mechanical stirring sample, spatial correlation is scrambled in the data record. If N/4 = 150 samples are taken  from each antenna, ideally there would be N eff = N = 600 spatially uncorrelated samples (although spatial correlation due to the longer electrical wavelength may reduce this number somewhat at lower frequencies). For this stirring sequence, the mechanical mode stirrers move slowly but continuously. A fast switching speed relative to the mechanical mode-stirrer speed may result in increased correlation between acquired samples because the chamber conditions will not have significantly changed from one acquired sample to the next. Likewise, for the unswitched case, we would expect increased correlation (and a lower value of N eff ) when the sampling occurs so rapidly that the channel has not significantly changed between samples. This effect, unique to continuous-mode stirring, was documented in [37]. We investigate the ability of the correlation-matrix-based approaches to identify these effects, and compare them to the current IEC/3GPP approach.
We studied the switching configurations described in column two of Table 2 for a stirring sequence with N = 600 samples. For the "Not Switched" case, we intentionally disabled the antenna switch and a single antenna recorded the same data four times. We expect these repeat measurements to be highly correlated, with N eff = N/4 = 150. For the other cases, as the switch time increases, we expect to see less spatial correlation between measured samples and correspondingly higher values of N eff , with the highest value for Case 5.
For these measurements, the reverberation chamber was smaller than in the example of Section III-A, having dimensions of 1.945 m × 2.0 m × 1.44 m, specified for use above 650 MHz. The chamber was unloaded, with stirring mechanisms consisting of a rotating turntable, one translational vertical plate, one translational horizontal plate and the 4-element switched antenna. The data were collected with a 1 MHz frequency step between 400 MHz and 1 GHz.
In Fig. 12, we plot one row of the correlation matrix for Case 1, the unswitched case. The four spikes correspond to the four repeated measurements, which were intentionally used to study the analysis approaches. The height of the spikes changes somewhat because the continuously moving stirrers moved slightly during the acquisition time (below 1 ms). Fig. 13 plots the effectively uncorrelated samples (N ind or N eff ) for the three approaches, where each approach is denoted with a different symbol. We averaged the IEC/3GPP and Circular-Shift Correlation Matrix approaches over 100 MHz (yielding nine data points) and used a 400 MHz correlation-matrix calculation bandwidth for the Full Correlation Matrix approach (yielding five data points). The IEC threshold ρ 0 = 0.37 was adjusted according to (1) to provide r lim = 0.3255. This threshold was applied to all three methods. The horizontal dashed line denotes N eff = 150.
Three cases from Table 2 are plotted in Fig. 13 with different colors. We expect to see some correlation between samples in the 400 MHz -600 MHz frequency range because this is below the chamber's rated frequency of operation. All three approaches capture this effect. Case 1, not switched, is plotted in blue. For this case, we expect to see N eff ≈ 150 above 600 MHz, as shown by the Circular-Shift Correlation Matrix (triangles) and Full Correlation Matrix (squares) approaches Note that N eff < 150 at the lower frequencies because the stirrer speed did not sufficiently exceed the sampling speed as noted in Table 2. However, we see that the IEC/3GPP approach (circles) cannot identify the spatial correlation between non-adjacent samples (illustrated in Fig. 12). This method overestimates the maximum value of N eff = 150, with values ranging from 300 to 600. As discussed in Section II-A and highlighted in [16], the IEC/3GPP method was not developed to detect non-adjacent spatial correlation, but an extension to the correlation-matrix-based approaches would provide an easy correction for this.
Case 3 (200 ms switching time) and Case 4 (500 ms switching time) are plotted in red and green, respectively. Here we expect all three approaches to provide the maximum value of N eff = 600 above the 650 MHz rated lower limit of the chamber, and all are very close to this. Below 650 MHz, we see some additional correlation between samples, especially for Case 3, because the antenna switch changes between antenna elements so quickly that the positioners have not adequately randomized the boundary conditions for the lowest frequency points.

C. MILLIMETER-WAVE DATA IN LOADED AND UNLOADED CHAMBERS
Our next example demonstrates the effectiveness of the correlation-matrix-based approaches with data collected in a large reverberation chamber at millimeter-wave frequencies.
For this example, the stirring sequence again has 600 samples with three different loading conditions. While we expect the number of effectively uncorrelated samples to decrease as the loading with RF absorber increases, because of the large size of the chamber with respect to the electrical wavelength, the reduction in N eff with loading is much smaller as compared to the lower-frequency cases. This example illustrates that the Full Correlation Matrix approach may provide additional information on the reverberation-chamber configuration because it retains the phase relationships between measured samples prior to computing |r corr ij | 2 . We used a chamber of size 3.50 m × 2.30 m × 2.69 m for these measurements. The stirring mechanisms consisted of a rotating turntable, two vertical plates and two horizontal plates. All were operated in stepped-mode conditions. During the stepped-mode stirring sequence, the number of samples was evenly divided between three stirrer groups. In this example, with 600 samples, each stirrer group is measured at nine positions ( 3 √ 600 rounded up to the nearest integer). The stirrer groups are (1) the turntable; (2) one horizontal one vertical plate; and (3) the other horizontal and vertical plate. A single, unswitched omnidirectional antenna was used.
We collected data from 28 GHz to 40 GHz in 1 MHz frequency steps. The chamber was unloaded, or loaded with one or two RF absorbers having dimensions 60 cm × 60 cm × 10 cm. The coherence bandwidths for the zero-, one-, and two-absorber cases were approximately 2.3 MHz, 3.3 MHz and 4.0 MHz, respectively. These values were somewhat lower in the lower half of the 28 GHz to 40 GHz band, and somewhat higher in the upper half of the band, on the order of 0.1 MHz. As mentioned earlier, while not currently required for 3GPP base-station test procedures, loading broadens the coherence bandwidth, allowing demodulation of the signal without distortion introduced by the chamber set-up [20], [21], [26].
For the Full Correlation Matrix approach, the analysis was performed over 600 MHz correlation-matrix calculation bandwidth intervals, each separated by 300 MHz. The large analysis bandwidth is intended to provide approximately as many observations as stirring-sequence samples for this approach. From (1), the value of r lim for ρ 0 = 0.37 and N = 600 was 0.3255. Fig. 14 plots the number of effective samples when the chamber was loaded with two RF absorbers. For the other cases, N eff = 600 across the frequency band. For the two-absorber loading case, both the IEC/3GPP and Full Correlation Matrix approach show a small reduction to around N eff = 595. As the right axis on the graph shows, this is within the 95% confidence interval for the Full Correlation Matrix approach. Note that correlation between stirring-sequence samples between adjacent pairs (near to the diagonal of the correlation matrix) is detected by all three approaches, but off-diagonal correlation would not be. Thus, the three approaches do not track identically. Fig. 15(a) provides a zoomed-in plot of the unthresholded correlation matrix of |r corr ij | 2 for the two-absorber case computed over the 600 MHz correlation-matrix calculation bandwidth. This graph shows a structure of correlation that is clearly apparent for any calculation bandwidth. For this dataset, both the Circular-Shift Correlation Matrix approach and the Full Correlation Matrix approach capture changes in the off-axis correlation with frequency. However, the retention of the phase relationships between frequencies during the analysis with the Full Correlation Matrix approach allows us to view spatial correlation patterns related to the stirring sequence.
To visualize this clearly, Fig. 15(b) plots one row of the unthresholded correlation matrix |r circ ij | 2 . Each line is computed at a single frequency, and low-level correlation is visible for various pairs of stirring-sequence samples. Fig. 15(c) plots one row of the unthresholded correlation matrix |r corr ij | 2 . Here, the frequencies refer to a center frequency around which the 400 MHz correlation-matrix calculation bandwidth is centered. The structure in the correlation becomes apparent with this analysis.
These results show that even for large chambers at millimeter-wave frequencies, the effects of spatial correlation can be clearly identified. For the case shown here, the level of correlation is insignificant and is thresholded out. However, such analyses could be used to improve the chamber configuration or stirring sequence, if desired.

IV. DISCUSSION
Results illustrate that, when thresholded, both of the correlation-matrix-based methods described here can be used to compute the correlation between all pairs of samples in a stirring sequence, not just the adjacent samples as is done in the current standards. A brief discussion highlighting the differences between the two approaches follows.
The Circular-Shift Correlation Matrix approach computes correlation at a single frequency (i.e., a single observation of the stirring sequence). As discussed, this method may then require frequency averaging of the N circ eff values to obtain a meaningful result if N eff changes rapidly as a function of frequency, as shown in Fig. 4(a). As well, thresholding must be performed at each frequency, which increases the computational burden. However, this method is well-aligned with current standardized autocorrelation approaches and can capture frequency dependent behavior in the chamber well.
For the Full Correlation-Matrix approach, correlation is computed over a band of frequencies, providing multiple observations of the stirring sequence. For this approach, frequency averaging is effectively carried out during the calculation. However, the use of several frequencies may smooth out frequency dependent behavior of the reverberationchamber configuration that may be caused by antenna nulls or other effects that change rapidly with frequency, as shown in Section III-A, where the uncertainty increased when the chamber characteristics changed rapidly with frequency. On the other hand, this approach allows the user to estimate the variance and perform a rigorous calculation of the uncertainty in the estimated value of N corr eff based on multiple observations of the same stirring sequence. As shown in Section III-C, retention of the phase information also allows the observation of structure in the stirring sequence not observable with the frequency-point-by-frequency-point autocorrelation methods.
Turning to the uncertainty analyses, our study of bootstrapping to find the uncertainty in an estimate of N eff allows us to draw some important conclusions. First, our estimate of the bootstrap statistics was enhanced by use of all of the measured frequencies F within the correlation-matrix calculation bandwidth, not the subset F spaced by the coherence bandwidth, as was proposed in [7]. For example, in the five-absorber case with CBW ≈ 3.3 MHz discussed in Section II-D1, instead of using 400/3.3 ≈ 121 samples to form the data matrix, we used all 400 samples. Using as much information as possible to construct the correlation matrix reduced the uncertainty in our estimate of the variance. However, as pointed out in [7], it is indeed necessary to provide a sufficient number of independent observations F to adequately approximate the variance. We found that a rule of thumb that the number of observations be greater than or equal to the number of stirring samples reduced the bootstrap variance. In summary, a minimum number of independent observations F is necessary, but restricting the covariance matrix to use only uncorrelated frequencies is not a requirement.
Second, we observe that the estimate of N eff can be sensitive to the choice of threshold. This indicates that use of a standardized value of threshold may result in an approximate value for N eff .
Finally, in this paper we do not explicitly evaluate the uncertainty of estimates in [7] of the actual (unthresholded) number of independent samples in a collection of reverberation-chamber measurements, focusing instead on the lower bound corresponding to the IEC/3GPP method. However, the bootstrapping approach we develop here is general and easily adapted to evaluating the uncertainty in N eff, unthresholded from [7] as well, potentially allowing for better evaluation of the actual uncertainty of other quantities derived from reverberation-chamber measurements, including TRP and TIS. Such uncertainties would also be used when one wishes to compare measurement systems to determine which system is more accurate.

V. CONCLUSION AND FUTURE WORK
We presented two approaches for estimating the number of effectively spatially uncorrelated samples in reverberationchamber measurements of wireless devices. The approaches both utilize full correlation matrices to identify pairwise correlation between samples in the stirring sequence collected at non-spatially adjacent locations.
We applied a bootstrap approach to study components of uncertainty related to the use of correlation-matrix-based approaches. We found that uncertainty in the estimate of N eff from the use of imperfect measurements of the correlation coefficients had a limited effect in our microwave-band measurements, but was important for our measurements in the band between 28 GHz and 40 GHz. The sensitivity of N eff to the choice of threshold was shown to be significant for situations in which N eff changes rapidly as a function of frequency. Finally, measured data illustrated the importance of frequency averaging for autocorrelation approaches when the value of N ind or N eff changes rapidly as a function of frequency.
All three of the measurement examples presented in Section III demonstrated that both of the correlation-matrixbased approaches provide estimates of the number of effectively uncorrelated samples that are sufficiently similar to the existing IEC/3GPP approach that they could be considered "backward compatible" with current standards documents. Use of these approaches could then allow standardized test methods involving innovative, efficient stirring sequences and the assessment of new chamber types such as the vibrating intrinsic reverberation chamber (VIRC), for which a predictable, repeatable stirring sequence is not possible. Stirring methods such as antenna switching and diversity could be readily used to increase the number of effective samples, allowing improved efficiency while reducing uncertainty related to the stirring sequence for both loaded and unloaded chambers.

APPENDIX CREATING AND THRESHOLDING THE COMPLEX CORRELATION MATRIX
For the correlation-matrix-based approaches described in Sections II-C1 and II-C2, once the data matrix of complex S 21 parameters has been formed as illustrated in those sections, only a few lines of code are required to calculate the correlation coefficients, form the correlation matrix, apply the threshold and compute the number of effective samples. We present an example implementation, along with the corresponding commands for popular computational packages MATLAB and Python with the NumPy library [38] 1 : 1.a) Circular-Shift Correlation Matrix approach: Form N × N data matrix X circ of measured complex S 21 parameters having N rows and columns of stirring samples.
1. Mention of product names does not imply endorsement by NIST. Other products may work as well or better.