CC BY
45
ON THE COMPARISON OF BAYESIAN INFORMATION CRITERION AND DRAPER’S INFORMATION CRITERION IN SELECTION OF AN ASYMMETRIC PRICE RELATIONSHIP: BOOTSTRAP SIMULATION RESULTS
Henry de-Graft Acquah, Senior Lecturer Department of Agricultural Economics and Extension, University of Cape Coast, Ghana E-mail: henrydegraftacquah@yahoo.com
Joseph Acquah, Assistant Lecturer Department of Mathematics, University of Mines and Technology, Ghana
ABSTRACT
Alternative formulations of the Bayesian Information Criteria provide a basis for choosing between competing methods for detecting price asymmetry. However, very little is understood about their performance in the asymmetric price transmission modelling framework. In addressing this issue, this paper introduces and applies parametric bootstrap techniques to evaluate the ability of Bayesian Information Criteria (BIC) and Draper’s Information Criteria (DIC) in discriminating between alternative asymmetric price transmission models under various error and sample size conditions. The results of the bootstrap simulations indicate that model selection performance depends on bootstrap sample size and the amount of noise in the data generating process. The Bayesian criterion clearly identifies the true asymmetric model out of different competing models in the presence of bootstrap samples. Draper’s Information Criteria (DIC; Draper, 1995) outperforms BIC at either larger bootstrap sample size or lower noise level.
KEY WORDS
Bayesian Information Criteria (BIC); Draper’s Information Criteria (DIC); Model Selection; Bootstrap; Asymmetric Price Transmission Model.
Numerous studies have examined the performance of information criteria in choosing the ‘best’ model from a set of competing models or theories of asymmetric price transmission with the aid of relevant empirical data. This is underpinned by the fact that information criteria provide an attractive basis for comparing alternative theories or models.
Traditional information-theoretic criteria such as Bayesian Information Criteria (BIC) (Schwarz, 1978) and lesser-known criteria such as Draper’s Information Criteria (DIC) are used for the purpose of identifying the true asymmetric model. However, very little is known about the relative performance of BIC and its extension in the asymmetric price transmission modelling context. Acquah and Von Cramon-Taubadel (2009) presents empirical evidence on the relative performance of BIC and DIC in a Monte Carlo Experimentation but neglected the use of bootstrap techniques to analyse the relative performance of BIC and DIC. However, little is understood about their relative performance in selecting the true asymmetric model in bootstrap samples.
An essential question which remains unanswered is how well BIC and DIC will perform when bootstrap samples are used in the price transmission analysis. In the presence of bootstrap samples, will BIC and DIC point to the true model as noted in previous Monte Carlo studies? Does the alterations that lead to the development of DIC results to improvement in model selection within the price transmission framework? Deriving new samples from the original data using bootstrap techniques gives an advantage over the previous Monte Carlo model selection studies which makes implicit assumption about the true values of the parameters.
In order to address these issues, this paper evaluates the ability of BIC and its extension, DIC to choose between alternative methods of testing for asymmetry in the presence of bootstrap samples. Primarily, the study is intended to understand the behaviour of the model selection criteria in the presence of bootstrap samples. In effect this study compares the relative performance of the well known Bayesian Information Criteria with a
lesser-known criterion, DIC (Draper, 1995) in terms of their ability to recover the true data generating process (DGP) in the presence of bootstrap samples. The true asymmetric data generating process is known in all experiments and the Bootstrap simulations are necessary in deriving the model recovery rates of the true model.
The rest of the paper is organized as follows. In the following section, an introduction of the model selection criteria is presented. This is followed by an introduction of bootstrap methods and a brief description of asymmetric price transmission models. A practical application in which the performance of the model selection methods in selecting the correct asymmetric model are evaluated using bootstrap samples is presented. Finally, the study ends with conclusions.
BAYESIAN INFORMATION CRITERIA AND DRAPER’S INFORMATION CRITERIA
Model selection is often employed as a means to identify the model that is best supported by the data from among the candidate set. Two fundamental approaches are used to frequently address this in asymmetric price transmission modelling. One commonly used information criteria is the Bayesian Information Criteria (BIC). BIC is usually explained in terms of the Bayesian theory, specifically, as an estimate of the Bayes factor for two competing models (Schwarz, 1978; Kass and Rafftery, 1995). BIC is defined as:
BIC = - 2 lo g (L ) + p log (n ) (1),
where L refers to the likelihood under the fitted model, p is the number of parameters in the
model and n is the sample size. Models that minimize the Bayesian Information Criteria are selected. From a Bayesian viewpoint, BIC is designed to find the true model given the data.
Exploration of Bayes factor estimation has resulted in various modifications of the BIC being suggested. Generally, many of these modifications are typical of the modifications
incorporated into Drapers’s (1995) information criterion which is given by the following
equation:
DIC = - 2 lo g ( L ) + p l o g (n / 2 n ) (2).
In DIC, the sample size n is replaced by the term n/2n . Draper (1995) asserts that the 2n term typically appears in approximations to the Bayes factor but is often omitted because it is asymptotically negligible. Draper (1995) notes that it should nonetheless be incorporated in Bayesian Information Criteria because of improvements in finite samples. Pauler, 1998 suggested a similar criterion that incorporates the same term. In order to justify that this alteration is preferable, we evaluate the performance of both formulations in identifying the true asymmetric model to ascertain whether the alteration leads to improvement in model selection within the price transmission framework. In general, models that minimize the Draper’s Information Criteria are selected.
The bootstrap. The basic concept of bootstrapping is that inference about a population from sample data can be modelled by resampling the sample data and performing inference on resample sample or bootstrap sample. In effect, bootstrap assumes that the sample is a good representation of the underlying population distribution. In bootstrap resamples, the 'population' is infact the sample and this is known; hence the quality of inference from resample data sample can be measured. A detailed discussion on bootstrap methods is provided in Efron and Tibshirani (1993).
The bootstrap method involves taking the original data set, and using a computer to sample from it to form a new sample (called a 'resample' or bootstrap sample) that is also of the same size as the original data. The bootstrap sample is taken from the original sample by sampling with replacement so it is not identical with the original "real" sample. This process is repeated a large number of times and for each of these bootstrap samples, we calculate the estimates of interest.
Parametric Bootstrap. This involves fitting a parametric model to data using maximum likelihood and resampling from the model. The resample data are used to estimate statistics of interest. In this context the method proceeds as follows:
1. Fit the model, retain the residuals and resample from the retained residuals. In other words , e* is generated by sampling with replacement from e1en
2. The resampled residuals are then added to the x values in the original regression equation to generate new bootstrap values for the outcome variable as y* = x/ + e*
3. Ordinary least squares are then used to estimate the new bootstrap regression coefficients, for this bootstrap sample as X/* from (X,y* )
4. Resampling of the residuals, adding them to the fitted values and estimating the regression coefficients as outlined in steps 1 to 3 is repeated lots of times to estimate parameters of interest with the bootstrap samples. This approach to resampling is referred to as “parametric bootstrap" where residuals from a parametric model are bootstrapped to give estimates of interest.
Measuring Asymmetric Price Transmission. Granger and Lee (1989) Error Correction Model data generating process can be specified as follows:
Ayt = /1Axt + p2(y- x)t_1 + e2 t E2 t~N(o,s2) (3)
Using different sample sizes, y and x are generated as integrated of the order one processes, I( 1) that are cointegrated with an equilibrium relationship between y and x which is defined by an error correction term. The long run dynamics captured by the error correction term are implicitly symmetric. In order to allow for asymmetric adjustments, the error correction term can be segmented as follows:
(;y - =
(y - *)i =
(y - x)t, if (y - x)t > 0 (4)
zero otherwise
(y - *)t, if (y -x)t<0
zero otherwise
The resulting Granger and Lee asymmetric model is defined as:
Ayt = PiAxt + p2 (y - +Pi(y- x)t-i + e3,t e3it~N(0, S2) (6)
This specification is referred to as the Granger and Lee asymmetric model. Asymmetry is integrated by allowing the speed of adjustment to differ for the positive and negative components of the Error Correction Term (ECT) since the long run relationship captured by the ECT was implicitly symmetric. Symmetric adjustments in equation (6) is tested by determining whether the coefficients (P2+ andP2-) are identical (that isH0: P2+ = P2-).
A more complex method to detect price asymmetry is suggested and implemented by Von Cramon-Taubadel and Loy (1996). In this methodology, asymmetries specified affects the direct impact of price increases and decreases as well as adjustments to the equilibrium level. Where Ax? and Ax- are the positive and negative changes in xt and the remaining variables are defined as in equation (6).
Ayt = /IAxt + Ax- + /I(y - x)t-i + (y - x)--i + e4it e4it~N(0, S2) (7)
A formal test of the asymmetry hypothesis using the above equation is: H 0 : p/ = p j- andP2+=P2-. In this case, a joint F-test can be used to determine symmetry or asymmetry of the price transmission process.
In a depature from Von Cramon-Taubadel and Loy (1996) model specification, Houck (1977), applies a model in which asymmetries specified affects the direct impact of price increases and decreases and does not take into account adjustments to the equilibrium level. The Houck method can be written as follows:
Ayt = fcAxt + Axï eSit~N(0,52) (8)
The variables in the model are defined as in equation (7). Symmetry is tested by determining whether the coefficients (P+ andp- ) are identical (that is H° : p/ = Pj- ).
RESULTS AND DISCUSSION
The performance of BIC and DIC in recovering the true data generating process (DGP) is investigated by simulating the effect of sample size and noise levels on model selection. In agreement with the experimental designs of Holly et al. (2003), the value of Pj is set to 0.5
and (P2+,P2-) e (-0.25,-0.75) are considered for the coefficients of the asymmetric error
correction terms in the true model. The competing models are fitted to the bootstrap data and their ability to recover the true model was measured. The recovery rates were derived using 1000 bootstrap simulations. The data generation process is defined in equation (6) and the data is simulated from the standard error correction model as follows:
Ayt = °.5x - °.25(yt - x )+t-! - °.75(yt - x )-t-! +e (9)
The variables y and x are generated as I (1) non stationary variables that are cointegrated. The error correction terms denotes the positive and negative deviations from the equilibrium relationship between y and x. However, we attempt to evaluate the abilities of BIC and DIC to select the appropriate asymmetric model from competing alternatives.
The relative performance of the two model selection methods are compared in terms of their success rates or ability to recover the true data generating process (DGP) across various bootstrap sample size conditions (i.e. Model Recovery Rates) as detailed in Table 1.
For the purpose of brevity, the standard asymmetric error correction model, the complex asymmetric error correction model and the Houck’s model in first differences are denoted by SECM, CECM and HKD respectively.
For each model selection method, the model recovery or success rate defines the percentages of bootstrap samples in which each competing model provides a better model fit than the other competing models. The model selection methods performed reasonably well in identifying the true model, though their ability to recover the true asymmetric data generating process (DGP) increases with increase in bootstrap sample size as illustrated in Table 1.
Table 1. Bootstrapping the Performance of the Selection Methods across Sample Size
Experiment Criterion Methods CECM HKD SECM (DGP)
n = 50 BIC 6.9% 13.2% 79.9%
<7= 1 DIC 5.9% 15.1% 79%
n = 150 BIC 4.00% 0.10% 95.90%
<7= 1 DIC 3.00% 0.10% 96.9%
n = 500 BIC 2.40% 0% 97.60%
<7= 1 DIC 1.70% 0% 98.30%
Note: Recovery Rates Based on 1000 Bootstrap Replications.
In small samples (upper part of Table 1), the model selection methods recovered at most 79.9% of the true model. When the bootstrap sample size is large (lower part of Table 1), the model selection method recover at most 98.3% of the data generating process.
Though BIC and DIC are consistent and improves in performance in large bootstrap samples, at a larger bootstrap sample size, DIC performs better than BIC.
From the ongoing discussion, the model recovery rates of BIC and DIC strongly depends on the bootstrap sample size. This is consistent with the Monte Carlo experimentation of Acquah and Von Cramon-Taubadel (2009). They noted that larger samples improve the ability of BIC and DIC to recover the true model. Numerous other studies, generally demonstrated that BIC is consistent, that is, it tends to choose the true model with probability equal to one in large samples but performs poorly in small samples (Hurvich and Tsai, 1990; Bickel and Zhang, 1992).
In order to simulate the effects of error size on model selection, this study considers three standard deviations ranging relatively from small to large and corresponding to 1.0, 2.0 and 3.0. Using 1000 bootstrap simulations, data is generated from equation (9) with the different error sizes (a) and a sample size of 150.
The fitting abilities of competing models are compared in relation to the true model as the error in the data generating process was increased progressively. Table 2 shows the percentage of simulated data in which the correct model (i.e. SECM) was recovered among competing models by the model selection criteria as the amount of noise in the data generating process was decreased.
Table 2. Bootstrapping the Performance of the Selection Methods across Error Size
Experiment Criterion Methods CECM HKD SECM (DGP)
n = 150 BIC 1.60% 50.20% 48.20%
(7=3 DIC 1.00% 54.00% 45.00%
n = 150 BIC 3.10% 18.70% 78.20%
(7=2 DIC 2.10% 21.10% 76.80%
n = 150 BIC 4.00% 0.10% 95.9%
(7 = 1 DIC 3.00% 0.10% 96.90%
Note: Recovery Rates Percentages Based on 1000 Bootstrap Replications.
Model selection performance deteriorated with increasing amount of noise in the true asymmetric price transmission data generating process (i.e. SECM) as indicated in Table 2. DIC slightly outperforms BIC with recovery rate 96.90% at lower noise levels. Similarly, Acquah and Von Cramon-Taubadel (2009) notes in a Monte Carlo experimentation that BIC and DIC performance declined with increases in noise level. Yang (2003) also finds that the recovery rates of the true data generating process decreases with increasing noise levels in linear regression models. The study further examined the extent to which sample size and stochastic variance concurrently affects model selection performance. Bootstrap simulation results suggest that a small error and large bootstrap sample improves recovery of the true asymmetric data generating process and vice versa. With a small sample of 50 and an error size of 2.0, the true data generating process was recovered at least 38.10 percent of the time by the model selection criteria as noted in Table 3.
Table 3. Effects of Sample Size and Stochastic Variance on Model Recovery
Experiment Criterion Methods CECM HKD SECM (DGP)
n = 50 BIC 2.70% 56.20% 41.10%
(7=2 DIC 2.30% 59.60% 38.10%
n = 150 BIC 4.00% 0.00% 96.00%
(7= 0.5 DIC 3.00% 0.00% 97.00%
Note: Recovery Rates Based on 1000 Bootstrap Replications.
However, with a relatively large sample of 150 and error size of 0.5, at least 96 percent of the true data generating process was recovered across all the model selection methods as indicated in the Table 3. The model recovery rates of the model selection methods are derived under combined conditions of a small sample size of 50 and large error size of 2 (i.e. unstable conditions), and a relatively large sample size of 150 and a small error size of 0.5 (i.e. stable conditions).
CONCLUSIONS
This study examined the ability of BIC and its extension DIC to clearly identify the true asymmetric model out of different competing models in the presence of bootstrap samples. Generally, both BIC and DIC clearly identify the true model in bootstrap samples. The results of bootstrap simulations indicated that the bootstrap sample sizes and noise levels are important in the selection of the true asymmetric model. With larger bootstrap sample sizes or lower noise levels, the ability of the model selection methods to identify the true asymmetric price data generating process was enhanced. Noticeably, under stable conditions such as large bootstrap sample and low noise level DIC slightly outperforms BIC in recovering the true model. These results suggest that the alteration in Draper, 1995 leads to a slight improvement in recovery of the true asymmetric model in large bootstrap samples. The bootstrap comparison provided, sheds light on the empirical performance of the Bayesian Information Criteria and Draper’s Information Criteria in choosing an asymmetric price transmission model in the presence of bootstrap samples. Bootstrap simulation results further illustrate the usefulness of combining parametric bootstrap techniques with model selection methods to identify the true asymmetric price transmission model. Future research will investigate the performance of other extensions or formulations of the Bayesian Information Criteria using non-parametric bootstrap techniques.
REFERENCES
[1] Acquah, H.D. and Von Cramon-Taubadel, S. (2009): Alternative Formulations of the Bayesian Information Criterion for Comparing Asymmetric Price Transmission Models in Journal of Asian Business Management Vol. 1 No. 1: 459-465.
[2] Bickel, P. and Zhang P. (1992). Variable Selection in Nonparametric Regression with Categorical Covariates. J. Am. Stat. Assoc, 87, 90-97.
[3] Draper, D. (1995). Assessment and Propagation of Model Uncertainty. Journal of Royal Statistical Society, Series B (Methodological) 57, 45-97.
[4] Efron, B. and Tibshirani, R. (1993). An Introduction to the Bootstrap, London: Chapman and Hall.
[5] Holly, S., Turner P. and Weeks M. (2003). Asymmetric Adjustment and Bias in Estimation of an Equilibrium Relationship from a Co-integrating Regression. Computational Economics, 21, 195-202.
[6] Houck, J.P. (1977). An Approach to Specifying and Estimating Nonreversible Functions. American Journal of Agricultural Economics, 59, 570-572.
[7] Hurvich, C. M. and Tsai C-L. (1990). The Impact of Model Selection on Inference in Linear Regression. Am. Stat., 44, 214-217.
[8] Granger, C.W. J. and Lee T.H. (1989). Investigation of Production, Sales and Inventory Relationships using Multicointegration and Non-symmetric Error Correction Models. Journal of Applied Econometrics, 4, 135- 159.
[9] Kass, R.E. and Raftery A. (1995). Bayes factors. Journal of the American Statistical Association , 90, 773-795.
[10] Pauler, D. K. (1998). Schwarz Criterion and Related Methods for Normal Linear Models. Biometrika, 85, 13-27.
[11] Schwarz, G. (1978). Estimating the Dimension of a Model. Annals of Statistics 6, 461-464.
[12] Von Cramon-Taubadel, S. and Loy J.P. (1996). Price Asymmetry in the International Wheat Market: Comment. Canadian Journal of Agricultural Economics, 44, 311-317.
[13] Yang, Y. (2003). Regression with Multiple Candidate Models: Selecting or mixing? Statistica Sinica, 13, 783-809.
