CC BY
19
DOI https://doi.org/10.18551/rjoas.2017-11.13
CRITERIA FOR ASYMMETRIC PRICE TRANSMISSION MODEL SELECTION BASED ON KULLBACK'S SYMMETRIC DIVERGENCE
Acquah De-Graft H., Associate Professor
Department of Agricultural Economics and Extension, University of Cape Coast,
Cape Coast, Ghana E-mail: [email protected]
ABSTRACT
In econometric modelling of asymmetric price transmission, selection of an optimal model among a collection of candidate models is a critical issue. In order to address the problem, criteria that targets Kullback's Direct Divergence has been extensively used in asymmetric price transmission model selection. An alternative criteria recently introduced by Cavanaugh that targets Kullback's Symmetric Divergence (KIC and KICc) remains unexplored in asymmetric price transmission model selection. In this paper, a Monte Carlo study is conducted to evaluate the relative performance of the recently developed selection criteria based on Kullback's Symmetric Divergence (KIC and KICc) against commonly used alternatives based on Kullback's Direct Divergence (AIC and AICc) in terms of their ability to recover the true asymmetric data generating process. Monte Carlo simulation results indicate that the performance of the model selection methods is influenced by the sample size, the level of asymmetry and the amount of noise in the model used in the application. KICc is comparable to KIC and both outperform AIC and AICc in both small and large samples. At lower noise levels, KICc is comparable to KIC and both outperform AIC and AICc. As difference in asymmetric adjustment parameters or speeds increases KICc is comparable to KIC and both outperform AIC and AICc. These results suggest that criteria based on Kullback's Symmetric Divergence (KICc and KIC), is a very reliable and useful criterion in asymmetric price transmission model selection.
KEY WORDS
Price asymmetry, Akaike's Information Criteria, Kullback Information Criteria, model selection.
Researchers in asymmetric price transmission modelling have developed competing models for detecting price asymmetry. Model selection criteria provide useful tools for choosing an appropriate model from a set of competing models. In asymmetric price transmission model selection, criteria that target directed divergence such as Akaike Information Criteria (AIC), and its corrected versions, Corrected Akaike Information Criteria (AICc) have been employed. For example, Acquah (2013) addresses asymmetric price transmission model selection problem using AIC and its corrected version. The directed divergence, also known as the I-divergence, is an asymmetric measure of separation between two statistical models, meaning that an alternate directed divergence may be obtained by reversing the roles of the two models in the definition of the measure. As an alternative to estimating the directed divergence, model selection criteria that targets symmetric divergence between the generating model and a fitted candidate model has been developed. The symmetric divergence, also known as the J-divergence, is the sum of the two directed divergences. Kullback Information Criteria (KIC) and the Corrected Kullback Information Criteria (KICc) were proposed as asymptotically unbiased estimators of the J-divergence (Cavanaugh 1999). Cavanaugh 2004 asserts that the symmetric divergence is arguably more sensitive than the directed divergence towards detecting improperly specified models when used to evaluate fitted candidate models. There is, therefore the need to extend previous literature on comparison of model selection methods in asymmetric price transmission modelling to include new criteria based on symmetric divergence (KICc and
KIC) which has been successfully applied to a number of applications such as linear regression, longitudinal data and overdispersed data.
Previous studies addressing issues of asymmetric price transmission model selection did not consider the use of criteria based on symmetric divergence such as KIC and corrected KIC. In effect, little is understood about the performance of criteria based on symmetric divergence in selecting the correct asymmetric price transmission model. A fundamental question in asymmetric price transmission model selection is, will criterion based on symmetric divergence outperform criterion based on directed divergence in identifying the true asymmetric price transmission model. In effect, will symmetric divergence serve as a more sensitive discrepancy measure than the direct divergence in asymmetric price transmission modelling context?
In order to address these issues, the study compares the relative performance of model selection criteria based on directed divergence ( AIC, AICc) with the criterion based on symmetric divergence ( KIC, KICc) in terms of their ability to recover the true asymmetric data generating process (DGP). In effect, the paper evaluates the ability of AIC, AICc, KIC, and KICc to choose between alternative methods of testing for price asymmetry.
METHODS OF RESEARCH
Model Selection Methods. Model selection criteria are statistical instruments that serve the purpose of choosing a suitable statistical model from a set of candidate models. Researchers studying a price asymmetry often postulates different theories that could explain the phenomenon. The different theories hypothesized usually generate a group of candidate models that could viably characterize the data. Model selection criteria are used to assign scores to each of the fitted candidate models in order to assist the researcher in selecting a good model. Different model selection criteria have been developed. Akaike pioneered the work in this area and developed the Akaike information criterion (AIC) (Akaike, 1973). AIC remains the most widely known and used model selection criterion. Subsequently, extentions of AIC have been developed. AIC and related measures were developed based on the notion of the Kullback Directed Divergence (KDD). Cavanaugh, (1999, 2004) developed KIC and extensions based on the notion of Kullback Symmetric Divergence (KSD) as an alternative measure of the Kullback's discrepancy. Models that minimizes the information criteria are declared as the best model among the set of candidate models under consideration. In this section commonly used information criteria such as Akaike Information Criteria (AIC and AICc) and recently developed Kullback Information Criteria (KIC and KICc) are discussed.
Akaike Information Criteria (AIC). Akaike (1973, 1974), proposed AIC as an estimate of Kullbacks's directed divergence. AIC is an asymptotically unbiased estimate of the Kullback's directed divergence. With this motivation, AIC is defined as:
AIC = - 2 lo g (L ) + 2 k
(1)
Where: (L) denotes the maximum likelihood function of the model with k covariates. The -2 log (L) is the goodness of fit term and 2k is the penalty for model complexity. The difference between the expected value of AIC and the expected Kullback's directed divergence should tend to zero as sample size increases. AIC chooses the best-approximating model to the data generating process and models with minimum AIC values are preferred.
Corrected Akaike Information Criteria (AICc). AIC severely underestimates Kullback's directed divergence for high dimensional fitted models in the competing models. Hurvich and Tsai (1989) proposed another information criteria, corrected AIC as an estimate of the Kullback's directed divergence which corrects the overfitting nature of AIC. AICc can be defined as:
2nk
AICc = -21og(L) + -
(n - k -1)
Where: (L) denotes the maximum likelihood function of the model with k covariates and n is the sample size. The first term represents the log-likelihood of the model and the second term penalizes the model for complexity. Models with minimum AICc values are preferred.
Kullback Information Criteria (KIC). Cavanaugh (1999) proposes KIC as an analogue of AIC that targets the symmetric as opposed to the directed divergence. KIC is asymptotically unbiased estimate of the J-divergence.
KIC is defined as:
KIC = - 2 1og( L) + 3 k
(3)
As sample size increases, the difference between the expected value of KIC and the expected value of Kullback's symmetric divergence should tend to zero. Where (L) denotes the maximum likelihood function of the model with k covariates. The -2 log (L) is the goodness of fit term and 3k is the penalty for complexity. Models with minimum KIC values are preferred.
Corrected Kullback Information Criteria (KICc). Cavanaugh (2004) proposed an analogue of the corrected AIC (AICc) for the normal linear regression model based on estimating Kullback's symmetric divergence. The criteria is defined as:
f n \ n Un - k + 1)(2k +1) - 2)
KICc = -21og(L) + n 1n I--— 1 + -^-----f-
{ n - k -1) (n - k - 1)(n - k -1)
(4)
KICc is exactly unbiased estimator of symmetric divergence in the normal linear regression framework, (Kim and Cavanaugh, 2005). Where (L) denotes the maximum likelihood function of the model with k covariates and n denotes the sample size. The -2 log (L) is the goodness of fit term and the remaining terms denote the penalty for complexity. Models with minimum KICc values are preferred.
Asymmetric Price Transmission Models. A simple Houck (1977) model based on Wolffram (1971) price variable segmentation approach can be employed to model price asymmetry as follows:
Ay, =ffAx++ftAx;+st et □ (5)
x Ax + Ax
Where: the increases and decreases in the price series * are denoted by ' and '
H ■ R + = R -
respectively. Symmetry is tested using an F test of the null hypothesis 0'P1 . The development of the Houck's model does not take into consideration the concept of cointegration between the price series.
Cramon-Taubadel and Loy (1996) and Cramon-Taubadel (1998) established that Houck's model is not an appropriate test for asymmetry if the price series are co-integrated. Granger and Lee (1989) proposed the Error Correction Model as an appropriate test for asymmetry if the price series x and y are co-integrated. The proposed asymmetric error correction model can be represented as follows:
Ay * = R Ax( + RR2+ EC Tt+1 + RECTtZ 1 + et
(6)
The long run equilibrium relationship between the price series x and y is denoted by the Error Correction Term (ECT) which is partitioned into positive and negative component. In
the Error Correction Model, asymmetries specified affects the positive and negative components of the error correction term. Symmetry is tested in eqn (6) using an F test of the
null hypothesis H0: p2 = p2 .
Granger and Lee Model is extended in Cramon-Taubadel and Loy (1996) into a complex asymmetric price transmission model in which asymmetries specified affect the direct impact of price increases and decreases, as well as positive and negative components of the error correction term.
A y ( = A+A + A- A *r + A+ ECTt: 1 + p ECT-1 + ^
(7)
Ax+ Ax" x ECT+ ECT-
Where: ' and ' are the increases and decreases in * and t-1 and t-1 are the
positive and negative components of the error correction term. Symmetry is tested using a joint F test of the null hypothesis H 0 : Pl = P 1 and p2 = p2 .
RESULTS AND DISCUSSION
Model Recovery Rates of the Different Model Selection Criteria. The empirical performance of recently developed model selection criteria based on Kullback Symmetric Divergence (KIC and KICc) and frequently used alternative model selection criteria based on Directed Divergence (AIC and AlCc) in recovering the true asymmetric data generating process (DGP) under conditions of different sample sizes, level of asymmetry and noise levels are evaluated using Monte Carlo simulations. The data generating process is simulated from the standard error correction model in each Monte Carlo experiment as follows:
Ayt = 0.5AXt -0.25(y -x()+*- -0.75(y -x(+£
(8)
In the spirit of Holly et al (2003), the value of P is set to 0.7 and
(A2 jp )G (-025,-0-75) are considered for the coefficients of the true asymmetric error correction model data generating process. In the true model, y and x are generated as I (1)
nonstationary variables that are co-integrated. Ay in eqn (8) is obtained by summing up At and the positive and negative deviations from the equilibrium relationship between y and x.
The behavior of AIC, AICc, KIC and KICc are examined in 1000 Monte Carlo simulation where each criterion is used to select the true model from among a set of competing asymmetric price transmission models. For each Monte Carlo experiment, the candidate models are fitted to the data, and the criteria evaluated in terms of their ability to recover the true asymmetric data generating process. The fitted model favored by each criterion is recorded and defined as the model recovery rates. Model recovery rates were obtained using 1000 Monte Carlo simulations. The number of times each model selection criteria selects the true model provides the basis for comparison. In effect criteria success rate, in choosing the true data generating process forms the basis for comparing the selection methods.
The empirical performance of recently developed model selection criteria based on Kullback Symmetric Divergence (KIC and KICc) and frequently used alternative model selection criteria based on Directed Divergence (AIC and AICc) are compared in terms of their ability to recover the true asymmetric data generating process (DGP) across various sample size conditions, levels of asymmetry and noise levels (i.e. Model Recovery Rates). For the sake of brevity, the standard asymmetric error correction model, the complex asymmetric error correction model and the Houck's model in first differences are denoted by SECM, CECM, and HKD respectively.
The performance of the model selection methods is evaluated in terms of their ability to select the true model among a set of competing models. Table 1 reports the success rates with which each model selection criteria selects the true model. The model selection methods studied do point to the correct model, though their ability to recover the true asymmetric data generating process (DGP) increases with increase in sample size and a decrease in stochastic variance. Similarly, previous studies (Acquah 2013; Markon & Krueger, 2004; Bozdogan, 1987; Atkinson, 1980) also noted that model selection methods empirically do point to the correct model. Generally, as sample size increases, model recovery rates of KIC, KICc, AIC, and AlCc improved. In small samples (upper part of Table 1), the model selection methods recovered at most 82.6 % of the true data generating process.
In moderate samples (middle part of Table 1), the model selection methods recovered at most 92.4 % of the true data generating process. When the sample size was large (lower part of Table 1), the model selection methods recovered at most 92.8 % of the true model. Generally KICc performs similarly to KIC in small to large samples with low noise levels and both outperform AIC and AICc in small to large samples with low noise levels. In effect, at larger sample size, KICc is comparable to KIC and outperforms AIC and AICc.
Table 1 - Relative performance of the model selection methods across sample size Experiment criterion _Model fitted_
Methods_CECM (%)_HKD (%)_SECM (DGP) (%)
AIC 16.2 5.1 78.7
n = 50 o = 1 AICc 11.8 6.7 81.5
KIC 7.9 9.5 82.6
KICc 5.1 12.3 82.6
AIC 15.6 0 84.4
AICc 14.4 0 85.6
n = 150 o = 1 KIC 8.3 0 91.7
KICc 7.6 0 92.4
AIC 15 0 85
n = 500 o = 1 AIC 14.5 0 85.5
KIC 7.6 0 92.4
KICc 7.2 0 92.8
Note: Recovery rates based on 1000 replications.
Success rates of Corrected Kullback Information Criteria (KICc) strongly depended on sample size for the true data generating process (DGP). It increased from 82.6 percent to 92.8 percent when the sample size was increased from 50 to 500. Similarly, recovery rates of Kullback Information Criteria (KIC) also depended on sample size for the true data generating process (DGP). It increased from 82.6 percent to 92.4 percent when the sample size was increased from 50 to 500. This is consistent with the empirical studies which indicate that KICc or KIC is consistent (Cavanaugh, 1999; 2004).
On the other hand, recovery rates of AIC and AICc increased from 78.7 and 81.5 percent to 85.0 and 85.5 percent respectively for the true asymmetric data generating process (DGP) when the sample size was increased from 50 to 500. Though AIC and AICc performed well in the small samples, it is inconsistent and does not improve in performance as sample size increases. Notably, AICc performs better in small and large samples than AIC. Furthermore KICc and KIC outperform AIC and AICc in small and large samples. In summary, KICc performs similarly to KIC and both outperform AICc and AIC in both small and large samples.
Similarly, Cavanaugh (2004) in a traditional linear regression modeling notes that KICc outperforms AIC, AICc and KIC in both small and large sample settings when all possible combinations of covariates are considered for the class of candidate models. These results further document the advantage of using Kullback's symmetric divergence instead of Kullback's directed divergence as an oracle. Seghouane and De Lathauwer (2003) notes in bootstrap simulations that KICc outperforms KIC, AIC and AICc in small and moderate samples. Cavanaugh (1999) and Ye et al (2008) also notes that KIC outperforms AIC and
AlCc. Generally, these results are confirmed in the Monte simulation results presented in Table 1.
Using three error sizes (a) ranging from small to large and corresponding to 1.0, 2.0 and 3.0, the effects of noise level on model selection are studied. The asymmetric price transmission data generating process specified in eqn (8) is simulated via 1000 Monte Carlo experimentation with the different error sizes and a sample size of 150. The competing models are compared to the true model on the basis of their data fitting abilities as the error in the data generating process was increased systematically. Table 2 shows the percentage with which each model selection criteria selects the true asymmetric data generating process (i.e. SECM) among a set of candidate models as the amount of noise in the data generating process was increased.
Table 2 - Relative performance of the selection methods across error size
Experiment criterion Model fitted
Methods CECM (%) HKD (%) SECM (DGP) (%)
AIC 12.1 22.8 65.1
n = 150 j = 3 AICc 10.9 23.7 65.4
KIC 5.4 35 59.6
KICc 4.6 36.4 59
AIC 14.5 4.8 80.7
n = 150 j = 2 AICc KIC 13.3 7.2 5.1 8.5 81.6 84.3
KICc 6.3 9.3 84.4
AIC 15.6 0 84.4
n = 150 j = 1 AICc 14.4 0 85.6
KIC 8.3 0 91.7
KICc 7.6 0 92.4
Note: Recovery rates percentages based on 1000 replications.
Generally, as the amount of noise in the true asymmetric price transmission data generating process increased model selection performance declined. Recovery rates of corrected Kullback Information Criteria (KICc) decreased from 92.4 percent to 59 percent when the error size was increased from 1 to 3. Similarly, recovery rates of KIC decreased from 91.7 percent to 59.6 percent for the true data generating process (DGP) when the error size was increased from 1 to 3. Recovery rates of AIC and AlCc also decreased from 84.4 and 85.6 percent to 65.1and 65.5 percent respectively for the true asymmetric data generating process (DGP) when the error size was increased from 1 to 3. In effect, at higher noise levels, AlCc is comparable to AIC and both outperform KIC and KICc whilst at lower noise levels, KICc is comparable to KIC and both outperform AIC and AICc. Similarly, some previous studies (Acquah, 2013; Myung, 2000; Gheissari and Bab-Hadiashar, 2004; Yang, 2003, Hui et al 2011) found that the recovery rates of the true data generating process decreases with increasing noise levels.
The effects of sample size and stochastic variance on model selection performance was investigated concurrently. The Monte Carlo simulation results indicate that a small error and large sample improves recovery of the true asymmetric data generating process and vice versa as illustrated in Table 3. When the sample size was 50 with an error size of 2.0, the true data generating process was recovered at least 42.3 percent of the time by the model selection criteria as illustrated in the upper part of Table 3. On the other hand, when the sample size was 150 with error size of 0.5, at least 84.4 percent of the correct model was recovered across all the model selection methods as indicated in the lower part of Table 3.
In effect under stable conditions (a relatively large sample size of 150 and a small error size of 0.5), model selection performance or recovery rates improved whilst under unstable conditions ( a small sample size of 50 and large error size of 2 ) model selection performance or recovery rates decreased.
At higher noise levels and small sample size (unstable conditions), AIC is comparable to AICc and outperforms KIC and KICc. At lower noise levels and large sample size (stable conditions), KICc is comparable to KIC and outperforms AIC and AICc. Similarly, Hui et al
(2011) note that the model recovery rate of AIC decreases slowest at high noise level. Generally, these results are consistent with Monte Carlo experimentation results of previous studies (Acquah, 2010) which suggest that the recovery rates of the true data generating process decreased with increasing noise levels in asymmetric price transmission regression models.
Table 3 - Effects of sample size and stochastic variance on model recovery Experiment criterion
a = 2 n = 50 n = 150 a = 0.5
Note: Recovery rates percentages based on 1000 replications.
The effects of the increase in difference of asymmetric adjustment parameters on model selection are investigated by simulating data of sample size 150 and error size 1 from the standard error correction model and considering asymmetry values of
(P2+>p2 ) e ("0 25,-°.5°) or (-025>-075) for the coefficients of the asymmetric error correction terms. The effect of the level of asymmetry on model recovery is then investigated. Increasing the difference in the asymmetric adjustment parameters from 0.25 to 0.5 results in an increase in model recovery of the true asymmetric data generating process as illustrated in Table 4.
Table 4 - Effects of the level of asymmetry on model recovery
Model fitted
Methods CECM (%) HKD (%) SECM (DGP) (%)
AIC 10.8 37.5 51.7
AICc 7 41.7 51.3
KIC 4.7 48.7 46.6
KICc 2.3 55.4 42.3
AIC 15.6 0 84.4
AICc 14.4 0 85.6
KIC 8.3 0 91.7
KICc 7.6 0 92.4
Experiment criterion
Model Fitted
Methods
CECM(%)
HKD (%)
SECM (DGP) (%)
P2+ -P2- = 0.25 AIC AICc 15.6 14.4 0 0 84.4 85.6
KIC 8.2 0.4 91.4
KICc 7.5 0.4 92.1
P2+ -P2- = 0.50 AIC AICc 15.5 14.3 0 0 84.5 85.7
KIC 8.3 0 91.7
KICc 7.6 0 92.4
Note: Recovery rates percentages based on 1000 replications.
Recovery rates of model selection criteria based on Kullback Symmetric Divergence (KIC and KICc) respond more strongly to increases in the difference between the asymmetric adjustments parameters than other criteria based on directed divergence (AIC and AlCc). Without regards to the concept of information criteria, Cook et al. (1999) notes that the increases in the difference in asymmetric adjustments parameters from 0.25 to 0.50 have positive effects on the test for asymmetry. Notably, difference in asymmetric adjustment parameters or speeds is important in the performance of the model selection methods in recovering the true data generating process.
CONCLUSION
This study investigated the ability of the recently developed selection criteria based on Kullback's Symmetric Divergence (KIC and KICc) and commonly used alternatives based on Kullback's Direct Divergence (AIC and AICc) to clearly identify the correct asymmetric price transmission model out of a set of competing models via Monte Carlo experimentation. The
Monte simulations results indicated that the sample sizes, noise levels and the level of asymmetry are essential in the selection of the true asymmetric price transmission model. Large sample sizes or low noise levels improves the ability of the model selection methods to identify the correct asymmetric price data generating process. Generally, at lower noise levels and large sample size (stable conditions), KICc is comparable to KIC and both outperform AIC and AICc. Furthermore, KICc performs similarly to KIC and both outperform AICc and AIC in both small and large samples. As difference in asymmetric adjustment parameters or speeds increases KICc is comparable to KIC and both outperform AIC and AICc. These results suggest that criteria based on Kullback's symmetric divergence (KICc and KIC), is a very reliable and useful criterion in asymmetric price transmission model selection.
Empirically, the comparison provided contributes to knowledge and understanding of the relative performance of recently developed criteria based on Kullback's symmetric divergence (KICc and KIC) against commonly used criteria based on Kullback's directed divergence (AIC and AICc) in an asymmetric price transmission modelling framework. The study also adds to the literature on asymmetric price transmission modelling by drawing the attention and interests of asymmetric price transmission researchers to adopt more recent statistical model selection criteria based on Kullback's symmetric divergence, such as KICc and KIC, in asymmetric price transmission model selection problems.
REFERENCES
1. Acquah, H. D. (2013). On the Comparison of Akaike Information Criterion and Consistent Akaike Information Criterion in Selection of an Asymmetric Price Relationship: Bootstrap Simulation Results. AGRIS on-line Papers in Economics and Informatics, 5(1), 3.
2. Acquah, H. D. (2010). Comparison of Akaike information criterion and Consistent Akaike information criterion for model selection in Asymmetric Price Transmission Studies. Asian-African Journal of Economics and Econometrics. Vol. 9, No. 1, 49-56.
3. Akaike, H. (1973). Information Theory and an Extension of the Maximum Likelihood Principle. In: B.N. Petrov and F. Csaki (eds.) 2nd International Symposium on Information Theory. 3: 267-81 Budapest: Akademiai Kiado.
4. Akaike, H. (1974). A new look at the statistical model identification, IEEE Transactions on Automatic Control AC-19, 716-723.
5. Atkinson, A. C. (1980). A note on the generalized information criterion for choice of a model. Biometrika, 67, 413-418.
6. Bozdogan, H. (1987). Model Selection and Akaike's Information Criterion (AIC): The General Theory and Its Analytical Extensions. Psychometrika. 52(3):345-370.
7. Cavanaugh, J. E. (1999). A large-sample model selection criterion based on Kullback's symmetric divergence, Statistics and Probability Letters 44, 333-344.
8. Cavanaugh, J. E. (2004), Criteria for linear model selection based on Kullback's symmetric divergence, Australian and New Zealand Journal of Statistics 46.
9. Cook, S., Holly, S., & Turner, P. (1999). The Power of tests for nonlinearity: the case of Granger-Lee asymmetry, Economics Letters, 62, pp.155-159.
10. Cramon-Taubadel, V. S., & Loy, J. P. (1996) Price Asymmetry in the International Wheat Market: Comment. Canadian Journal of Agricultural Economics, 44:311-317.
11. Cramon-Taubadel , S. (1998). Estimating asymmetric Price Transmission with the Error Correction Representation: An Application to the German Pork Market", European Review of Agricultural Economics, 25, pp. 1-18.
12. Gheissari, N. and Bab-Hadiasher, A. (2004). Effect of Noise on Model Selection Criteria in Visual applications. Pattern Recognition, 2, Issue 23-26 pp. 229-232.
13. Granger, C. W. J., & Lee, T. H. (1989). Investigation of Production, Sales and Inventory Relationships using Multicointegration and non-symmetric Error Correction Models. Journal of Applied Econometrics, 4:135- 159.
14. Holly, S., Turner P., & Weeks, M. (2003). Asymmetric Adjustment and Bias in Estimation of an Equilibrium Relationship from a Co-integrating Regression. Computational Economics, 21:195-202.
15. Houck, J. P. (1977). An Approach to specifying and estimating nonreversible Functions. American Journal of Agricultural Economics, 59:570-572.
16. Hui M, Li J, Wen X, Yao L, Long Z (2011) An Empirical Comparison of Information-Theoretic Criteria in Estimating the Number of Independent Components of fMRI Data. PLoS ONE 6(12): e29274. doi:10.1371/journal.pone.0029274
17. Hurvich, C. M. and Tsai, C. L. (1989), Regression and time series model selection in small samples, Biometrika 76, 297-307.
18. Kim, H. J. and Cavanaugh, J. E. (2005). Model selection criteria based on Kullback information measures for nonlinear regression, Journal of Statistical Planningand Inference 134, 332-349.
19. Markon, K. E., & Krueger, R. F. (2004). An Empirical Comparison of Information-Theoretic Selection Criteria for Multivariate Behavior Genetic Models. Behavior Genetics, 34, (6), pp.593- 609
20. Myung Jae In (2000). The Importance of Complexity in Model Selection. Journal of Mathematical Psychology, 44, pp. 190-204.
21. Seghouane, A-K, and De Lathauwer L ( 2003): A bootstrap model selection criterion based on Kullback's symmetric Divergence. IEEE workshop on statistical processing. Pages 494-497
22. Wolffram, R. (1971). "Positivistic Measures of Aggregate Supply Elasticities - Some New Approaches -Some Critical Notes," American journal of Agricultural Economics, 53, pp. 356-356.
23. Yang, Y., (2003). Regression with multiple candidate models: Selecting or Mixing? Statistica Sinica, (13), 783-809.
24. Ye, M., P. D. Meyer, and S. P. Neuman (2008), On model selection criteria in multimodel analysis, Water Resour. Res., 44, W03428, doi: 10.1029/2008WR006803.
