Eliciting Hyperparameters of Prior Distributions for the Parameters of the Paired Comparison Models

In the study of paired comparisons (PC), items may be ranked or issues may be prioritized through subjective assessment of certain judges. PC models are developed and then used to serve the purpose of ranking. The PC models may be studied through classical or Bayesian approach. Bayesian inference is a modern statistical technique used to draw conclusions about the population parameters. Its beauty lies in incorporating prior information about the parameters into the analysis in addition to current information (i.e. data). The prior and current information are formally combined to yield a posterior distribution about the population parameters, which is the work bench of the Bayesian statisticians. However, the problems the Bayesians face correspond to the selection and formal utilization of prior distribution. Once the type of prior distribution is decided to be used, the problem of estimating the parameters of the prior distribution (i.e. elicitation) still persists. Different methods are devised to serve the purpose. In this study an attempt is made to use Minimum Chi-square (hence forth MCS) for the elicitation purpose. Though it is a classical estimation technique, but is used here for the election purpose. The entire elicitation procedure is illustrated through a numerical data set.


Introduction
The method of paired comparisons is a technique for ranking issues on the basis of subjective assessment.It is primarily used for subjective judgments where quantitative measurement is impossible or impracticable.It is also used in many cases where there may be a substantial effect of sampling error on the measurements.Therefore it is frequently used by psychometricians.Other applications include sensory testing; especially taste testing, consumer tests, personal rating and choice behavior.Probably the most cited among the applied uses of the method of paired comparisons is the tournament analysis in which the objects are players or teams competing with each other in pairs.David (1988) provides a detailed review of paired comparison models.
A vast literature exists to accommodate prior information in the analysis of PC models via Bayesian approach.Davidson and Solomon (1973) apply the Bayesian approach to paired comparison experimentation.As the prior distribution the natural conjugate family of priors is used.Davidson and Solomon (1973) also study the Bayesian analysis of paired comparison technique.Chen and Smith (1984) propose a Bayes type estimator for the worth parameter for the treatment effect parameter of the Bradley-Terry model for paired comparisons.The iterative procedure of estimation is avoided due to the closed form of the estimator.Aslam (1995) discusses in detail the confidence interval method of elicitation.Aslam (2002) performs the Bayesian analysis of two paired comparison models which are the Bradley-Terry model and the Rao-Kupper model with tie with the Bayesian approach.Moreover, Aslam (2003) contributes to the Bayesian statistics by describing a method to elicit hyper parameters of prior density for the parameters of PC model and uses prior predictive distribution to serve the purpose.Kim and Kim (2004) propose a Bayesian approach for the multiple ranking of several Products of Poisson Rates.Aslam (2005) presents a Bayesian comparison of the PC models which allow ties.Adams (2005) illustrates Bayesian approaches, based on the method of paired comparisons, for determining ranks and for estimating relationships between dominance ability and other attributes.Kim (2005) proposes a Bayesian method to provide optimal ranking in the parameter scalar function of several populations.Szwed et al. (2006) present a Bayesian paired comparison approach to assess relative accident probabilities and their uncertainty in a risk study of the largest passenger ferry system in the U.S.An important class of elicitation technique consists of psychological scaling models that use the concept of paired comparison and the paired comparison is elicited from multiple experts.Garthwaite et al (2005) discuss the elicitation theory in detail.
In Section 2, the suggested MCS elicitation approach is discussed in detail.Section 3 provides a numerical illustration of the entire elicitation procedure using real data on five top-rank ODI cricket teams, namely, Australia, India, New Zealand, Pakistan and South Africa.Section 4 concludes the entire study.

Elicitation of Hyperparameters Via MCS Approach
The underlying logic of all the elicitation methods is to minimize the difference between the elicited and the fitted probabilities obtained using a PC model.In MCS method, we try to search for those values of the hyperparameters which minimize the associated chisquare values found by using the posterior estimates obtained on behalf of all possible values of hyperparameters.The procedure is a bit lengthy and based on the data but free from the objection of subjectivity.
The entire elicitation technique may be accomplished through the following steps: Choose a PC model which we urge to study for Bayesian analysis.
(ii) Define an appropriate informative or conjugate prior for the parameters of the PC model.
(iii) Take a (real or simulated) data set for a PC experiment which is intended to yield the ranking for treatments or items under consideration.
(iv) Define likelihood function for the data accordingly.We usually use binomial distribution when the data contains no ties and tri-nomial distribution with three mutually exclusive and exhaustive classes when ties are permitted.
(v) Write down the prior distribution suggested for the parameters of the PC model.
(vi) Write down posterior distribution in the form of density kernel by multiplying the prior with the likelihood function.
(vii) Define the range of parameters of the prior i.e. the hyperparameters which are to be elicited.Definitely these hyperparameters will have some limit through which we are to search for their estimates.
(viii) Find the posterior estimates using the prior distribution with all possible values of the hyperparameters in their range.
(ix) For all estimates of the PC model parameters, find value of chi-square statistics using the relation , which has a  2 distribution with ( − 1)( − 2)/2 degrees of freedom where t denotes the number of treatments to be compared.Here denotes the observed-expected frequency pairs.The expected frequencies may be obtained by using the relation  �  =     , where   denotes the total number of comparisons made between the treatment i and j; and   stands for the preference probability provided by the PC model under consideration.

(x)
The value(s) of the hyperparameters for which the value of chi-square statistics defined above is the minimum, is (are) chosen as the desired estimate(s) of the hyperparameter(s).

Numerical Illustration
For the purpose of illustration, we consider the renowned Bradley-Terry model (BTM) for paired comparisons due to Bradley-Terry (1952) and data on five top-ranked one day international (ODI) cricket teams of Australia, India, New Zealand, Pakistan and South Africa given in Abbas and Aslam (2011) which is given in Table 1.Bradley-Terry model implies that the difference between two latent variables T i and T j has a logistic density with mean (lnθ i -lnθ j ).So the p.d.f. of T i −T j is: If   denotes the probability P{( T i >T j ) | θ i , θ j }, that treatment i is preferred to treatment j, (  ≠ ) , then For the estimation of the worth or strength parameters of BTM, we impose the restriction of their sum to unity for the purpose of identification and the parameters have the range from zero and 1.So it will be appropriate to use the dirichlet distribution as a prior for the model parameters i θ for all  = 1,2, … , , which may be written as where   , ∀ = 1, 2, … , , be the vector of unknown hyperparameters to be elicited.Due to complicated nature of the posterior distribution, we may use different methods, like Markov Chain Monte Carlo (MCMC), Gibbs sampling, Quadrature method etc. to estimates of PC model parameters which are then used to find the value of chi-square statistic.But, we use the Quadratures method of numerical integration, which refers to any method for numerically approximating the value of a definite integral ∫ ()   .The procedure is to calculate itat a number of points in the range a to b and find the result as a weighted average as∫ () , where   denotes the increment used to b through a.Here the accuracy of estimation procedure and the size of increment are inversely proportional to each other.The two dimensional case integration may be found by the relation , where the notations are pre-defined.The higher dimensions may similarly be accounted for.
Following the criteria suggested in Section 2, we execute C codes (given in Appendix) and the resulting output is reported in Tables 2 and 3.The estimates of the worth parameters show that the Kangaroos stand first, South Africans the second, Pakistanis being third, Kiwis being the fourth and finally Indians with the lowest rank.The entire estimates yield a small chi-square test value 3.928935 and the associated highly insignificant p-value 0.686487.

Conclusions
An elicitation technique based on the minimum chi-square approach is suggested for the estimation of hyperparameters of the prior distribution of the parameters of the PC models.The entire elicitation procedure is illustrated taking a real dataset on five ODI cricket teams.Frequentists usually object the Bayesians for the reason that they utilize subjective information collected from experts for elicitation and it makes the entire Bayesian procedures subjective.But by using MCS technique, there remains no issue of subjectivity.Having a view of the facts and figures of the analysis given in the form of posterior means, we see that the five ODI cricket teams under study may be ranked as Australia being the number one, South Africa the second one, Pakistan being the third one, New Zealand with the fourth position and finally India being the fifth and last one.It is important to know that the suggested technique can efficiently be used to elicit the hyperparameters of all types of priors.