Skip to main content

Infer.NET user guide : Tutorials and examples

Page 1 | Page 2 | Page 3 | Page 4

Click model prediction

This section shows how to run click model 1 and click model 2 in prediction mode. In fact, we will build a model especially for prediction. This is because the training models we used were structured according to label class, and, for prediction, we typically have no label information. Many of the components of the model are the same as for training, so we will only single out the key differences for discussion.

// The observations will be in the form of an array of distributions  
Variable<int> numberOfObservations = Variable.New<int>().Named("NumObs");  
Range r = new Range(numberOfObservations).Named("N");  
VariableArray<Gaussian> observationDistribs = Variable.Array<Gaussian>(r).Named("Obs");  
// Use the marginals from the trained model  
Variable<double> scoreMean = Variable.Random(marginals.marginalScoreMean);  
Variable<double> scorePrec = Variable.Random(marginals.marginalScorePrec);  
Variable<double> judgePrec = Variable.Random(marginals.marginalJudgePrec);  
Variable<double> clickPrec = Variable.Random(marginals.marginalClickPrec);  
Variable<double>[] thresholds = new Variable<double>[numLabels + 1];  
// Variables for each observation  
VariableArray<double> scores = Variable.Array<double>(r).Named("Scores");  
VariableArray<double> scoresJ = Variable.Array<double>(r).Named("ScoresJ");  
VariableArray<double> scoresC = Variable.Array<double>(r).Named("ScoresC");  
scores[r] = Variable.GaussianFromMeanAndPrecision(scoreMean, scorePrec);  
scoresJ[r] = Variable.GaussianFromMeanAndPrecision(scores[r], judgePrec);  
scoresC[r] = Variable.GaussianFromMeanAndPrecision(scores[r], clickPrec);  
// Constrain to the click observation  
Variable.ConstrainEqualRandom(scoresC[r], observationDistribs[r]);  
// The threshold variables  
thresholds[0] = Variable.GaussianFromMeanAndVariance(  
    Double.NegativeInfinity, 0.0);  
for (int i = 1; i < thresholds.Length - 1; i++)  
    thresholds[i] = Variable.Random(marginals.marginalThresh[i]);  
thresholds[thresholds.Length - 1] = Variable.GaussianFromMeanAndVariance(  
    Double.PositiveInfinity, 0.0);  
// Boolean label variables  
VariableArray<bool>[] testLabels = new VariableArray<bool>[numLabels];  
for (int j = 0; j < numLabels; j++)  
{  
    testLabels[j] = Variable.Array<bool>(r).Named("TestLabels" + j);  
    testLabels[j][r] = Variable.IsBetween(scoresJ[r], thresholds[j], thresholds[j + 1]);  
}

The first thing to notice is that the inferred variables from the trained model are used as priors for the prediction model. The second point is that the data is not partitioned according to label because we have no labels; consequently, there is no loop over labels. Thirdly, the lower and upper bound thresholds are respectively set to negative infinity, and positive infinity rather than 0.0 and 1.0 - this is so that the label probabilities that will be output by the model sum to 1.0 (scoreJ, being Gaussian, will always have some density outside 0.0 and 1.0). Finally, an array of bool variables is set up - the marginals distributions of these (in the form of Bernouilli distributions) will give the probability of each label.

Running the prediction

To run the prediction, we first must provide some click data - arrays of click and examination counts. This click data must then be converted into the Gaussian observations (obs in the code below) in the same way as the training (though not partitioned by label). This distribution array is then set as the value of the observationDistrib parameter, and the marginals are requested from the inference engine.

numberOfObservations.ObservedValue = obs.Length;  
observationDistribs.ObservedValue = obs;  
InferenceEngine engine = new InferenceEngine(new ExpectationPropagation());  
Gaussian[] latentScore = Distribution.ToArray<Gaussian[]>(engine.Infer(scores));  
Bernoulli[][] predictedLabels = new Bernoulli[numLabels][];  
for (int j = 0; j < numLabels; j++)  
  predictedLabels[j] = Distribution.ToArray<Bernoulli[]>(engine.Infer(testLabels[j]));

Interpreting the results

The following table shows some examples run through the model. The Clicks and Exams columns are the input. The score column shows the mean value of the latent score variable, and the label columns show the probabilities of each label. It can be seen that the more examinations there are, the more confident the model is of the labelling. For example, 9 clicks out of 10 is not enough evidence to favour label 2 over label 1, whereas 999 out of 1000 strongly suggests label 2.

Clicks   Exams   Score     Label0    Label1    Label2  
10       20      0.4958    0.1827    0.6436    0.1736  
100      200     0.4964    0.1731    0.6620    0.1649  
1000     2000    0.4965    0.1719    0.6643    0.1638  
9        10      0.7929    0.0344    0.4764    0.4891  
99       100     0.9328    0.0095    0.3278    0.6627  
999      1000    0.9489    0.0082    0.3102    0.6816  
10       100     0.1408    0.5767    0.4059    0.0173  
10       1000    0.0522    0.6838    0.3081    0.0081  
10       10000   0.0432    0.6940    0.2985    0.0075

Model usage

What we have gained from this model is the calibration of human judgement data against click data using query/document pairs for which we have both observations. We can use this either to identify data for which click data and human judgement data are inconsistent and therefore clean up the training data for a ranking model. Or we could use the predicted labels or score to supplement the human judgement training data.


Page 1 | Page 2 | Page 3 | Page 4