Quantifying uncertainty with conformal prediction

The purpose of this vignette is to show how we can detect which predictions are uncertain, using conformal prediction.

using SpeciesDistributionToolkit

We will work on the demo data:

sdm = SDM(PCATransform, NaiveBayes, SDeMo.__demodata()...)
variables!(sdm, ForwardSelection)

☑️  PCATransform → NaiveBayes → P(x) ≥ 0.537 🗺️

We are now going to set-up a conformal predictor with a risk level of α = 0.05:

cp = Conformal(0.05)

Conformal{Float64}(0.05, 0.0, 0.0, false)

We now train this conformal predictor:

train!(cp, sdm; mondrian=true)

Conformal{Float64}(0.05, 0.765706068203458, 0.9464555237461743, true)

We can now identify which proportion of the model predictions are uncertain:

pred = predict(sdm; threshold=false)
sets = predict(cp, pred)
n_uncertain = count(isequal(Set([true, false])), sets)
n_uncertain / length(pred)

0.14350945857795172

The reason this works is that predicting with a conformal predictor will return a set of values that are supported given the calibration data given to the model. For example, this is the set of possible values for the 10th prediction:

predict(cp, pred[10])

Set{Any} with 1 element:
  true

Note that we can also perform the non-Mondrian version of conformal prediction, with the appropriate keyword:

train!(cp, sdm; mondrian=false)

Conformal{Float64}(0.05, 0.9111659989355393, 0.9111659989355393, true)

Why is Mondrian-CP the default?

Essentially, this is because we do not really trust absence data when they are generated with pseudo-absences. Therefore, it is important to have an estimation of uncertainty that reflects the fact that the two classes are really distinct. This is the same reason why most data transformers are trained on the presence data only.

We can now identify which of the model predictions are uncertain:

sets_2 = predict(cp, pred)
count(isequal(Set([true, false])), sets_2) / length(pred)

0.13763861709067188

Related documentation

SDeMo.Conformal Type

Conformal

This objects wraps values for conformal prediction. It has a risk level α, and a threshold for the positive and negative classes, resp. q₊ and q₋. In the case of regular CP, the two thresholds are the same. For Mondrian CP, they are different.

This type has a trained flag and therefore supports the istrained function. If the prediction is attempted with a model that has not been trained, it should result in an UntrainedModelError.

source