Confidence calibration

Methodology in the field of uncertainty estimation in machine learning, focusing on confidence scores and their calibration. Application of the method to alleviate the uncertainty of an in-production machine learning model operating in the Basware case. The solution produces confidence estimates to increase the trustworthiness and dependability of the ML-based system, thus, improving quality and decreasing manual work

The confidence estimator used in the case study is a hybrid system using convolutional neural networks and statistical analysis to derive comparative features from a stack of heatmaps, which originate from the Basware distillation pipeline. These features are fed to an XGBoost model, which learns to assign a confidence score indicating whether the ensemble of models within the Basware pipeline made a correct prediction or not. Finally, these confidence scores are calibrated using a Beta calibration mapping to match with empirical probabilities.

The confidence estimator used in the case study is implemented in Python 3 using the TensorFlow, XGBoost, and betacal libraries.

The method presented in the case study can be adjusted to work with most machine learning models. The produced confidence scores can be used in failure prediction, model monitoring, and for other quality assurance needs.

Open access