Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models

Limberg C, Wersing H, Ritter H (2020)
Machine Learning and Knowledge Extraction 2(3): 327-346.

Zeitschriftenaufsatz | Veröffentlicht | Englisch
 
Download
Es wurden keine Dateien hochgeladen. Nur Publikationsnachweis!
Abstract / Bemerkung
For incremental machine-learning applications it is often important to robustly estimate the system accuracy during training, especially if humans perform the supervised teaching. Cross-validation and interleaved test/train error are here the standard supervised approaches. We propose a novel semi-supervised accuracy estimation approach that clearly outperforms these two methods. We introduce the Configram Estimation (CGEM) approach to predict the accuracy of any classifier that delivers confidences. By calculating classification confidences for unseen samples, it is possible to train an offline regression model, capable of predicting the classifier’s accuracy on novel data in a semi-supervised fashion. We evaluate our method with several diverse classifiers and on analytical and real-world benchmark data sets for both incremental and active learning. The results show that our novel method improves accuracy estimation over standard methods and requires less supervised training data after deployment of the model. We demonstrate the application of our approach to a challenging robot object recognition task, where the human teacher can use our method to judge sufficient training.
Erscheinungsjahr
2020
Zeitschriftentitel
Machine Learning and Knowledge Extraction
Band
2
Ausgabe
3
Seite(n)
327-346
eISSN
2504-4990
Page URI
https://pub.uni-bielefeld.de/record/2950092

Zitieren

Limberg C, Wersing H, Ritter H. Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models. Machine Learning and Knowledge Extraction. 2020;2(3):327-346.
Limberg, C., Wersing, H., & Ritter, H. (2020). Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models. Machine Learning and Knowledge Extraction, 2(3), 327-346. https://doi.org/10.3390/make2030018
Limberg, Christian, Wersing, Heiko, and Ritter, Helge. 2020. “Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models”. Machine Learning and Knowledge Extraction 2 (3): 327-346.
Limberg, C., Wersing, H., and Ritter, H. (2020). Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models. Machine Learning and Knowledge Extraction 2, 327-346.
Limberg, C., Wersing, H., & Ritter, H., 2020. Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models. Machine Learning and Knowledge Extraction, 2(3), p 327-346.
C. Limberg, H. Wersing, and H. Ritter, “Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models”, Machine Learning and Knowledge Extraction, vol. 2, 2020, pp. 327-346.
Limberg, C., Wersing, H., Ritter, H.: Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models. Machine Learning and Knowledge Extraction. 2, 327-346 (2020).
Limberg, Christian, Wersing, Heiko, and Ritter, Helge. “Beyond Cross-Validation—Accuracy Estimation for Incremental and Active Learning Models”. Machine Learning and Knowledge Extraction 2.3 (2020): 327-346.
Export

Markieren/ Markierung löschen
Markierte Publikationen

Open Data PUB

Web of Science

Dieser Datensatz im Web of Science®
Suchen in

Google Scholar