Parameterized mathematical framework for mutation analysis of machine learning systems

DOI: 10.31673/2412-9070.2026.318108

Abstract

Ensuring the quality and reliability of machine learning models is a pressing research challenge, particularly in the context of their growing adoption in safety-critical domains. Mutation analysis is one of the most established methods for evaluating test suite quality; however, its application to machine learning systems is complicated by the stochastic nature of training and inference, the absence of a deterministic oracle, and the influence of equivalent and trivial mutants on the resulting score. A key issue is the lack of a unified mathematical formalization that would encompass existing approaches as particular cases and enable their correct comparison, since the notion of mutation score is defined differently across works depending on the chosen rule for detecting differences between the original model and mutants and the method of aggregating results.
The paper proposes a formalization of mutation analysis for machine learning systems as a parameterized computational system. The basic objects of the process are defined: a parameterized mapping of the trained model, a set of mutation operators, a set of generated mutants, a test suite, and a generalized quality metric covering common measures for classification and regression tasks. The generalized mutation score is defined as a parameterized functional determined by two components: a mutant killing rule and a result aggregator. The killing rule is considered in two forms – a thre-shold-based variant and a statistically grounded variant that combines a significance criterion with an effect size measure, ensuring reproducibility for stochastic models. It is shown that existing app-roaches are particular realizations of the proposed framework. An optimization problem is formulated for selecting a subset of mutation operators and test inputs that maximizes the mutation score under resource constraints. The proposed framework provides a formal foundation for the correct comparison of mutation testing approaches and for the development of efficient mutation analysis algorithms for machine learning models.

Keywords: mutation testing, mathematical model, machine learning, machine learning models, processing of big data arrays.