Determination of admissible uncertainty levels for quality assessments of software systems

Abstract

In software quality monitoring, errors in measurement, as well as fluctuations in the execution environment, the software architecture, and the users’ behavior, all cause major problems for quantitative assessment of software quality. To address the problems mentioned above, the paper proposes an integrated approach comprising the Bayesian mean correction approach, robust probabilistic models, autoregressive models, and anomaly detection models. The composite approach is adaptive and determines the so-called allowable uncertainty based on historical data and the current system status. The paper presents an example application of the Bayesian mean correction approach and an anomaly-filtering approach, tested on a real data set from an operational enterprise resource planning (ERP) system. By applying approaches for anomaly filtering and Bayesian correction, the relative uncertainty of the considered response-time metric decreased from 84.1% to 4.1% and has remained stable since then, averaging 290–330 ms. In addition, average user satisfaction has increased by 3.8 percentage points, to 87.7% from 83.9%.

Keywords: software systems, quality assessment, uncertainty, monitoring, optimization, modeling, Bayesian correction.