On Correctness, Precision, and Performance in Quantitative Verification: QComp 2020 Competition Report

Authors

Carlos E. Budde, University of TwenteFollow
Arnd Hartmanns, University of TwenteFollow
Michaela Klauck, Saarland UniversityFollow
Jan Křetínský, Technical University of MunichFollow
David Parker, University of BirminghamFollow
Tim Quatmann, RWTH Aachen UniversityFollow
Andrea Turrini, Chinese Academy of SciencesFollow
Zhen Zhang, Utah State UniversityFollow

Document Type

Conference Paper

Journal/Book Title/Conference

9th International Symposium on Leveraging Applications of Formal Methods, Verification and Validation (ISoLA'20)

Publisher

Springer Verlag

Location

Rhodes, Greece

Publication Date

10-20-2020

Award Number

NSF, Division of Undergraduate Education (DUE) 1856733

Funder

NSF, Division of Undergraduate Education (DUE)

Abstract

Quantitative verification tools compute probabilities, expected rewards, or steady-state values for formal models of stochastic and timed systems. Exact results often cannot be obtained efficiently, so most tools use floating-point arithmetic in iterative algorithms that approximate the quantity of interest. Correctness is thus defined by the desired precision and determines performance. In this paper, we report on the experimental evaluation of these trade-offs performed in QComp 2020: the second friendly competition of tools for the analysis of quantitative formal models. We survey the precision guarantees - ranging from exact rational results to statistical confidence statements - offered by the nine participating tools. They gave rise to a performance evaluation using five tracks with varying correctness criteria, of which we present the results.

Comments

The final authenticated version is still under review. No DOI has been issued (9/21/2020).

Recommended Citation

Budde, C. E., Hartmanns, A., Klauck, M., Křetínský, J., Parker, D., Quatmann, T., Turrini, A., & Zhang, Z. (Accepted/In press). On Correctness, Precision, and Performance in Quantitative Verification: QComp 2020 Competition Report. In Proceedings of the 9th International Symposium on Leveraging Applications of Formal Methods, Verification and Validation (ISoLA'20) (Lecture Notes in Comuter Science). Springer.