Introduction and Context: When developing new libraries or improving on old implementations, the aim
generally lies on outperforming the solutions realized in the past. This improve-
ment of performance may be captured in multiple ways, be it increasing number
of solved instances, decreasing wall-clock run-time, or similar measures. In this
work we share our improved benchmarking setup to cover as many scenarios as
possible with one combined scripting framework, called the Simsala Script Col-
lection, which could also be described as self-hosted and primarily CLI-focused
StarExec. We use this setup in our distributed Cube-and-Conquer SAT and
QBF solver research project Paracooba and in our upcoming generic solver
interfacing library QuAPI. We hope that the developed methods and tools can
be helpful to other groups as-well, both during development of new solvers and
when organizing competitions. In the workshop, we will interactively go through
some usage scenarios. A public release of our scripts is available at:
http://simsala.pages.sai.jku.at/ (website with tutorials)
https://gitlab.sai.jku.at/simsala/simsala (repository)
Forschungs-