![best and fairest software testing best and fairest software testing](https://cdn.hackernoon.com/images/0k3by31rj.jpg)
It was a personal initiative, made possible by the relatively few solvers available (all solvers of the initial satex are available in our tool).ĭuring the first cycle of competitions (numbered 1 in Table 1) , submitters had to compile a static binary of their solver (to prevent library dependencies) via remote access to the same computer.
#Best and fairest software testing code
Some solvers were modified to comply with the input/output of the satex framework (like a normalized exit code value). satex was running SAT solvers on only one personal computer. Open an convenient solution for reproducibility (binaries, source code and receipt to build binaries are archived in a consistent way), thanks to strong connection with tools like Guix and Zenodo.įollowing the ideas of these first competitions organized in the 90’s, and thanks to the development of the web, the satex website published solvers and benchmarks gathered by the website maintainer.
#Best and fairest software testing archive
In summary, the goals of our open-source tool are to:Ĭollect and archive all SAT solvers, binaries and sources,Įasily retrieve a Docker image with the binary of any solver, directly from the Docker Hub, or, when source code is available, by locally building the image from the source code of the solver,Īllow to easily run any SAT solver that have ever been available (typically in the last 30 years), by a one line call (consistent over all solvers), Our tool allows, for instance, to add a DOI (thanks to Zenodo) to the exact version of any solver used in a paper, allowing simple but powerful references to be used. We also expect authors of previous solvers to contribute by adding informations about their solvers or special command lines not especially used during competitive events. The community is invited to contribute by archiving, from now on, all the solvers used in competitions (and papers). We build our tool, called SAT Heritage, on top of other recent tools, typically developed for archiving and distributing source code and applications, like Docker, GitHub, Guix, Zenodo. We propose in this work to structure and bootstrap a collective effort to maintain a comprehensive and user-friendly library of all the solvers that shaped the SAT world.
![best and fairest software testing best and fairest software testing](https://content.altexsoft.com/media/2016/10/testing-stages.png)
New technologies like Docker are now available, changing the way tools are distributed. However, since the first competitions, the landscape of computer science has changed a lot. If source code was not distributed, binaries were often available. SAT competitions, organized since the beginning of the 21st century, have almost always forced the publication of the source code of submitted solvers.
![best and fairest software testing best and fairest software testing](https://www.qadimension.com/wp-content/uploads/2018/12/QA-Testing.jpg)
Fortunately enough, binaries for Intel 386 machines distributed at that time are still executable on recent computers, given the availability of compatible libraries.Ĭollecting and distributing SAT solvers source code is, luckily, not new. Solvers were distributed with source code to be compiled on Intel or SPARC computers.
![best and fairest software testing best and fairest software testing](http://i1.wp.com/www.eteknix.com/wp-content/uploads/2013/11/R6250_GUI03.jpg)
Facebook and Google didn’t exist and Linux machines were running with kernels 1.X. At that time, some code was using EGCS, a fork of GCC 2.8 including more features. As input format for SAT solvers remains the same since more than 25 years , it is always possible to compare the performances of all existing solvers, given a suitable way of compiling and running them. In the domain of SAT solvers, however, collecting and archiving may not be sufficient: we must embed the recipe to build the code and to run it in the most efficient way. The Software Heritage initiative is, for instance, a recent and strong initiative to handle this. This is a recent but important concern in the more broadly field of computer science. However, nothing has yet been done to ensure that source code and recipes to build SAT solvers will be archived in the best possible way. The release of the source code of MiniSat had, for instance, a dramatic impact on the field. Archiving and maintaining this important amount of knowledge may be as important as archiving the scientific papers that made this domain. In this success story of computer science, the availability of SAT solvers source code have been crucial. As Donald Knuth wrote in , “The story of satisfiability is the tale of a triumph of software engineering”.