> Our most important finding is that the reproducibility rate in nixpkgs has increased steadily from 69% in 2017 to about 91% in April 2023. The high reproducibility rate in our most recent revision is quite impressive, given both the size of the package set and the absence of systematic monitoring in nixpkgs. We knew that it was possible to achieve very good reproducibility rate in smaller package sets like Debian, but this shows that achieving very high bitwise reproducibility is possible at scale, something that was believed impossible by practitioners4
I think people in this thread are focusing on the wrong thing. Sure, not all packages are reproducible, but the project is systematically increasing the percentage of projects that are reproducible while ALSO adding new projects and demonstrating conclusively that what was considered infeasible is actually readily achievable.
> The interesting aspect of these causes is that they show that even if nixpkgs already achieves great reproducibility rates, there still exists some low hanging fruits towards improving reproducibility that could be tackled by the Nix community and the whole FOSS ecosystem.
This work is helpful I think for the community to tackle the sources of unreproducible builds to push the percentage up even further. I think it also highlights the need for automation to validate that there aren't systematic regressions or regressions in particularly popular packages (doing individual regressions for all packages is a futile effort unless a lot of people volunteer to be part of a distributed check effort).
The first invocation will substitute binaries, and the second will rebuild those locally and validate the bit for bit reproducibility of the results.
In Debian there is significant ceremony and special tools/wrappers required to set up the reproducible environment, so no one would bother to use it unless they were specifically working on the https://wiki.debian.org/ReproducibleBuilds initiative.
I think people in this thread are focusing on the wrong thing. Sure, not all packages are reproducible, but the project is systematically increasing the percentage of projects that are reproducible while ALSO adding new projects and demonstrating conclusively that what was considered infeasible is actually readily achievable.
> The interesting aspect of these causes is that they show that even if nixpkgs already achieves great reproducibility rates, there still exists some low hanging fruits towards improving reproducibility that could be tackled by the Nix community and the whole FOSS ecosystem.
This work is helpful I think for the community to tackle the sources of unreproducible builds to push the percentage up even further. I think it also highlights the need for automation to validate that there aren't systematic regressions or regressions in particularly popular packages (doing individual regressions for all packages is a futile effort unless a lot of people volunteer to be part of a distributed check effort).