| run | pass | fail | error | total | wall seconds | |
|---|---|---|---|---|---|---|
| dev/cruxeval | 725 | 0 | 63 | 788 | 477 | functions |
| dev/humaneval_plus | 160 | 0 | 8 | 168 | 137 | functions |
| dev/thealgorithms | 1857 | 0 | 450 | 2307 | 1865 | functions |
| dev total | 2742 | 0 | 521 | 3263 | 2479 | |
| holdout/mbpp_plus | 166 | 0 | 37 | 203 | 128 | functions |
| holdout total | 166 | 0 | 37 | 203 | 128 |
pass: the source with the decompiled function spliced in passed the checks of the task.
fail: that source did not pass them (a failing check, a check that timed out, or no def of the output to splice in).
error: the decompiler produced no Python (it raised, or its worker was killed or timed out).
dev, of the errors: 0 WorkerKilled, 0 FunctionTimeout.
holdout, of the errors: 0 WorkerKilled, 0 FunctionTimeout.
wall seconds: from the start to the end of the run, with the workers in parallel; a total is the sum of the runs of that corpus.
The passed functions, as the original def and as the decompiled output (ours). The output of a function that did not pass means something else than the original, so it is not averaged. avg: per function.
| run | pass | avg lines (original -> ours) | avg cognitive complexity (original -> ours) |
|---|---|---|---|
| dev/cruxeval | 725 | 5.3 -> 6.2 | 1.8 -> 1.9 |
| dev/humaneval_plus | 160 | 16.9 -> 18.8 | 3.1 -> 3.9 |
| dev/thealgorithms | 1857 | 19.0 -> 20.8 | 3.2 -> 4.2 |
| dev total | 2742 | 15.3 -> 16.8 | 2.8 -> 3.6 |
| holdout/mbpp_plus | 166 | 3.5 -> 4.9 | 1.3 -> 1.5 |
| holdout total | 166 | 3.5 -> 4.9 | 1.3 -> 1.5 |
Every function that can be compared with angr, passed or not. avg: per function.
| run | functions | total | boilerplate share of angr | sum boilerplate (angr -> ours) | sum python ops (angr -> ours) | avg lines (angr -> ours) | avg cognitive complexity (angr -> ours) |
|---|---|---|---|---|---|---|---|
| dev/cruxeval | 787 | 788 | 85.4% | 45705 -> 238 | 7843 -> 63 | 269.4 -> 28.2 | 179.6 -> 3.8 |
| dev/humaneval_plus | 168 | 168 | 83.3% | 10973 -> 5 | 2194 -> 11 | 306.6 -> 32.6 | 235.4 -> 5.3 |
| dev/thealgorithms | 2297 | 2307 | 80.3% | 185680 -> 2196 | 45496 -> 752 | 420.9 -> 45.7 | 447.3 -> 8.7 |
| dev total | 3252 | 3263 | 81.4% | 242358 -> 2439 | 55533 -> 826 | 378.3 -> 40.8 | 371.6 -> 7.3 |
| holdout/mbpp_plus | 203 | 203 | 86.2% | 9289 -> 2 | 1484 -> 32 | 202.2 -> 22.0 | 145.0 -> 2.3 |
| holdout total | 203 | 203 | 86.2% | 9289 -> 2 | 1484 -> 32 | 202.2 -> 22.0 | 145.0 -> 2.3 |
functions: those of the total with a pseudo-C of the last pass and angr values, whatever their status; the rest has none (its worker died, the binary failed to load, or the pseudo-C could not be rendered).
angr: the C of angr's own decompiler, without the passes of this project. ours: the pseudo-C after the last pass.
boilerplate: work CPython does on its own when the recovered source runs (reference counting, frames, exception propagation); every store, and the calls of a fixed list of runtime functions checked against Nuitka's code generation.
python ops: calls that carry a Python operation but no pass lifted to a marker.
boilerplate share of angr: boilerplate / (boilerplate + python ops) in angr's output.
One dot per passed function, larger where several functions share the values; a dot on the line "same" is as large as the original, one above "2x" is more than twice as large. The axes are square-root scaled. A dot opens the function it stands for, or the functions with those values where several share them.
The original Python sources are the work of the authors of each dataset and are subject to the licenses of the dataset.