Nuitka decompiler eval

We evaluate once a day how well Python source code can be recovered from programs compiled with Nuitka.

Nuitka: It is a compiler that turns Python into C, which a C compiler (GCC) then compiles into a binary.
Our decompiler: It extends the decompiler angr to recover Python source code from binaries compiled with Nuitka.

We measure whether the recovered source code passes the original's tests, and how its lines and cognitive complexity differ from the original's. We also compare the pseudo-C output of our decompiler with that of angr alone.

Correctness evaluation

We replace the original function in its program with the recovered one, and check that it returns the same results as the original on the dataset's test inputs. The tests of each function must finish within 30 seconds. A dataset's name links to the list of its functions.

dev / holdout: The functions are split into two sets. We develop the decompiler against dev; holdout is kept unseen during development, to show how it does on code not seen in development.
pass: The tests pass.
fail: A test fails or times out, or the output lacks the function.
error: The decompiler failed, timed out, or produced no Python.

dataset	pass	error	total
dev/cruxeval	725	63	788
dev/humaneval_plus	158	8	166
dev/thealgorithms	1697	398	2095
dev total	2580	469	3049
holdout/mbpp_plus	166	37	203
holdout total	166	37	203

passfailerror

Readability evaluation

For lines and cognitive complexity, a smaller value means code that is easier to read. avg: per function. sum: over the functions of the dataset. Cognitive complexity counts branches and loops, weighted by how deeply they nest.

Python

We compare the original source code with the recovered source code, over the passed functions only.

dataset	pass	avg lines (original -> ours)	avg cognitive complexity (original -> ours)
dev/cruxeval	725	5.3 -> 6.2	1.8 -> 1.9
dev/humaneval_plus	158	17.1 -> 18.9	3.1 -> 3.9
dev/thealgorithms	1697	20.1 -> 21.9	3.3 -> 4.4
dev total	2580	15.8 -> 17.3	2.9 -> 3.7
holdout/mbpp_plus	166	3.5 -> 4.9	1.3 -> 1.5
holdout total	166	3.5 -> 4.9	1.3 -> 1.5

Pseudo-C

We generate pseudo-C source code from the intermediate representation our decompiler uses internally, and compare it with angr's output, over every function that has both, passed or not. angr has no handling for Nuitka, so this measures how much boilerplate our additions remove and how many Python operations they turn back from C calls.

boilerplate: Calls for bookkeeping that CPython does and Python source code never shows (reference counting, frames, error propagation), plus memory writes.
python ops: Python operations (such as additions and attribute accesses) still left as calls in C form.
boilerplate share of angr: boilerplate / (boilerplate + python ops) in angr's pseudo-C.

dataset	number of functions	boilerplate share of angr	sum boilerplate (angr -> ours)	sum python ops (angr -> ours)	avg lines (angr -> ours)	avg cognitive complexity (angr -> ours)
dev/cruxeval	787	85.4%	45705 -> 236	7843 -> 63	269.4 -> 28.2	179.6 -> 3.8
dev/humaneval_plus	166	83.4%	10802 -> 5	2157 -> 11	305.3 -> 32.4	232.9 -> 5.3
dev/thealgorithms	2085	80.4%	168099 -> 1395	40856 -> 625	421.4 -> 44.7	446.6 -> 8.4
dev total	3038	81.5%	224606 -> 1636	50856 -> 699	375.7 -> 39.7	365.8 -> 7.0
holdout/mbpp_plus	203	86.2%	9289 -> 2	1484 -> 32	202.2 -> 22.0	145.0 -> 2.3
holdout total	203	86.2%	9289 -> 2	1484 -> 32	202.2 -> 22.0	145.0 -> 2.3

Original and recovered source code compared

Each dot stands for one passed function. The horizontal axis is the value of the original function and the vertical axis that of the recovered one. The ticks are not evenly spaced: a square-root scale makes changes among small values easier to see. For example, a dot on "same" means the recovered value equals the original, and a dot on "2x" means it doubled. Click a dot to see the details of the function.

lines

devlines of the original and recovered functionsmedian 11 -> 13

holdoutlines of the original and recovered functionsmedian 2 -> 3

cognitive complexity

devcognitive complexity of the original and recovered functionsmedian 2 -> 2

holdoutcognitive complexity of the original and recovered functionsmedian 0 -> 0

Limitations

Most functions in the datasets are small; evaluating real-world programs is future work.
The binaries used for evaluation keep their symbol names and are compiled with inlining and similar optimizations turned off.
The dev binaries are consulted during development, so their results may be better than the real performance.

Datasets

The original Python source code belongs to the authors of each dataset and is under its licenses.

dataset	source	licenses
cruxeval	https://github.com/facebookresearch/cruxeval/tree/10d71b4ca99f2b6624bef9533ade5a0594ab7716	MIT
humaneval_plus	https://github.com/evalplus/humanevalplus_release/tree/68cd26d53a0dec69f85eafe1f82a2a74155a2bd6	Apache-2.0, MIT
mbpp_plus	https://github.com/evalplus/mbppplus_release/tree/64fc4195b858a17cdfdb3324f0baf37939144e14	Apache-2.0, CC-BY-4.0
thealgorithms	https://github.com/TheAlgorithms/Python/tree/f8084d92b2df4faecad57e0cd8c5ab2adbbf8b19	MIT

Environment

decompiler commit hash: 443af03fcb1b

dev

started: 2026-10-17T18:06:01Z, finished: 2026-10-17T18:46:22Z
environment: Linux x86_64, Python 3.13.14, angr 9.3.0, Nuitka 4.1.3
corpus: corpus-dev-2026-10-12-5ca4be0093b1
CI job: 39056274864

holdout

started: 2026-10-17T18:46:53Z, finished: 2026-10-17T18:49:17Z
environment: Linux x86_64, Python 3.13.14, angr 9.3.0, Nuitka 4.1.3
corpus: corpus-holdout-2026-10-12-5ca4be0093b1
CI job: 39056274864