Part 2: a blind test on a program I had never seen

The first experiment had an obvious weakness: the programs were mine, they were small, and the assistant had seen their file names. A fair test needs a program nobody on the reverse-engineering side has seen, and tests that side never sees either. So I set one up.

The setup

• I asked another AI assistant, not Claude, for a deterministic Java console program and 30 test inputs. It wrote a store management system: categories, products, customer tiers, coupons and orders, in 13 classes and 1,772 lines.
• I kept the source on my machine and ran a script that compiled it without debug information, renamed every class, field and method with ProGuard (the entry class too), and left only the obfuscated jar where Claude could see it. The source, the original jar and the name map went into a private folder.
• Claude decompiled the jar, rebuilt readable Java from it, and tested the rebuild against the jar with inputs it generated itself.
• Then I ran my 30 hidden tests on both versions. Claude saw only the final count.

Results

• All 30 hidden tests produced byte-identical output from the original and the rebuild.
• So did 200 randomly generated command scripts (39,820 commands) and 150 targeted scenarios.
• From opening the jar to a working rebuild took about 7 minutes of the assistant's time, by the system clock.
• Every one of the 204 text strings in the rebuild matches the original exactly. That was checked by a script this time, because in the first experiment the assistant had silently "fixed" a typo while retyping.

"200 out of 200" almost hid a gap

The random scripts all passed, and a quick count of what they had actually exercised showed that the loyalty bonus had never been triggered once. It needs one customer to complete five orders before a sixth, and random scripts spread orders across too many customers. So the 200 identical runs said nothing about that part of the program. Targeted scenarios then hit the loyalty bonus 35 times, a rejected coupon 80 times and the quantity discount 131 times, all identical. A passing comparison only vouches for the code it reached.

Names: easier than it looks

10 of the 13 classes got exactly their original names. That sounds impressive until you notice that six of those names were sitting in the program's own text: its toString methods print things like Category{id=...}. The other three differ only in wording, such as DiscountEngine versus PricingEngine. Across 181 fields and methods, 34% came back with the exact original name and half once getter prefixes and capitalisation are ignored. Only 12 of those 181 names appear anywhere in the program's strings, so most of the meaning was recovered from what the code does.

Three bugs, and why its tests missed them

• A category can be given an extra bulk discount. The command answers "OK" and the setting shows up in the category list, but it is never applied: a 2,000 TL order still costs 2,000 TL. The class's own documentation promises "an extra 5% on orders of 500+ TL".
• When a single-use coupon has already been used, the error says the order does not meet the minimum amount, even when the order is 300 TL and the minimum is 50.
• Viewing an order that has not been finalized yet shows a total of 0.00, although a comment in the code says that summary only makes sense for finished orders.

The tests that came with the program wrote their expected results as comments, like "expected: 950.00", but never checked them. I checked them: all 18 expected amounts appear in the output. The only test that sets a bulk discount writes no expectation for it. The bug lives exactly in the one place the tests never looked at. An AI wrote the program and an AI wrote its tests, and both looked complete. It took a check from outside, reading the compiled program with no access to its source, to show what neither had covered.

What this doesn't show

• The program describes itself. Its messages and toString formats name most of its concepts, which made recovering meaning much easier. Commercial software often encrypts or strips those strings.
• I tried adding string encryption with Skidfuscator, but on a program whose behaviour I already knew, its output crashed after the first line. Testing the tool first saved the blind test from a broken jar.
• The decompiler wasn't perfect: it printed both enum constructors as possibly corrupted, and they had to be rewritten from the logic.
• It is one program of under 2,000 lines, rebuilt by one assistant.

Check it yourself

The obfuscated jar, the name map, the decompiled code, the original (its comments translated to English, with a check that the compiled classes are byte-identical), the rebuild, the 30 tests and every script are in the blog-experiments repository. The full name-by-name comparison is in names.md.