Will Cook

FaithformBench Reader

Their work
Rob Cornish, Iacopo Ghinassi and coauthors changed each step of a worked solution to make it wrong, had four AI programs translate every step into a Lean statement, and published each statement with whether a prover proved it. They call a proved statement of a wrong step a false positive.
My work
I built a website that shows each of their steps beside the Lean statement each of the four programs wrote for it. In their published results I counted how many false positives do not contain the number that the change put into the step.

In one case a step is changed to say that 3 miles plus 3 miles is 5 miles. The Lean statement written for that step is 3 * 2 = 6, and the prover proved it. Where a change put a new number into a step, that number is missing from the Lean statement in 9,223 of 13,606 false positives (68%).

Limits: I counted in the authors’ published results and ran no translation program or prover again. The authors have not reviewed or endorsed this project.

Lean is a program that checks mathematical proofs.

Open the reader

I use AI to make AI-written maths and code easier to understand.

All twenty-one projects