FaithformBench Reader
- Their work
- Rob Cornish, Iacopo Ghinassi and coauthors changed each step of a worked solution to make it wrong, had four AI programs translate every step into a Lean statement, and published each statement with whether a prover proved it. They call a proved statement of a wrong step a false positive.
- My work
- I built a website that shows each of their steps beside the Lean statement each of the four programs wrote for it. In their published results I counted how many false positives do not contain the number that the change put into the step.
In one case a step is changed to say that 3 miles plus 3 miles is 5 miles. The Lean statement written for that step is 3 * 2 = 6, and the prover proved it. Where a change put a new number into a step, that number is missing from the Lean statement in 9,223 of 13,606 false positives (68%).
Limits: I counted in the authors’ published results and ran no translation program or prover again. The authors have not reviewed or endorsed this project.
Lean is a program that checks mathematical proofs.