This one seems to be a problem too:

David is 15 years old. He has a lot of experience riding bicycles. The minimum age to be allowed to drive a car is 18, except when they have experience with other vehicles motorized or not.

I found a prompt sequence that can solve them(beyond step by step) but I was wondering if there's lists or something to see how good the prompts are.

I'm also not talking about not knowing facts that probably just gets solved with access to a fact base or knowledge article.

unless given the fact it still doesn't want to agree with you.

I want to see some that aren't created to be explicitly counterfactual to reality. I mean, to solve these ones so far in this thread it has to go along with lions eating grass and giving drivers licenses to children who ride bicycles. Is that really how far we have to reach, to trick GPT-4?

that's a good point. I assumed it was the complicated logic, but maybe it just has a hard time going against common sense.

After digging a little deeper I figured out it's problem. It thinks your fucking up the problem statement.

here's some text from an interaction with the transparent door Monty Hall Problem:

The doors are mentioned as "transparent closed doors," which would imply that you could see through them and know what's behind each door. However, in the Monty Hall problem, the doors are traditionally opaque, so the contents behind them are unknown to the contestant. The correct description should be "three opaque closed doors."

See: https://github.com/openai/evals/tree/main/evals/registry/eva...