Exactly, not every charity is worthy. Some charity wasting my money on CO2 mitigation has a different value to me than a charity helping battered women escape abuse. ..as an example.
Now think for a minute which one of these actually matters more for continous well-being of mankind, and you'll see why effective charity is a hard problem.