One of the things neuroscience and psychology showed us is that humans always have a post hoc reasoning they can explain, but it's not always the actual reason they did something.
So humans explaining themselves is often misleading data, which may be worse than no data.
Personally, I currently believe that this research is quite probably bunk. There was a recent HN thread in which it was pointed out that some of the results of experiments involved might have been data processing errors. And then there's the most obvious confounding factor - peoples' reported reasoning may not be their true, or complete, reasoning, because people aren't always comfortable with revealing all the details.
But regardless of that, while post-hoc explanations may not be that good for building predictive models of behavior, observed behavior is. We generally know how people behave, and we know the expected variance due to individual and situational circumstances. Truly unexpected behavior is rare in society, and we tend to filter it out. Truly unpredictable people get locked up and/or are given medical help, but even less unpredictable people are tested and kept out of professions and activities where that unpredictability could cause problems.
True, humans are more predictable and we can kind of probe at it until we reveal the truth. I guess we could do sensitivity analysis on ML models but most people aren't equipped for that.
>Truly unpredictable people get locked up and/or are given medical help, but even less unpredictable people are tested and kept out of professions and activities where that unpredictability could cause problems.
Given our knowledge about e.g. Narcissists or psychopaths their behavior is not totally unpredictable. I guess a serial killer could be extremely predictable, but not something we want in society.
Variance reduction isn't the direct reason we lock away people or restrict their opportunities; personal and public safety is. So a serial killer is hunted and imprisoned, and work is done to detect would-be killers before they kill. Variance, however, is a safety issue in many areas - e.g. gun ownership, police service, flying, driving. So we have psychological evaluations not just to weed out the individuals obviously incapable of handling a task safely, but also the highly unpredictable ones.
"humans explaining themselves" is entirely different thing than "humans understanding human behaviour". You may have rationalized your choices, but I understand both your choices AND that you rationalized them.
So humans explaining themselves is often misleading data, which may be worse than no data.