>The only way this gets fixed is if there are consequences at every level for false positives.
Do we really want consequences for false positives? If a kid is smoking a cigarette in the bathroom and the smoke detector goes off, the school should evacuate. The Smoke Alarm went off. No principal is going to sign off on the assumption that "Timmy is smoking, it's not a real fire". The principal shouldn't be punished for responding to the alarm. Timmy...probably should get reprimanded, but that feels off-metaphor.
In the example we are given, Timmy did nothing wrong. Having a clarinet is not contraband, and he should not be punished. The admin who called a lockdown did nothing wrong, as they were responding to the system in the way they were trained to use it. This is all in the name of safety, where things are done in 'an abundance of caution'.
>"It's not my fault the cops shot the kid, the system said it was a gun."
No, its the cop's fault. The cop hasn't been trained to use the AI security system, and is instead given their own SOP for assessing threats.
Do we really want consequences for false positives? If a kid is smoking a cigarette in the bathroom and the smoke detector goes off, the school should evacuate. The Smoke Alarm went off. No principal is going to sign off on the assumption that "Timmy is smoking, it's not a real fire". The principal shouldn't be punished for responding to the alarm. Timmy...probably should get reprimanded, but that feels off-metaphor.
In the example we are given, Timmy did nothing wrong. Having a clarinet is not contraband, and he should not be punished. The admin who called a lockdown did nothing wrong, as they were responding to the system in the way they were trained to use it. This is all in the name of safety, where things are done in 'an abundance of caution'.
>"It's not my fault the cops shot the kid, the system said it was a gun."
No, its the cop's fault. The cop hasn't been trained to use the AI security system, and is instead given their own SOP for assessing threats.