One possibility is that speed limits are set so drivers are safe/efficient at the maximum observed traffic density.

If the maximum traffic density doesn't happen often, then people can understandably be annoyed when they can safely drive faster most of the time.