I think it's more because 1. you are storing it in the fridge (which would speed up the staling, but retard the mold formation), and 2. because I only eat bread that's just made with wheat, flour, water, salt, and yeast.

Shop bought bread takes a lot longer to stale and to mold, because it has plenty of additives designed to do just that.