If they occur in nature, esp. in food, our bodies have had time to adapt to them, so that's a distraction. But even restricted to "man-made chemicals" (that can get into our bloodstream and interact with our cellular biology) it's still an enormous class of substances that it's silly to treat as dangerous-until- proven-otherwise - our scientific understanding is a little more advanced than that.
That's not a distraction at all. There are plenty of naturally-occuring chemicals in plants that are deadly poisons to humans. Some of them are even naturally present at low levels in food we eat, such as cyanide in some fruits.
And yes, if manufacturers were creating unnatural quantities of those and polluting the environment with them in a way our bodies couldn't handle, they'd need to be regulated too (some probably are). But the chemicals that naturally occur in food we've been eating for 10s or 100s of 1000s of years aren't the issue here.
