I looked into it. It seems that crossing the border not at a port of entry is a misdemeanor offense and typically results in "only" a few days in jail.

I was confusing that with the US law which says that the asylum process does not require entry at a designated port of arrival.

> Any alien who is physically present in the United States or who arrives in the United States (whether or not at a designated port of arrival and including an alien who is brought to the United States after having been interdicted in international or United States waters), irrespective of such alien's status, may apply for asylum in accordance with this section or, where applicable, section 1225(b) of this title. - https://uscode.house.gov/view.xhtml?req=granuleid%3AUSC-prel...

Since the people are being held for more than a few days, and not as a result of a judgement or plea bargain done over a misdemeanor offense, then "illegal" in the context seems more like a red herring. It's like saying that most people are illegal drivers, because few have never exceeded the speed limit.