Can it become a proxy for AI companies to collect patient data and medical history or "train" on the data and sell that as a service to insurance companies.

There's HIPAA but AI firms have ignored copyright laws, so ignoring HIPAA or making consent mandatory is not a big leap from there.

That's likely DAX Copilot, which doesn't provide medical advice.