I also recently came across some of these fake company campaigns with attached employee profiles. It's very hard to distinguish them from legitimate companies.
It's especially hard now that many legitimate companies use a lot of generic sounding AI-generated content, which seems to be same approach the spam/phish/malware teams are using.
IMO we need some kind of zero-knowledge proof system that can be checked to verify if a message sender is a US citizen, employed by who they say they are employed by etc.
I don't see how we can trust anything in a post-generative AI world any other way.
>IMO we need some kind of zero-knowledge proof system that can be checked to verify if a message sender is a US citizen, employed by who they say they are employed by etc.
I think this could be a great opportunity for Google. Lots of organizations already make use of Google Workspace/Gmail. Imagine if Google Workspace offered the equivalent of a "Twitter blue check", where you pay extra, and anyone who views your email in Gmail sees a little check mark next to it, that shows Google verified you are who you say you are, and Google thinks you're not malicious. Salespeople sending cold emails would love it.
I don't think you can solve this problem purely cryptographically. An attacker could always bribe a US citizen to set up a shell corp or whatever. Most objectively verifiable indicators can be gamed. There has to be an organization that's good at security, like Google, which is in the business of continuously keeping up with adversaries. Actually Google might not be the best because they kinda suck at tailored customer service, but anyway.
It's especially hard now that many legitimate companies use a lot of generic sounding AI-generated content, which seems to be same approach the spam/phish/malware teams are using.
IMO we need some kind of zero-knowledge proof system that can be checked to verify if a message sender is a US citizen, employed by who they say they are employed by etc.
I don't see how we can trust anything in a post-generative AI world any other way.