I was using OpenBSD for my firewalls for a long time, but with the arrival of 10Gbit/s ethernet, I realized that I had to move back to ASIC based firewalls.
Yes, you can forward 10Gbit/s with linux using VPP, but you cannot forward at that rate with small packets and stateful firewall. And it requires a lot of tuning and a large machine.
A used SRX4200 from juniper runs at around 3k USD and you can even buy support for it and you can forward at like 40Gb/s IMIX with it.
I still prefer PF syntax over everything else though.
You can definitely build an x86 system to route 40Gb/s with small packets for under $3k and it's been the case for many years. A Xeon-D can hit 100gbps forwarding and filtering.
OpenBSD is going through a slow fine grained locking transformation that FreeBSD started over 20 years ago. Eventually they will figure out they need something like epoch reclaimation, hazard pointers, or rcu.
I just today deployed an $800 mikrotik in my house that can route 10 gbps at wire speed. on the CPU. with firewall and nat rules applied. no joke. 4 million packets per second is, like, a lot, post-filtering and with any substantial packet size.
This was doable back in 2008 with about $15k of x86 gear and a Linux kernel and a little trickery with pf_ring. The minute AMD K10 and Intel Nehalem dropped, high routing performance was mostly a software problem... Which is cool as hell, compared to the era when it required elaborate dedicated hardware, but it does not make it cheap or easy. Just, commodity. Expensive commodity.
Now you can buy a device off the shelf for $800 that will do it on the CPU, to avoid the cost of Cisco or Juniper, and it has a super simple configuration interface for all the software-based features. Everything you could do in L3/L4 on a Linux platform in 2008, for like, 1/16th the price, with vastly less engineering effort. It is just like, a thing you buy, and it all kinda works outta the box.
No pf_ring trickery, no deep in-house experience, just a box you buy on a web site and it moves 10 gbps with filtering for $800
There's no real magic here: they use absolutely shockingly enormous ARM chips from Amazon/Annapurna. You can build an $800 commodity platform that rivals a $15k commodity platform in 2008, and both of them replace what used to cost $500k.
Is it as good as Cisco or Juniper? oh, certainly not. Will it route and filter traffic at much greater rates, for $800, than anything they have ever been bothered to offer? ABSOLUTELY
I'm really confused by "about $15k of x86 gear ... The minute AMD K10 and Intel Nehalem dropped, high routing performance was mostly a software problem". What kind of $15k machine would you have needed? That's a heck of a lot more than even the most expensive K10 2008 CPU (which according to Wikipedia seems to be Opteron 8384 (quad core, 2.7GHz, 1.0GHz HT, $2149 November 2008), supports up to 8 CPUs per machine, I guess that's what you mean.)
Yes, you can forward 10Gbit/s with linux using VPP, but you cannot forward at that rate with small packets and stateful firewall. And it requires a lot of tuning and a large machine.
A used SRX4200 from juniper runs at around 3k USD and you can even buy support for it and you can forward at like 40Gb/s IMIX with it.
I still prefer PF syntax over everything else though.