> A new blog post shows you how to use Elastic Load Balancers and NAT Gateways for ingress and egress traffic, while avoiding the use of a public IPv4 address for each instance that you launch.
It would be nice if this came with reasonably priced NAT gateways. The current pricing is outrageous.
Not to mention the absurd fact that accessing (IPv4) AWS APIs from a private subnet requires paying for either a NAT gateway or an interface endpoint (we got bitten by sending a ton of Kinesis traffic through a NAT gateway once)
This is one thing Google Cloud does well - traffic to Google services bypasses NAT gateway, even over IPv4.
I was curious how they do this, so I set up a service on Google Cloud Run that just echo'd the user's public IP address. When curl'd over IPv4, it said I was coming from a unique local (i.e. private) IPv6 address. The private IPv4 address of my server was embedded in the address, along with some other random-looking bits that probably identified my VPC somehow. So they must have been doing some sort of stateless IPv4 to IPv6 translation behind the scenes.
It was a clever solution that takes advantage of the fact that all of Google's API endpoints are dual-stack, even though (at the time) they didn't support IPv6 on customer VMs. The problem AWS currently has is not all of their internal endpoints are dual-stack, so even using IPv6 can't save you from cloud NAT costs when accessing AWS services.
Our network is completely software defined, so we just fake it to the VM and make it look like it's talking right to the service, and do all the routing via magic.
Honestly, I really like that the AWS implementation is not magic. AWS is the only one of the big 3 cloud providers where I can reasonably assume I get what it says on the lid, and that it works with the pieces it advertises working with (whereas other cloud providers tend to be more nebulous in their documentation).
GCP especially takes a lot more trial and error building systems that compose a bunch of different primitives. That the API is awful doesn't help either.
I agree with this, having built large dev platforms on both. GCP in my experience takes 2-5x the engineering effort, to deal with "GCPisms" and the terrible documentation. AWS is simpler and does what it says most of the time.
Have quite a bit of experience with AWS and Azure, and only recently learning about GCP, it’s very clear that Google nailed Some of GCP’s core cloud engineering concepts and got them exactly right.
Although unfortunately they will never reach the size of AWS or (maybe Azure? It’s hard to tell Azure’s market size as they don’t disclose it.)
I know Google’s load balancers use BGP. So a load balancer will have a single IP address, but you don’t talk directly to that IP. Google’s servers take over as traffic is being routed.
AWS didn't start with "VPC", and people who still had access to the much-easier-to-conceptualize EC2 Classic only got forced off recently; Amazon VPC wasn't actually launched publicly until after Google Cloud.
Another dead simple solution would be if AWS would provide us a simple subdomain (such as myapp.xxxx.aws-hosting.com), no need to meddle with IPs at all in that case. Google Cloud already does this with xyz.appspot.com subdomains, same with Github Pages as they provide you xyz.github.io subdomain for your app.
I wonder if someone at AWS noticed that the interface endpoint pricing was offensive for accessing S3 and therefore created the free “VPC Endpoint for Amazon S3.”
I would find it rather surprising if the actual cost to Amazon of connecting a VPC to S3 were substantially lower than the cost of connecting a VPC to any other AWS service.
Yeah, the endpoints bother me. I get charging for IPv4 space but they shouldn’t charge you for calling their APIs, especially since it’s one ENI per endpoint so I have a few VPCs which have half the allocated addresses used by endpoints (the old trade off between multi AZ reliability and the cost of allocating redundancy).
AWS NAT gateway is $0.045 per hour plus $0.045 per GB. The hourly fee seems mostly okay - for largish users, one or two per region is fine.
$0.045 per GB is nuts. That’s $20.25/hour or $14580/mo for 1 Gbps. One can buy a cheap gadget using very little power that can NAT 1 Gbps at line rate for maybe $200 (being generous). One can buy a perfectly nice low power server that can NAT 10Gbps line rate for $1k with some compute to spare. One can operate one of these systems, complete with a rack and far more power than needed, plus the Internet connection, for a lot less money than $14580/mo. (Never mind that your $14580 doesn’t actually cover the egress fee on AWS.)
A company with a couple full time employees could easily operate quite a few of these out of any normal datacenter, charge AWS-like fees, and make a killing, without breaking a sweat. But they wouldn’t get many clients because most datacenter customers already have a NAT-capable router and don’t need this service to begin with.
In other words, the OpEx associated with a service like this, including the sysadmin time, is simply not in the ballpark of what AWS charges.
Is that $0.045/GB for all data transferred through it, or just egress to the public internet? If it's the latter, that's half the price of normal EC2 instance egress to the public internet.
If it's the former... oh sweet jesus, what? Probably way cheaper to just run an a1.large or something with Linux on it, plus a very short shell script to set up NAT. That's assuming well more than half of the traffic going through it is ingress from the internet. If it's 50/50 ingress and egress, then it's basically the same pricing as NAT gateway.
> You also incur standard AWS data transfer charges for all data transferred via the NAT gateway.
Yes, the $0.045/GB “data processing” charge is in addition to the usual $0.09/GB egress charge. You are paying an effective $0.135/GB for all of your egress, in addition to the $0.045/hr just to keep the NAT gateway running.
And yes, your ingress and even internal-to-AWS traffic is also billed at the $0.045/GB rate. (An example given on the aforementioned page is traffic from an EC2 instance to a same-region S3 bucket, which they note doesn’t generate an egress charge but does generate a NAT processing charge.) As far as I can tell, the only traffic which isn’t billed is traffic routed with internal VPC private IP addresses, which don’t hit the NAT gateway and thus aren’t counted.
There are highly paid AWS consultants who shave literal millions of dollars off of many company’s AWS bills by just setting it up a cheap EC2 box to handle their NAT instead of using the built-in solution. Doing that instantly wipes out the ingress charges and effectively halves the egress charges, and it’s probably a lower hourly cost than they’re already paying: an a1.large is $0.051/hr on-demand but that immediately drops to just $0.032/hr with a 1 year no upfront reserved plan. If you’re willing to pay upfront and/or sign a longer contract, you can get it as low as $0.019/hr.
I say sorta because it's built on an old version of Amazon Linux and is headed towards EOL with no replacement except "go build your own" as you suggest.
Another thing: EC2 instances (VMs) have a "Source/Destination IP check" which makes them ignore any packets not intended for them. If you want an instance to do NAT, you need to turn this off.
You also have to do it in AWS if you don't want to use the NAT Gateway service and still desire reliability over and above the MTBF for an EC2 instance or AZ, or ever want to do anything requiring a reboot.
For example, rather than simply routing IP packets and then forgetting them, you need to statefully inspect every TCP segment and every supposedly connectionless UDP conversation, you need to maintain state for every live conversation, and you need to mitigate DOS with all those resources.
At that point, you might as well be running a Layer 7 Firewall or an Intrusion Protection System.
> At that point, you might as well be running a Layer 7 Firewall or an Intrusion Protection System.
If you go down this path consider using Transit Gateway so you can route multiple VPC traffic to a central security VPC in a region. I’ve done this a Palo Alto VM and it seems to work well.
UDP is connectionless precisely so you can build novel stateful protocols on it. There’s no promise in UDP that you’ll be able to statelessly monitor it.
UDP is actually more expensive to NAT than TCP is. The reason is UDP fragmentation, which is my vote for the worst, and least forgivable, design error of TCP/IP.
Instead of putting the fragmentation in L4 (like QUIC now does) and including a UDP header on every fragmented packet in a datagram, UDP only includes the header on the first packet. With fragmentation happening; firewalls, NATs, and end-hosts have to buffer and coalesce IP packets based on IP IDs, before the destination can be identified. It's a real nuisance. A lot of CGNAT "stateless" implementations can't handle this and you get very hard to debug issues when there are fragmentation and MTU mismatches.
This is probably more accurately called IP fragmentation (since that is the layer where the fragmentation happens), and a lot of companies make it optional to support in networking gear. I'm surprised that you are using it or seeing it, because it is essentially obsolete today.
It has a legitimate purpose in old-timey systems which have bespoke MTUs on each link, but now the usual thing is to use 1500 bytes for WAN traffic, which is the generic Ethernet MTU, and reserve larger sizes for intra-datacenter communications.
There's a number of UDP protocols that have large enough payloads to fragment. DNSSEC and EDNS0 in particular made it much more common, though the EDNS0 flag day in 2020 partially undid some of the damage by getting folks to ratchet down their EDNS0 buffer sizes.
1500 is absolutely not a pervasively usable WAN MTU, you're going to need pMTUd if you're sending 1500 byte packets broadly. Plenty of WAN links won't tolerate it. If you don't want to deal with fragmentation at all ... 500 is the minimum guaranteed MTU, but in practice it's exceptionally rare to see anything below about 1200 require fragmentation. But you can always only control what you send, not what others are sending you.
One thing I've learned since joining Fly.io in 2020 is to laugh when people point to the 1500 MTU. You absolutely can't count on that: IPv6 cuts into it, and so does every additional layer of encapsulation on your path.
Yeah, you have to account for the headers in the 1500 byte MTU, which I suppose can be substantial if you have several VLAN tags, IPSec, IPv6, and a bunch of IP options. Presumably most of that encapsulation happens inside a datacenter, though, where you can use jumbo frames.
Even well-behaved unfragmented UDP should be more expensive to NAT because it doesn't have an end-of-stream "FIN" marker, meaning stateful middleboxes need to retain state for longer because they can only time out.
TCP does not use IP fragmentation, and the IP packets are marked "Don't fragment". TCP performs its own fragmentation and every packet gets a TCP header in its leading section. A NAT, Firewall, or end-host can L4 route the TCP packet as-is and does not need to correlate with other packets.
Edited to extend: this is why TCP has a "Maximum Segment Size", and why Path MTU Discovery information has to be passed into the TCP state machine. It is TCP that takes responsibility for carving up the data into the packets, not IP.
One of the goals of UDP was to avoid needing this kind of state, which is why the IP layer handles fragmentation for it instead. This is allowed on a hop-by-hop basis, unless the DF bit is set; so when a "too big" packet gets to a node with a smaller MTU, it can just split it and send on the fragments. No PMTUD needed.
The design could have been for the fragmenting node to also add a UDP header as part of that process, but was not. It would have been a simple change at the time. It's had a lot of consequences since and is responsible for a decent amount of complexity in hardware and software packet pipelines.
Several other protocols solve this in a layering agnostic way by simply having a header length field. The header bytes can then be copied without any understanding of the format. This is even how IP's own ICMP protocol knows how much of an IP packet it should (at least) include in an error message so that the sender can know what triggered the error.
TCP, UDP, ICMP and IP were all designed contemporaneously; UDP fragmentation could also easily have just been specified for. It's just an odd regrettable quirk.
Also, if you get UDP completely right, do you need any other IP protocols? The whole point of UDP is programming directly to the datagram interface. Before IPv6 you could even disable the checksums.
MSS was also super annoying for me doing re-encapsulation of TCP packets! We wanted to do eBPF cut-through routing of TCP connections for WebRTC stuff, where proxy bounces would be problematic because connections need to live a long time. If you're shuttling packets around, you're going to eat into the MTU with your own headers. 99.9% of our TCP connections weren't cut through so we don't want to dial in new settings into VMs for that feature, so we did it in eBPF, and parsing/adjusting TCP headers in BPF C (pre-bounded loops!) wasn't fun.
Which is why game networking libraries put a lot of emphasis on NAT traversal, forcing NATs to recognise the "connection". And why game console manufacturers tell users to just forward all incoming traffic unmanaged by the NAT to the console.
This is missing the point mostly, my own sites have supported ipv6 for a going on a decade because it was fun to get it working. But that's a very different thing than supporting only IPv6.
It's best for an ISP to deploy IPv6 and CGNATv4 in parallel, so the NAT only needs to handle traffic for services that don't support IPv6 (e.g. news.ycombinator.com)
NAT and Stateful firewalling are commonly bundled together (especially on home systems) but I would not go so far as to say “NAT has a stateful firewall”-
I hear such takes all the time and its really frustrating; usually in threads regarding IPv6, incidentally it is usually programmers who think they understand everything about networks because they know how tcp operates.
In almost all NAT implementations, public-side ports are dynamically assigned, which implies that inbound connections aren't possible (unless port forwarding is explicitly configured).
Is that really conceptually so different from a stateful firewall allowing inbound packets only for established connections/flows?
"NATs are good because otherwise people wouldn't have any firewalls" is a tired take, yes, but I don't see the point being needlessly pedantic about the semantics of NAT vs. stateful firewalls when in this case, the effect is the same: No inbound packets without prior outbound packets (or a connection establishment for TCP).
$40/mo is outrageous? We spend thousands a month on AWS and drive most traffic thru a single NAT gateway. It's rock solid and it "just works" without any fuss. Totally worth it.
yep, and they should. aws has never really been suited to the hobbyist. does it work for that? of course. is it most cost effective? absolutely not. is it cost effective for people who need the resources? yes.
I run a number of personal projects on AWS entirely on their serverless offerings and pay $0 outside of domain registration as I'm well within their free tiers. That seems pretty cost effective.
Yes, if you can abuse the free tiers, you can essentially run a small SaaS company for free. Once you scale past that point, you are on the hook for a (probably much too large) bill, when you could still be using the same $5 VPS.
You must be talking about renting resources that run 100% of the time. AWS rents us gpu instances by the second. We have to run sporadic jobs throughout the day that take 50 seconds to two hours. Depending on customer activity we might need to run 10 or more at once, or we might lie idle for an hour. The elastic economics are unbeatable.
so I have an HTTP endpoint which gets maybe 10 hits per day, and does some lightweight computations and records small amount of data.
Right now, this is done on AWS, with lambda + S3, and costs under $0.02/month.
Can you point me to something more cost effective that that? Don't forget I also need backup for data, automatic failover in case of machine failure or crash, amd no maintenance (like OS upgrades) for 5+ years.
Stateless applications are far cheaper than stateful applications to host on AWS. Computing is cheap, object storage is where AWS make unreasonable profit and lock you in their platform.
That does not answer my question though... I am not spending $600,000 on hardware!
I have no doubt that there are plenty of cases when local hardware is cheaper, but gp said "There is no possible use case in no possible universe where AWS is cost effective."... and I claim there are many use cases where AWS is cheaper.
Which hosting companies are there which are SoC-2 compliant and are 20 times less cost effective than AWS?
Enterprise workloads need compliance. AWS and GCP provide that. They are very few hosting companies out there who are better at security and compliance than those two.
Renting the same compute resources might cost you less but you are on the hook for maintenance and administration which can cost you more in the long run.
The issue tends to be that people do not actually stay on top of their spend- they claim to need less headcount but then spend more than a few salaries worth on their cloud spend.
They claim they do not need headcount but then spend the same headcount in infra people anyway, or finops people in the best case.
people have lost touch with how much compute actually costs, because its little by little and claims to scale to zero or you only pay what you want. - yet every installation I’ve ever seen has had a base cost higher than the largest colo installation cost we would have needed times 2.
Its not cost effective, because its on average 11x more expensive than a fully managed colocation installation. - your packets dont care that you spent 11x more on half the performance.
And all those cloud compute instances are probably strangled for io if there is any real load.
Colocating your own equipment is going to give way better base performance. Compute is not just processor and memory, it's also dependent on network and disk i/o. Disk is often overlooked because modern disks are so fast, people don't even realize it's crippled in the cloud.
It really is crippled to an absurd degree. A basic RDS install with a gp3 volume will get 3k IOPS and 0.125 GB/s transfer if your volume is under 400 GB, or 12k IOPS and 0.5 GB/s transfer if it's larger. The monthly per-GB storage cost for RDS is the same as the capital costs to buy the disks in a mirrored setup. Meanwhile, if you bought the disks, you'd get over 100x the IOPS and 10x the transfer.
For a provisioned IOPS volume, you can get up to 256k IOPS (so still a fraction of a single drive) at a cost of $25600/month (plus per-GB storage costs). For that price, you could buy 8x of these: https://www.newegg.com/micron-30-72-tb-9400/p/N82E1682036315... giving you 240 TB of raw SSD storage.
I ran into a SaaS company recently that had a guide for how to setup a white-label domain using route 53 and Cloudfront for one of their services. The SaaS company charges for service bandwidth usage, and they host their infrastructure on AWS, so if you opt to follow their guide they get a fat margin bump in the form of avoiding an egress charge and you get to be double-charged for bandwidth. You've gotta love it.
If I follow what you're saying I suppose my understanding could be wrong, but there's no "cloud transfer" required. It's just a matter of both the distribution and the configured origin being on AWS. If the origin doesn't direct traffic outside of the datacenter, AWS doesn't bill that as egress to whoever owns the origin. The Cloudfront distribution, on the other hand, will take the hit once it exceeds the free tier because it's the AWS service distributing data to end-users. It has to make a request to the AWS internal service then cache it, so The SaaS lambda or s3 bucket or EC2 instance or whatever they're using is none-the-wiser. It's just how the AWS billing mechanism works.
$40/month just to run it, but then $0.045 per GB data rates. The data rates are what is outrageous. NAT Gateways comprise a non-trivial portion of many customer's bills for this reason.
Exactly, it is $32 just to have it turned on 24/7 and then you pay additionally. Looked into it as it is the only(?) way to get dedicated IP for Lambda which is a common use case, which also explains why it is so costly. They lure you in free tier and then charge for all the necessities.
Amazon owns millions of IPv4s which they purchased for probably less than $5 a pop. So it completely pays for itself after less than a year then it’s just free cash flow
I completely agree. It’s odd they would announce charging for dedicated IPv4 while not having a free shared egress solution (unless I’m misunderstanding).
I would expect them to reduce NAT pricing in the long run, but who knows.
I'm shocked this isn't a feature of a VPC out of the box (shared internet bound traffic). You should only need a NAT gateway if you want the traffic to come out of a single set of external IPs that you control.
Almost all of my use cases I could easily ride out to the internet through a shared pipe (apt updates and such) and don't care whatsoever what IP that exits the AWS network from, since I'm not applying firewall rules or anything.
I think that as a business and given the fact they are now charging for a previously free service (public IPs), offering a now paid service as free would nullify the reasons for doing what they are doing. They don't owe anyone anything for free.
Look I’m the last one that will bang the capitalism, consume, spend, drum, but nothing is ever free. You want a free nat box? Your managed database cluster just got a fraction of a fraction of a cent more expensive every hour. You would never get it for free even if they billed it as such. They wouldn’t be the biggest company in the world if they were in the habit of using their profits for your bills instead of the shareholder’s dividends. This is a business, not a family style steak house.
That’s not anything new. They’ve been scarce for a decade and they’ve footed the bill this long so why change it now other then someone noticed the opportunity
How much does the NAT gateway cost? Quick search didn't turn up anything (and I don't care about this enough to spend more than a few seconds on it). You can turn a regular EC2 instance running Linux into a NAT box by giving it two network interfaces (hell, you can even do it with a single interface) and a few shell commands; I wonder if that's cheaper, even including the price of the public IPv4.
Edit: I see from another post that NAT gateway costs $0.045/hr + $0.045/GB of transfer. That seems... not terrible? An a1.large on EC2 is $0.051/hr + $0.09/GB transfer to the internet (which I assume this type of box would be doing a lot of).
100% agree, they need to offer steep reserved instance pricing for NAT gateways. To deploy 3 NAT gateways (HA one in each availability zone) is $99/mo just for the instances.
You don't have to use AWS' appliance (the NAT GW) to do NAT. You can NAT your traffic yourself from a t2.micro Linux.
AWS used to maintain a AMI to do just that, nowadays you have to do it yourself, but it's honestly not much more than adding 2/3 iptables rules.
I find this trade-off to be exactly the reason why AWS is so good even for small startups. You can bootstrap something quickly, though it will be a tad expensive.
And if you need to down your costs later on, you start chasing the quickwins like maintaining your own NAT gateway. The same could apply for all managed services.
Maintaining your own OpenVPN VS AWS VPN.
Maintaining your own Postgres VS RDS.
etc
"Cheaper" is not the only dimension to consider. Using managed services is also faster, more reliable, and scale better. You cannot just get that for free.
You can stand up your own on top of a t3.micro or something if you don't care too much about HA (e.g. you just wanna be able to hit the internet when SSHed into your instances).
What kind of workloads require a lot of NAT gateway usage?
I think my team's use is kind of high, with 16 TB going through NAT last month. The bill for that came to ~1300, which is higher than I'd like, but that's only about 1.5% of our AWS spend. Tbh I never really looked at the spend for NAT before, but this doesn't alarm me.
Last time we used GCP's NAT gateway it was constantly dropping SYN packets.
We had to revert to using External IPs on machines that talked to the wider internet.
It would be nice if this came with reasonably priced NAT gateways. The current pricing is outrageous.