> While the SLA says 100%, don't expect perfection
When you have an SLA, understand what it is: a financial arrangement whereby you can request a prorated refund for certain types of outages. It is not in any way a guarantee on the part of a provider that you'll experience even average uptime equaling or exceeding the SLA, just that they can pay out the fraction of customer requests for service credits they receive for the covered outages they have and still make money.
The reality for the type of service the author of this post purchased is that for any physical damage to the fiber plant, he will experience hours of outage while a splice crew locates and repairs the damage. Verizon might offer a 100% SLA, but they didn't engineer it to even five nines of availability. That would require redundant equipment and service entrances at his premises along with path diversity end-to-end.
It's still a very high motivator to keep the service up though. It's not a guarantee of anything as you said but I've been on call for this kind of contract.
And then you get very good at pointing fingers too. Not too sure about this though :D
The reality of business is that a contract is only worth what you can enforce. SLAs are usually worthless:
1. Unless you are a large customer who accounts for an important amount of their bottom line, probably you have little financial leverage with the vendor.
2. The amount at stake in the SLA is not worth going to court for. It's unenforceable and in fact, the amount is usually meaningless.
Let's say you pay $10,000 per month for your 100% SLA dedicated circuit, and it goes down for an entire month. Let's say the vendor doesn't get around to paying you. Is it worth hiring a lawyer to collect $10K? Is it worth distracting you from your job emotionally and mentally, and consuming many hours of your time? Probably not.
Let's say your circuit is down for 3 hours. Let's say the SLA even pays you 3x what you pay for the service for any downtime (most I've seen just refund the money for that time). Let's see: ($10,000/month) / (720 hrs/month) = $13.89/hour. The SLA pays $41.67/hr of downtime, or $125 for your downtime. Is it even worth figuring out how to apply and filling out the form? No. You have much bigger issues in business, and if you don't ... well, then you have bigger issues.
3. The cost to the vendor is reputation: You tell your peers how much your service sucks, and word gets around. I've had techs take disinterested attitudes toward their poor uptime on our circuit; when I've called the account managers, they can have a very different response - they want me spreading their name around in a different way. That has nothing to do with the SLA.
> Verizon might offer a 100% SLA, but they didn't engineer it to even five nines of availability. That would require redundant equipment and service entrances at his premises along with path diversity end-to-end.
Agreed. There is no substitute for the physical reality of the circuit and service, which you should understand if you are buying it. Putting a shiny SLA on it will have no effect on the outcome.
The ISP I own will give you dedicated if you desire. We charge an uplift for the installation (due to additional splicing) and a little bit more on the monthly price for the consumption of dedicated ports and cores rather than using the PON. However, as is the case with most provided services like this, the majority of the cost is covering the risk of the SLA. The likelihood of an outage is not too dissimilar to the likelihood of outage on PON, so really it becomes a financial and service guarantee more than an uptime guarantee. As a healthy middle ground, we will also offer BGP on regular services and we do a bit of a referral system with a couple of other friendly ISPs, who will also do BGP. I actually prefer multi homing to “dedicated” from a resilience perspective because it separates you from the entire network stack all the way to the transit and peering.
Perhaps you’ll be able to help. Why do ISPs, including yours, not provide a map of service? Almost every time it is this “check if you have service” API. Every time I have moved I have wanted to look for areas with support under various ISPs and each time it’s an annoying process of sending each address to this API, dealing with rate limits, anti-fraud etc.
One time I contacted the business contact with a bunch of addresses and that was useful but the round trip is so slow I was better off just querying.
We actually used to maintain coverage maps, but after expanding through several huge providers such as Cityfibre and Openreach it became silly to maintain.
Certainly in UK that could be also be quite a risky thing to do, as it could expose build strategies to competitors. That said, we expose an API to Better Internet Dashboard so they can do map based coverage exploration.
(Potential non-commercial customer) Checked the help page - couldn’t see anything about static IP addresses, IPv6 etc?
Edit: Ah, “Our network. You acknowledge that We may change your Internet Protocol (IP) address from time to time without giving notice unless You have purchased a fixed IP address from us;”
The website is honestly not the most loved, however you get a static v4 address and v6 network as standard. We reserve the right to shift them if we need to for network reorg, though that has never happened to date.
I had "real fiber" run to our farm. 18 miles of cable was run and it took a year going back and forth. Originally it was a priced at $4500, which was to be a business expense. After install I had consistent issues with performance, after doing a lot of work to show them the issues and threatening disconnect, they upgraded the circuits for proper speeds and knocked it down to $2100. It's definitely still a car payment but it's much cheaper than it was.
Ultimately I get the SLA, Direct access to cloud providers maximizing performance, i'm also able to host a few IP blocks which allow a couple internet facing machines.
The home we sold recently had it pretty good too though, ended up with 3 5gig AT&T lines (no redundancy obviously) for only $450 a month total. That was pretty darn rad, even if the SLA wasn't the same.
Benefit of working from the farm is that I can also snag some bw for personal use ;D
In most cases if you have Comcast, your best option for a DIA is Comcast Business. Which again, is drastically better than Comcast's residential service... but it's still Comcast.
Comcast Business by itself is not DIA. You are still on the same CMTS as residential users. So you share the same resource pool. I have had Comcast Business for ~14 years and still fight with normal residential problems.
Such as over-subscription and having to contact the BBB to finally get a non-"Let me look at my book, ah yes! it's your modem" response. I finally was contacted by the Technical Operations Manager to affirm "Yes [name] is correct, bandwidth demand exceeds the capacity of the system in their area. We are working on a permanent solution to allocate more bandwidth"
That was 9 years ago, and I'm back again. I pay for the catchy "UP TO" 35 MBit/s upstream and can barely hold 2 MBit/s during peak and about ~25 outside peak.
Not all Comcast Business, no. There are different tiers of product for sure. I have "normal" Comcast Business in my house, where I have basic Comcast coax service for a little more money with some better service guarantees and a little less nonsense. (The biggest upgrade is being able to email an account manager. The biggest downgrade is no bundling with TV because Business won't install it in a house, so if someone in your house wants Xfinity TV you get two unbundled bills.)
At work we deal with the sort of folks in this blog, where adding a link between a couple sites requires four or five months, multiple teams boring new fiber runs, etc.
I’m just on residential Comcast and the slow upload speeds are such a pain in the butt. I understand most of the users in this area use the Internet to consume from, but some of us want to serve stuff too!
My measured upload was 26.2Mbps. They offered me an upgraded upload speed package, I accepted it and am paying for it. It is still 26.2Mbps. I rebooted the CPE, waited months. Nothing changed.
Starlink shared bandwidth amidst their network makes Comcast look generous.
Between fighting for bandwidth amidst everyone else going to the same base station, random assigned IP addresses that occasionally end up with accusations of pirating you had no partaking in, and storms messing with your signal quality, I would heavily advocate against any reliance on it for business related operations.
> Starlink also does not offer static IP addresses.
They do, albeit I don't know the exact details as to pricing and everything. My company uses Starlink as a backup WAN connection at one of our sites, and it has a static IP.
> Although truly static IPs are not available, a reservation system retains the public IPv4 address and IPv6 prefix even when the system is off or rebooted.
You have a potentially stable address, but not a truly static one.
Starlink is competitive with rural lines which tend to be DSL over unmaintained copper. If you have cable you will probably do better with that. If you are in a dense ish area just about anything will do better.
In my experience it's just a case of knowing who to call and what to ask for. We are served by Spectrum. They will drop fiber into any building in their HFC service areas. We had it into an apartment building for several years then moved to another building with only coax, but they quoted me $2000 install for fiber. Problem for me is that the fiber service is symmetric and so to get the kind of download needed for Netflix you have to pay for 200M+ upload. That's quite expensive for me so I passed. They don't offer 95% billing.
In this case, he said it's a fun and interesting way to add resiliency to the existing co-location they have.
I frequently see people default to AWS, without any consideration of any other options. If you're running beyond a couple of small EC2 instances, it's worth looking at other options such as colocation. 37signals wrote about their cloud exit and how much they saved.
I have a few machines I use to mirror / duplicate data from my tenants and client tenants when working on larger projects. It makes it much much easier.
While Egress pricing is a pain in the ass on AWS, that's usually a small fee on the customer side comparatively.
Good luck when the fbi inevitably kicks in your door after running a tor node in the US on a datacenter connection in your name, with the physical hardware sitting in your home.
Hey! That's my setup as well! I have one DIA connection and a backup VPN over a shitty Comcast business connection that gets terminated in a nearby datacenter.
Getting an ISP to even _talk_ to me required quite a bit of sleuthing. And I was saying from the outset that I was ready to fully pay for the fiber run.
Apparently, ISPs in my locality actually divide the city into the service areas. How the heck this is legal, I don't understand.
Some tidbits from me: my ISP installed a big honking ADVA optical line terminal on my premises. Getting them to move it to their side and just provide me with an SFP connection is still my work-in-progress.
The support is also outsourced into India, and getting them to understand what you want over the phone is... painful. Fortunately, the web ticket system is good enough.
Isn't it very typical for an ISP to run their line onto your premises and put the demarcation device on your premises with your power supply etc? Usually there isn't a "their side" where it still works. It's your premises, then up to kilometers of wiring, then the central exchange. Equipment required to terminate the wiring at your end obvious has to be at your end, not theirs.
It's possible that the termination equipment could be a bare SFP and not a big box, but the ISP wants to be able to monitor the status of your connection up to the termination equipment, because that is their responsibility to keep online, and they can't do that if it's just an off-the-shelf SFP. They probably wouldn't agree to do it and still have any SLA.
If there's a physical problem with the box (too big/loud) you can try negotiate for a different box but if you just want control over your network, sorry but that just isn't how it works. Your network starts at the demarcation, and you don't want to be responsible for speaking whatever protocols the ISP is using internally, either. Up to the demarcation, it's ISP internal network, and past that, it's your network, with a standard handover interface like Ethernet.
> Isn't it very typical for an ISP to run their line onto your premises and put the demarcation device on your premises with your power supply etc? Usually there isn't a "their side" where it still works.
There is, in my case. Their point of presence is about 800 meters away, and they have a simple switch that aggregates connections. Their optical terminal on my terminal (a switch-sized 1-U rack-mounted box with ANNOYING fans) does all the traffic shaping, authentication, etc.
> It's possible that the termination equipment could be a bare SFP and not a big box, but the ISP wants to be able to monitor the status of your connection up to the termination equipment, because that is their responsibility to keep online, and they can't do that if it's just an off-the-shelf SFP. They probably wouldn't agree to do it and still have any SLA.
Sure, and having equipment on my premises makes it much easier to debug the issue. But they actually monitor the BGP session state, not the optical path.
This depends on the technology. When I had a PON connection (FiOS), the ONT belonged to the ISP but was on my premises. However, with the cable Internet I have now (Xfinity aka Comcast), there is nothing between me and the street save a splitter and PoE filter, both of which are passive.
But you have a cable modem. You either have one provided directly by Comcast or you have one that adheres to their standards.
It used to be that you had to get a cable or DSL modem from your ISP. Now, cable and DSL networks are standardized enough that there's a good chance you can get a third-party one to work. It will be the same with PON in time.
I think it's fine to be required to use your ISP modem, since different ISPs have different physical layer networks. It starts to suck when they try to force you to use their router. I think in the EU it's actually a legal requirement that if they do, it has to support bridge mode (a.k.a. behave as only a modem).
> Now, cable and DSL networks are standardized enough that there's a good chance you can get a third-party one to work. It will be the same with PON in time.
It’s mostly the same with PON now. I’ve always had success with the FS.com PON SFPs, once I get the necessary information for the connection to program them with (the difficulty of which can vary from “ask the tech installing the connection nicely” to “take apart the ISP-provided CPE and solder wires to the debug console pads”).
Regarding the city divided into service areas, if you are in the US many cities provided franchise rights to cable providers that gave them exclusive monopoly rights to provide services in exchange for the cable provider spending all the money to install the lines and infrastructure. Most of those franchise deals were done in the 1970s or early 1980s, essentially mimicking the agreements that were in place for AT&T (or RBOCs).
Franchise agreements have been non-exclusive by US federal law since a long time.
Lack of competition is more about the cost to establish service in an area, and the ROI on service in an area with competition. It costs a lot to pull wires past a lot of potential customers and if many of them won't sign up because they already have a good enough option, it doesn't make much sense to do it.
Cable and Telco compete because when cable was built, it was a completely different service, but they've both evolved to fill the same role.
This is why mandatory line sharing is important for competition, and it's in the telecom act of 1996. But the FCC first said it only applied to telecoms, and then said it doesn't apply in remote terminals because of lack of space, and then courts said it doesn't apply at all because telecom and not cable isn't fair.
In my very limited experience, this is common even for real business customers. I think it's because going from commercial ISP service to BGP service is not really an upsell. It's a completely different product category (carrier interconnect) and usually results in less revenue for the ISP (greatly reduced bandwidth charges etc.). As a result, sales folks aren't trained on it, and it is difficult to get through regular channels.
If you get it, it can be great. Imagine your ISP calling you when you reboot your router.
I'm running BGP to the home, over wireguard, from a couple of VPSes. It wasn't worth it to upgrade to DIA for me. It's mostly for hobbyist purposes so reliability is not really a huge factor.
This sounds like so much fun. Thank you to the author for writing this up and sharing the blow-by-blow.
While fascinated with the network stack, I've only gotten as deep as reading Illustrated TCP/IP and pretending I understand tcpdump. I would love to rovel around in BGP and, um, all that jazz.
Any suggestions on how to get started? My vague understanding is that most people get apprenticed into this stuff through work. Are the relevant systems involved just too expensive and locked behind corporate walls to be amenable to autodidactism?
Oddly, my recommendation for getting started is non-technical (and assumes you’re in the US): if you don’t have one already, stand up an LLC. Doesn’t need to be anything fancy; your state’s Secretary of State probably accepts an online form. I’m in CO and at its most basic form this costs $25 a year.
You’ll need some kind of business entity to have easier conversations with ARIN, which is the starting point for getting yourself an ASN.
Once you’ve got an ASN, you have an entity that can “own” IP blocks instead of just relying on other networks to handle that for you. Now, have a look at Neptune Networks’ offerings—they’ll rent you an instance for a reasonable monthly cost that they’ll allow transit to and from. Note that their smallest instance size doesn’t have enough RAM to store the global internet’s full BGP table; this will matter only once you know what that means.
That’ll definitely get you started, and you can learn a lot on the cheap before even looking into your local colocation options.
Even if you're in the US, you can go through a European LIR and get an ASN there as an individual. They will want you to have "infrastructure" in the EU (a VPS is fine.) Generally, it is much cheaper than ARIN.
You're right that a lot of this stuff is learned on the job, but you can get really, really far with the right tools and the right resources and only a modicum of financial investment (if you're set on getting some hardware). Even more so now that things have open source versions. It makes it super easy to start out. That's one of the great things about this field, it's all built on more-or-less open standards. You're not far from the original RFCs used to create all of this: RFC 791 (IP), 793 (TCP), 4271 (BGP4), 9499 (overview of DNS), and many others. They can be dry, almost too much, but for the most part that's the gold right 'thar.
Stevens' book is also a stupendous resource for the down-and-dirty, so good work on starting there. Beyond that you need to start just building things.
Almost every virtualization suite allows you to create network resources (or at least it abstracts the low-level OS calls or commands required to do so). Set up two VMs. Make them talk. Break that link and learn how to repair it, using the tools that you've mentioned you are now using, tcpdump in particular. Figure out how ARP works at a low level, or NDP (neighbor discovery) if you're running IPv6. Learn how to subnet, too! Then work your way up the stack. Set up a VLAN interface, set a 802.1q tag on an interface, try to get two or more vlans to talk to each other, route between them. Break that. Set up a basic OSPF area. Set up a BGP adjacency between two private ASNs you have created. Redistribute routes among different protocols. Set up higher-level services like DNS. Set up a play anycast network on your local host. Play around with load balancers and web servers. Play, break, fix, repeat. That's pretty much what the 'professionals' do all day anyway. It all comes from practice. Software like BIRD, quagga, nginx, haproxy, ip/nftables, dnsmasq/powerdns, etc etc.
When you think you've exhausted the software side of the above tools and beyond and want to lay your hands on some actual hardware, look at picking up a 'white box' switch, a cast-off on eBay from the likes of Quanta/QCT, Edge-core or others. Don't spend more than a couple hundred bucks on this. Throw an ONIE network os on them (I suggest Sonic for open source, or if you want to pay, Cumulus) and start using 'real' hardware and play around with that. Learn the basics of sfp transceivers, fibre optics and the different mode types they come in, direct attach cables, port channels and the like. You can find super cheap transceiver hardware, fibre optic patch cables and all that at a discount vendor like fs.com, or from ebay as well. Learn how to interrogate the firmware on those, find out power transmission levels, error rates and other system info.
There's a sibling comment here suggesting you start out by setting up a LLC and going to ARIN and getting an autonomous system number. Please ignore that advice. You will be just wasting your time and your money and be distracted for no reason until you have the most basic of foundations. Use the abundance of resources you have to learn first. If you really feel like you want to take that next step, then be confident and do it!
Like a lot of things in this industry, the complexity can get fractal in nature the more you look at it. Don't let that overwhelm you. Take it one step at a time and don't be afraid to break shit, fixing it is how you learn best.
> ISPs tend to oversubscribe these services as well (where you and your 10 neighbors might all be able to sign up for 1Gbps symmetric service, but not everyone can leverage that full 1Gbps at the same time).
How does this work for FTTH? I know nothing about fibre optic networks. I had the impression that each subscriber has their own wavelength, or rather a range of wavelengths that captures their bandwidth, and that does not overlap with other subscribers.
Otherwise I have no idea how passive optical networks could even work.
In a typical passive optical network, one PON port is connected to 128 clients through the use of PLC splitters - unlike a WDM splitter which will insert or remove a specific wavelength, these simply split the whole signal. Where I work, that is a 4-4-8 configuration using 3 layers of splitters.
The OLT (optical line terminal, head-end) will tell each ONU/ONT (optical network unit/terminal) how much airtime they can use to transmit - each ONT will take their turn in transmitting so as not to interrupt others. Part of this calculation is the distance the ONT is from the OLT - each ONT will be a different distance depending on the geographic location, which means each ONT will have a different latency. The ONU can request additional airtime if it has a large amound of data to transmit. The amount of airtime the OLT will allocate depends upon the CIR (committed information rate i.e. what will the ISP guarantee at a minimum) and the PIR (peak information rate - the maximum rate based on the subscribers service).
> How does this work for FTTH? I know nothing about fibre optic networks. I had the impression that each subscriber has their own wavelength, or rather a range of wavelengths that captures their bandwidth, and that does not overlap with other subscribers.
The keyword to search for is GPON. It's multiplexed, each subscriber receives a few time slots in a shared wavelength. The transmissions from the subscribers don't collide because their time slots don't overlap.
Verizon must make some killer margins on a connection once it’s up and running, given they’re willing to eat the cost of 4 employees and a police detail splicing fiber in manholes for a week.
Back in 2015, Time Warner Cable testified they make a 97 Percent Profit Margin on High-Speed Internet Service for residential customers. I would guess Verizon is doing okay.
That was because according the accounting infra is mostly amortized for 'triple play' and TV was ( is ? ) highly profitable. Basically 97% math assumed that infra was built for TV channel delivery and the Internet is 'free bonus'.
I don't work for TWC and have no their services. However friend of mine worked in past for a regional competitor of TWC around 2010 and explained the logic above.
Very similar to the process I took getting DIA to a commercial building. They said they spent $60,000 pulling fiber half a mile through a business park (we didn’t pay this). We only have one ISP device in our rack (Verizon truly ships their org chart). We paid our contractor to have fiber installed from the DEMARC to the server room but apparently the ISP would’ve done that for free, oops.
And yeah the quality of customer service we’ve gotten from three different business providers has been exceptional. It’s crazy to have actual engineers you can call who know what’s going on. You get what you pay for.
This was about 15 years ago, back when I was working at $MEGACORP we had an OC-48 running to our lab. We were having some problems with it. Thanks to 20 years of near constant layoffs all details of who was responsible for that circuit on our side were lost to the sands of time.
I went down to the basement and saw a faded UUNET sticker on the demarc, but there was no circuit id on it. Some googling showed that through the years of corporate takeovers they were now owned by Verizon. So I called Verzion Business and explained the situation. The lady spent a hour on the phone with me, but we tracked down the circuit. The address listed on the circuit was a manhole up the street from our building. They dispatched a technician and we were back up and running in about an hour. They also put new circuit ID label on the demarc so we wouldn't go through that again.
When you have an SLA, understand what it is: a financial arrangement whereby you can request a prorated refund for certain types of outages. It is not in any way a guarantee on the part of a provider that you'll experience even average uptime equaling or exceeding the SLA, just that they can pay out the fraction of customer requests for service credits they receive for the covered outages they have and still make money.
The reality for the type of service the author of this post purchased is that for any physical damage to the fiber plant, he will experience hours of outage while a splice crew locates and repairs the damage. Verizon might offer a 100% SLA, but they didn't engineer it to even five nines of availability. That would require redundant equipment and service entrances at his premises along with path diversity end-to-end.