I've been running BGP at home for the last 10 years, and in my area, most ISPs prefer to use Dedicated Internet Access (DIA) for BGP sessions. Fortunately, I live in a place where there are a few smaller ISPs with around 2000-3000 customers that are willing to run BGP with me as a home user. They're impressed that I have BGP, ASN, and IPs as a home user, and I think it's pretty cool too. Additionally, running one more BGP session makes their network appear larger, so they benefit as well.
I currently have a BGP session with one of my ISPs on the 15 euro plan (1 Gbps), and another local ISP is giving me 10 Gbps for free because I'm a test user. Both of these ISPs support BGP, and it's amazing that I can have BGP sessions as a home user. However, it can be challenging to speak with regular sales reps who may not be familiar with BGP. It takes persistence and some effort to get to the network administrator.
It's frustrating that some ISPs make it artificially difficult for home users to obtain a BGP session, even when they're not asking for SLAs or dedicated connections. Technically, you can even run a BGP session over dial-up. It seems like these restrictions are in place to squeeze more money from customers.
It's no secret that we've run out of IPv4 addresses, and I happen to have a /24 subnet just for myself. Even with all my usage, I barely use half of it. There's no rule that says ISPs should only accept /24 or shorter prefixes, and this is contributing to the depletion of IPv4 addresses. I would happily announce a /25 and return the other /25 to the world, but I must have a /24 to do it. This practice of requiring /24 subnets may have made sense in the past to preserve router memory, but with today's technology, it's nonsensical and only serves to artificially deplete available IPv4 addresses.
I have a super rudimentary understanding of what BGP is (it allows you to advertise addresses something something), but I don't really understand why it's useful and thus I'm finding TFA to be a little inaccessible. Could you provide some context about what you're using BGP for (what is your use case; what does it enable for you; etc) such that it might fill in some gaps for me?
BGP is what "the Internet" uses to say what IP addresses can be reached over the various network links that connect different providers. If you had links to two different providers, you would talk BGP to them and say "You can reach this set of IP addresses via me." They then propagate that announcement on to other providers that they connect to.
This is done via ASNs (Autonomous Systems Numbers), like 32150 (my old ASN). Each hop along the way gets their ASN tacked on. It's like a flood-fill to all the BGP routers in the world.
When you "traceroute", you see IP hops along the way, but the ISPs routers choose links based on this list of autonomous paths (among other things). Basically, a AS-level traceroute is built into the BGP routing information. At it's most basic, a path is selected based on the shortest AS path (traceroute, but at the organization level rather than IP level); the fewest number of networks the traffic has to traverse.
If a link goes down (either because of a network failing, or because of administrative reasons), traffic will eventually switch over to another way of reaching the destination. You can also do things like pad the AS path with multiple copies of your AS to cause traffic to switch to another link more frequently, as a kind of primitive load balancing.
>"If you had links to two different providers, you would talk BGP to them and say "You can reach this set of IP addresses via me."
There is really need to do this is many cases though. If all you are looking for is connectivity redundancy you can just take the default routes from two ISPs and configure floating static routes on your router and not worry about BGP at all.
>"This is done via ASNs (Autonomous Systems Numbers), like 32150 (my old ASN). Each hop along the way gets their ASN tacked on. It's like a flood-fill to all the BGP routers in the world."
Each hop does not get its own ASN tacked on. This only happens when crossing an AS boundary i.e at eBGP speaker. There are generally many hops inside an AS.
> There is really need to do this is many cases though. If all you are looking for is connectivity redundancy you can just take the default routes from two ISPs and configure floating static routes on your router and not worry about BGP at all.
That only gives you outgoing redundancy. You can talk to the world with one of the links down, but the world can't necessarily talk to your IPs with the link down.
BGP lets you advertise IPs to the world and what links they're reachable on. You can announce the same IPs on multiple links to multiple ISPs. This is what makes the internet "route around damage".
Generally ISPs will only let you announce the same prefixes to multiple upstream ISP if you have PI address space. If you have PA address space you are probably out of luck. An ISP won't accept your prefixes if they are PA space from another ISP. At any rate you can can still use floating static routes and use a GSLB provider like Dyn to get your incoming redundancy and still not need to run BGP. In fact if you have PA address space this is your only option.
Correct if we are talking about IP hops (router 1 to router 2 to router 3), the AS does get tacked on (possibly multiple times) for each AS hop (Level-3 to Cogent to Zayo). Thanks for clarifying that.
The internet is a decentralized collection of autonomous systems (AS) and the primary benefit of this setup isn’t exactly realized afaict, which is multiple paths via different carriers. Your BGP AS is advertised via N paths, so your IP networks are thus reachable via multiple paths. Systems will take the shortest/best path adding to redundancy and throughput. Compared that to “business” service without routing options, when your “dedicated” connection drops, your IP addresses are no longer reachable from the greater internet.
>"The internet is a decentralized collection of autonomous systems (AS) and the primary benefit of this setup isn’t exactly realized afaict, which is multiple paths via different carriers."
The primary benefit if fully-realized and is not about "is multiple paths via different carriers."
An AS is an administrative entity. Route prefixes belong to an AS. All routes with an AS have a common routing policy that is managed by a single entity. I can look up a prefix to find it's AS and then I can look up that AS's routing policies. Here is an example for Tier 1 ISP Spring:
I could have single ISP that I buy transit from and I might have multiple links with them. If I want to understand how to take advantage of that for example with traffic engineering then I would look the policies for ISPs AS. I would then use those BGP community attributes on my BGP links to them in order to accomplish my own routing goals.
> I have a super rudimentary understanding of what BGP is (it allows you to advertise addresses something something)
The TL;DR:
Yes, BGP allows you to advertise addresses.
Ergo, it is allows you to build resilience, i.e. you can connect to multiple independent ISPs and advertise the same addresses. Hence traffic will always be able to reach you even if one ISP goes FUBAR.
By the same token, it makes pretty much zero sense to have BGP on a home connection (or, in technical speak, a "single-homed connection"). Since really BGP is a zero-sum game on single-homed connections (no real benefit, much added complexity).
> It's frustrating that some ISPs make it artificially difficult for home users to obtain a BGP session
Don't worry, it can be frustrating for businesses too !
Cogent famously nickle-and-dime their customers and consider BGP to be a chargeable extra, even on their IP Transit product which is somewhere where you would expect BGP to be a given.
Of course whether you should do business with Cogent is another matter, especially given their spammy cold-calling tactics, seemingly once you're on their list you can never get off it.
You're lucky. There are basically no small ISPs around here anymore. It's the cable company (DOCSIS) or telco (fiber, or DSL if you're unlucky.) Without paying for an enterprise grade connection, it would be impossible to get a BGP session through either.
I also have a /24 for myself. I've had it since the mid-90's, from the InterNIC days, predating ARIN. I've gone years without using any of the addresses. Currently it's routed to my home lab. I'm doing BGP through a couple VPSes, then tunneling it back.
> This practice of requiring /24 subnets may have made sense in the past to preserve router memory, but with today's technology, it's nonsensical and only serves to artificially deplete available IPv4 addresses.
You should be surprised about how much of the internet still runs on gear from late 1990s
There seems to be a lot of misconceptions in your post.
>"Additionally, running one more BGP session makes their network appear larger, so they benefit as well."
This doesn't make any sense. The number of "BPG sessions" is not a metric any ISP thinks about. A BPG session is just TCP connection. The number of prefixes an ISP is advertising might be something they care about however but for smaller ISPs that doesn't really matter as they are just customers of the Tier 1 ISPs and that just requires paying the Tier 1 ISP. It has nothing to do with the number of sessions or prefixes. I could have a single /22 prefix for my company and the Tier 1 ISP will still sell me the same service.
>"There's no rule that says ISPs should only accept /24 or shorter prefixes, and this is contributing to the depletion of IPv4 addresses. I would happily announce a /25 and return the other /25 to the world, but I must have a /24 to do it."
Well yes there is. Many ISPs will generally not accept routes smaller than a /24. They configure filter-lists that prevent those routes from being accepted. The reason for this is to reduce the size of the global routing table. Each prefix has to be stored in TCAM(memory) on a router. There have been many incidents where older routers couldn't handle the number of routes in the global routing table and caused them outages. See:
Further if you have a /24 you can certainly advertise 2 /25s to the same upstream ISP if you have more than one transit connection with them. In fact this is exactly how do traffic engineering.
>"It's frustrating that some ISPs make it artificially difficult for home users to obtain a BGP session, even when they're not asking for SLAs or dedicated connections."
This because home users absolutely don't need to run BGP! You just take a default route from your ISP and you're done with it. There is not point in running BGP if you are single-homed! You can't do any traffic engineering on it or influence the way upstream ISP route traffic to you. From the hobbyist point view it would be extremely boring. What can you actually do with it? View the global routing table? You can do this without ever running BGP by using a public looking glass server[1]. There is no incentive for ISP whose business is selling internet connectivity to cultivate a hobbyist community.
I rent a single ~600 meter fiber that connects my home to the network infrastructure of the company I run. The rent is $75/mo or so. I have passive CWDM muxes on the line to run two 10G connections to two separate edge routers.
My home router is a MikroTik CRS309—it’s about $250, fanless, has 8 SFP+ ports and advertises my home network blocks over BGP to the two routers (for HA).
The setup works great. The best part is how it’s small and fanless and fits inside the very small wall box the fiber terminates in.
> My home router is a MikroTik CRS309 ... advertises my home network blocks over BGP
How do you fit two full BGP tables into 512MB of RAM? I've looked into MikroTik boxes before and maybe they're doing something I'm not understanding. On the routers I have manage, two IPv4 feeds take up about 1.1GB and three IPv6 adds another 450MB.
If by 'full BGP tables' you mean the entire internet routing table, you don't need that to advertise a network on the internet. You can receive just a default route if you wish and still be able to advertise to any carriers you have a connection with. You won't lose out on any functionality in dual homing or anything like that.
> You can receive just a default route … You won't lose out on any functionality in dual homing or anything like that.
Except you do lose out on best path routing / any other outbound TE, and you’re now restricted to rudimentary load balancing methodologies / manual prefix-specific hackery.
If you aren't multi-homed that doesn't matter at all though.
For a home network I'm guessing you are pretty unlikely to do the multi-homing from the house, more likely you will either just have a single upstream or if you are connecting it into your own collocated infrastructure you will do iBGP and let your actual edge BGP routers handle the multi-homed upstreams and sync the full route table etc.
I was tempted to do this once before when I was running my own hosting company but it was prohibitive cost wise to get the circuit I wanted. :(
And that's why SD-WAN (software defined wide area network) technology now exists. In a lot of ways it is even better than BGP in that you direct traffic based on performance measurements - either actively through SLA tests or passively watching traffic flows and measuring latency. Using BGP routing based on hop counts and AS paths is akin to following road signs rather than getting live routing that knows traffic for instance from Google Maps
SD-WAN technology also creates a number of new ways for unthinking people to shoot themselves in the foot.
Noction is a wonderful example of this. It has sane defaults, but insane customers, who think that it’s a good idea for them to originate more specifics for other outside networks, because they’ll never leak them outside of their own AS (narrator: of course, the prefixes leaked).
I think the most notable example of this recently was Verizon (the insane customer) using Noction (the SD-WAN technology) and doing exactly that, causing mass traffic disruption as they announced more specifics for other peoples prefixes, drawing all that traffic to their own network instead.
> Except you do lose out on best path routing / any other outbound TE, and you’re now restricted to rudimentary load balancing methodologies / manual prefix-specific hackery.
After 3-4 hops you're probably hitting a Tier 1 network, after which point you can basically think of the Internet like cloud icon you see in many diagrams, because your route choices are no longer really determining reachability, rather the choices of other people/companies are:
If you're talking about reachability of a network on another continent or the other side of the planet, your local decisions aren't going to much to determine the path.
The main thing to have locally for routing decisions is the ASNs/networks of the other customers of your ISPs: if Service A is also a customer of ISP #1, you want to send traffic for them through that service instead of ISP #2.
The other nice thing to have is the reachability to the closest IXP, as quite often many CDNs have connections to those.
Beyond knowing IXP reachability and other-customers reachability, I don't think there are many other advantages for a smaller entity on the Internet, so a full Internet-wide BGP is not needed.
> I’m just getting the default route from both upstream routers.
If they offer the option, instead of a "default-only" feed from each ISP, you may wish to see if they have "default-plus-our-customers" feed: if Service A is also a customer of ISP #1, then why bother sending the packets to ISP #2 in the first place?
In general, at some point you'll hit a Tier 1 network, after which it won't matter, but until that point getting the connectivity to other customers of the ISPs could be useful. The other reachability destination to pay attention to would be of IXPs, where CDNs often connect to.
Both of the upstream routers are mine, there’s no difference to the routes they have available. (They’re connected both to Tier 1 transit as well as the major local IXP)
I have their RB5009 with 1Gb RAM and I got 4 full feeds and have around 200Mb free. I do need to tweak stuff to get IPv6 working (again... only been sorting this out in the last couple weeks)
The only thing SLAs do is tell you how much you will be compensated for downtime. If a fibre cut happens, you can end up with a 16 hour outage regardless of what the SLA guarantees. It all depends on the nature, location and extent of the damage. A couple of summers ago one of my wavelengths was down for more than 16 hours. 3 x 432 count cables had to be replaced through 3 manholes because a fibre seeking backhoe ignored the locates. Splicing high count cables takes a lot of time (even with ribbon fibre).
Huh, so per the last paragraph, they're paying a ton to have an SLA, all the ISP has to provide is basic connectivity because BGP and stuff is being done by the customer themselves, and yet they have more outages in a few months than I have on a consumer connection in years?
The whole thing has a lot of hack value, it's cool and also worth something to be in control of your own networking, even if it's more expensive. Like buying apple: overpriced for the specs you're getting, but you're assured it'll be good quality (that's the idea anyway) and it looks cool (to most). Except... apparently it's still got issues, regularly? Now I'm really wondering what the point was, at least with hindsight
The post said that consumer connection does not let you do BGP peering with the ISP.
There are real value in doing so in addition to the hack value, by doing so you can steer the Internet traffic to whatever IP blocks you owned to your house, dynamically. For example OP mentioned "add some resilience" as motivation, i.e. anytime their services running in the DC failed he can reroute the traffic to ... their house.
You're absolutely feeling the same problems on consumer connections. With DIA, you're the only subscriber on that circuit, so if upstream is having a problem then your neighboring consumers will have them too.
Lower bandwidth dedicated circuits can often "feel" faster than a higher bandwidth consumer connection.
The main issue he wanted to avoid (trees falling on lines) still seems to be there.
Looking at past outages, I feel like given a certain budget, you would get much better uptime with two or more diverse (i.e. not "two fibers going exactly the same path") residential providers than with one commercial one, and for anything but the most critical projects, that's the way to go IMO.
Most importantly though, anything really really important that is so important to use custom fibers should be designed so it also works over the public Internet. Unless degraded service is worse than no service, it doesn't matter that the Internet is theoretically not reliable enough/doesn't provide enough guarantees - at least keep it as a backup so that when a backhoe takes out your custom fiber, you can switch over to the VPN instead of shutting down air traffic at one of Europe's largest airports.
Seems like another major reason for the decision here were SLO/SLAs. Those seem rather meaningless. The residential ISP will not try to max out their 95% SLO - an ISP that's down 1.5 days every month or leaves you offline for 18 days wouldn't be very popular. The commercial SLOs, on the other hand, sound great but they're not magic. If the issue can't be fixed in that time, they will be violated, and you'll typically get a credit for a few months of service - money that you wouldn't have spent in the first place if you just went with residential.
In the end, he still has countless single points of failure, e.g. the above-ground fiber line waiting for a tree, or a non-redundant power supply that he can't easily replace himself. Sure, he has an SLO, but with a dual uplink residential (which is cheaper), a CPE ("modem"/router) failure wouldn't be an outage in a first place.
They state at the beginning of the article that the main service location is a data center in NYC. This is a redundant service location. An outage would require failure of connectivity to his home and to the NYC data center.
I'm curious to know details like what sort of house or dwelling this person lives in; anyplace with enough capacity for all that (4U rack space, power and cooling, etc.) seems spacious.
The article says it's right off a major street. So is it urban? Suburban? Single-family home? Inquiring minds want to know.
Just so you know, 4U of rack space is a whopping 7 inches of height. One rack holds many Us, with most "full size" racks clocking in at 42U.
Open-air racks are typically less than 2 feet across and 3-4 feet deep (example [1]), hardly a significant stretch unless you live in a very small space.
4u of space is less volume than many PC cases. I’ve got a 22u “half rack” in my apartment (half for testing gear for work, half because I like to tinker). Bigger concern would be noise. 1u network gear tends to have high pitched 20mm fans which can drive you nuts if you don’t have it in a separate room. Noctua makes a great kit I use in my own gear but Verizon probably wouldn’t be very happy with you if you tried to do that :).
I went the route of “find a dirt cheap gigabit colo” ($150/m for quarter rack with gigabit and bgp peering) and just tunnel the IP space back home. Upside is you don’t have to pay to have the circuit put to where you currently live and you don’t have to worry about space/noise. Downside is if you want to use it locally at home you get extra latency due to the tunnel.
If your use case is just to use the unique space locally then you don't need BGP at all but the second you want to use it elsewhere you incur the latency of going through the tunnel.
Currently I'm doing BGP over a wireguard VPN back to a couple of VPS providers. Actually, I only run OSPF to my home, the BGP part only happens on the VPS side. The tunnel does add about ~8 ms of latency worst case.
I have a couple of 2U racks vertically mounted behind a door.
Takes up zero usable space!
It took me way too long to realize this option - many builds using miniture casings with cooling issues later, when buying large 2U cases with 120mm fan walls is the ticket.
It's more likely to reflect a hierarchy in the network (e.g. monomode span dropping down to multimode for the demarc). The telco just colocated their box on the subscriber's premises.
I don't buy it. You seem to be contending that the second NID is just a fancy transciever between fibers. But both boxes almost certainly have QSFP or some such, could input or output whatever fiber the job calls for. Quibbling about fiber types as an excuse for having two boxes seems absurd.
Your assertion also seems contested by the words in this post:
> Dispatch 5: NID #2 install. This is where I began to see the distinction between Verizon Telecom (VzT) and Verizon Business (VzB). While the layer 1 infrastructure had been installed and tested, there still remained the IP layer, which required another dispatch for another technician to install a dedicated NID for Verizon Business -- a Ciena 3903, with 1310nm single mode optics.
It does mention a different fiber connection, but to me, this is describing each layer of the stack having both it's own organizational unit & it's own physical box that it manages.
Two different companies, one provides the last mile, one provides the customer service layered on top.
It adds a lot of flexibility, for instance in this case the person was on-net for Verizon, but what if they were not? Then it wouldn't change at all, just the first NID would be a different provider than Verizon.
First "NID" is a Telco term of art, it's a contractual concern more than anything else.
The first device terminates the Fiber. It has an SPF transceiver in at for the wavelength the customer is getting the service on which is 1310nm. This is pure layer 1. A technician could/would use this "NID" to get a light reading during provisioning and/or troubleshooting. This is the end of the "local loop."
The second device is the one that provides Carrier Ethernet. Carrier Ethernet is like regular Ethernet but with some extensions that allow the provider "OAM"(operation, administration and management) capabilities. This is a layer 2 device.
I'm curious about this too.. it seems like it might be some intricacy because of Verizon business and Verizon telecom being two different entities, but I am curious if there is a technical reason for it too.
The guys on the ground and their dispatch are CWA unionized employees. The businessy guys are not.
Verizon is like a state or federal government agency. Massive complex bureaucracy that nobody really understands. As one of the ultimate legacy businesses they have all sorts of crazy settlements, agreements, tariffs, etc and segment their businesses around them.
>(as of this writing, VZ was pricing 50Mbps commited @ $455/mo; 100Mbps @ $661/mo; 1Gbps @ $999/mo; 5Gbps @ $2,099/mo; and 10Gbps @ $3,099/mo).
Ten gigabit is not yet a common residential internet service in North America. If that's what you want, this would be the only way to get it.
I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc. https://www.seattleix.net/join As expensive and pointless as buying a yacht, but at least getting it online would require a lot of tedious work!
(Like OP, I also jumped through the hoops for a Comcast fiber drop years ago (but to a business location) for the same eyewatering price but dramatically fewer megabits per second. The only upside is that the pair traveled directly to a CO in the same office park, so we would routinely see 1-2ms pings to anycast IPs like 4.2.2.2, 8.8.8.8, etc)
> I've occasionally daydreamed about buying a condominium near one of the Seattle Internet Exchange's PoPs, buying a 400 gigabit port, paying to run a fiber pair, arranging for transit, etc etc
For what it's worth, this isn't the same as what the person who wrote the post did. They bought a DIA circuit to an ISP.
What you're seeking is a point-to-point fiber link or MPLS to a panel in KOMO Plaza or the Westin building. You don't need to be near either of those locations to do it and CenturyLink (Lumen, these days) will sell you the circuit but you're not going to like what it costs. We have three of these circuits where I work and they are juuuuust a bit higher than what the article's author says they're paying for the DIA.
(This, along with the eyewatering cross-connect fees that the operators of those two facilities charge, is why getting a SIX port is so expensive on retail colo. You're either paying for the privilege of running some strands of glass or you're paying to use someone's extension switch and the connection they're paying for to get back to the SIX core.)
It really depends on the metro area. Here in Chicago I can get a dark pair about 3 houses down the block from me (never quoted out how much it'd be to extend to my address) going back to 350 Cermak. This ran about $600-1200/mo or so depending on contract terms and who you know. If you have other gear in the facility (e.g. transit and routers with a spare port) you're looking at another $150-350/mo (again, depending on your contract) x-connect to get into your panel in your cage.
Given I have equipment already in the datacenter, it's pretty tempting to pull the trigger on a 100G link. I just have no idea what I'd do with it.
Completely agreed. Seattle and Portland seem to be high on the list of "places telcos think they can take complete advantage of." For what my employer pays for two (locked) racks of colocation here, we could get an entire small suite somewhere like LA or Chicago.
Wireline costs are similarly sky high, I think because CenturyLink has all of the cabling and no one has bothered to dig up the streets to run any more. When we connected a building in Eastlake to our old colo in Tukwila, CenturyLink was the only one who would quote us.
> The only upside is that the pair traveled directly to a CO in the same office park, so we would routinely see 1-2ms pings to anycast IPs like 4.2.2.2, 8.8.8.8, etc
Those are the pings I see on my residential connection, though?
> If that's what you want, this would be the only way to get it.
10Gbps PON is available from AT&T Fiber in some markets, and lower tiers (5Gbps) are coming online from Google Fiber in other markets. In SoCal, I've seen billboards for 10Gbps AT&T Fiber, for example. Obviously you don't get an SLA with a residential connection, though, and I highly doubt that many customers operating at full capacity will be problem-free.
This is how any larger business (e.g. a hospital, university, anyone doing any serious networking) gets their connectivity these days. Basically the modern equivalent of a T-1.
You get proper service: no random dropouts, and a clueful human on the phone and you could complain about QoS and they'd not laugh in your face.
The different thing here is that the service was provided to a residential building, but that's not entirely unheard of. I've done it myself for example.
If I had more money than sense, I’d do the same thing.
If I was filthy rich (which I’m not and probably never will be), I’d build a data centre in my mansion and buy a brand new IBM mainframe and stick it in there. Because I can. For the same reason other rich people build some huge garage with dozens of cars in it.
Which means you'd need a IBM mainframe, since IBM mainframes require enterprise storage arrays which come with special software for emulating the old mainframe hard disks (ECKD), and if you are going to buy one of those, you'd need a mainframe to drive it.
Yes, they do use Fibre Channel (FC), but the protocol they run on top of FC is very different from the usual FC protocols. FC has four layers, from FC-0 (the physical layer) through to FC-4 (the application layer). At level FC-4, Linux/Unix/Windows/etc servers mostly speak "Fibre Channel Protocol for SCSI" aka SCSI-FCP, which transmits SCSI commands over FC. However, while IBM mainframes do use SCSI-over-FC, they also use something else called "FICON", which instead of transmitting SCSI commands, uses the same command set as IBM's legacy non-SCSI mainframe hard drives (such as the IBM 3390, which was IBM's last non-SAN mainframe hard disk line from the late 80s / early 90s).
Mainstream platforms have what IBM calls FBA (Fixed Block Array) disks, in which the disk is an array of sectors which all have the same size. While the IBM mainframe hardware supports that, most IBM mainframe operating systems don't support those, however; Linux and VSEn are the exceptions, and z/VM is a partial exception.
As well as FBA, IBM has ECKD (Extended Count Key Data) disks, like the IBM 3390 was. (ECKD was preceded by CKD, which is conceptually the same – the same sector format, etc – but had a less efficient command set.) With ECKD, sectors can have variable sizes – each disk track can have different sized sectors, you can even mix different sector sizes within the same disk track. Also, sectors optionally have keys, and the disk has commands to search for sectors based on their keys. Originally, these variable sized sectors actually physically existed on disk; nowadays, the SAN only has standard hard disks (or SSDs), but ECKD disks are emulated in the SAN software on top of them. z/OS, the most popular IBM mainframe OS, only supports ECKD disks, not FBA ones. Linux and VSEn support both ECKD and FBA. For ECKD disks, you can't use the standard SCSI command set, you have to use IBM's DASD command set.
Many high-end enterprise arrays do support ECKD and FICON, but you usually have to pay significant extra license fees to enable the software that supports that.
Hearing people talk about IBM mainframes makes me realize what I sound like talking about my job to my mom. IBM could have 90% of the home computer and server market if they didn't charge so much for their stuff. They always had the best technology.
> IBM could have 90% of the home computer and server market if they didn't charge so much for their stuff. They always had the best technology.
In this particular case, there is no contemporary technological advantage of ECKD over standard hard disks/SSDs - it is simply a matter of legacy backward compatibility, lock-in, and an attempt to financially sustain the mainframe storage ecosystem. The idea of variable sector sizes may have had some real advantages when it was invented back in the 1960s, but nowadays it is just adding unnecessary complexity for no real benefit.
Even the idea of having the hard disk do key searches is something IBM has been moving away from, because doing them in software on the CPU turned out to be much faster in practice. It is still required because the primary traditional mainframe filesystem (VTOC) is based on it, but in areas where performance is critical (such as databases) IBM now positions it as a legacy technology.
Maybe there's still some value in moving key searches into the storage layer, but if there is, it would have to be something much more advanced than the rather simplistic 1960s implementation that ECKD storage provides. Oracle came up with a conceptually similar idea much more recently, Oracle Exadata (released 2008), in which some aspects of database query execution are delegated to the storage–especially table scans. ECKD only supports very basic comparison operations on the key field; Exadata, from what I understand (I used to work for Oracle, but not in this area) can evaluate complex predicates over the whole database row.
I worked for a state government that was in the process of migrating hundreds of Oracle DBs from VMWare VMs to Exadata servers. I think combined they had over 20TBs of RAM. I asked one of the DBAs about how much better performance was and he said he typically saw at least a 10x increase in QPS and reduction in latency.
Samsung has a SmartSSD with a FPGA on it to offload compute to the drives.
It sounds like they wanted 2 separate network paths but they still end up with only 1 where they can use BGP. Maybe they can still use their residential connection as failover, with a VPN to a VPS where they can use BGP?
They all likey traverse the same physical path and would all be taken out by the same gravel truck hitting a pole. The two connection thing was I think more about defense against random retail connectivity flaps (someone decides to upgrade a router's firmware and just takes the town offline, meanwhile nobody answers support calls, that sort of thing).
Yeah true multihoming is expensive and involves things like hiring people to actually inspect the physical lines and Make sure they don’t run down the same street at some point (and be subject to the same backhoe of fiber death)
In the event that there are issues at the primary location, internet routing protocols will re-route the relevant network prefixes to the secondary location.
The public /24 that was previously routed to their NYC datacenter now goes to that Connecticut house.
General reminder that an SLA is a contractual agreement, and not a guarantee of availability. It literally describes the renumeration due when the service doesn't meet the promised availability.
I absolutely understand- but is still amazingly cheap. The install was not simple. Despite being oversubscribed I get good speeds both ways. Most devices on my network can only handle 1 gig though my vmware/ synology stuff is 10g. I have business service from att at a different location - Even ordering process was kind of annoying- you get a rep and going around on price / term etc takes a bit vs clicking buy now.
At home I pay for 5 but just need 2. I have failover to comcast. Works well . And yes I do get reported outages on the att side but because failover works I’ve not chased things down.
Oversubscribed means different things to different groups of people. (I'm replying to people in general, and not to the commenter above me)
To an ISP, it means that a customer could exeperience a shortage of service, but that a group of customers would have to simultaneously utilize the infrastructure. ISPs try to keep a ratio of customers to service where this isn't likely to happen, or for discount providers, that this happens only during peak times. Ultimately it is a balance of customer attrition.
To a customer, it sounds like they are being deprived of what they are paying for. (Residental customers agree to a best-effort service typically - the opposite of agreeing to a specific service level). Even with ample service capacity, it is often the first factor considered when any component in the aggregate network is failing or underperforming.
I sure do not miss being a residential ISP. But I carry with me just enough sympathy for service providers that I am "always a blast at a party".
The problem I have is the lack of transparency, obtuse terms and the huge amount of funds taken in by the bigger operations that instead of being transformed into moar infra, get turned into exec bonuses/lobbying to not have to build infra/marketing to convince people the infra that was built is the best that's possible, yes sirree Bob.
When all you have is a string, and tin cans, you do the best you can. When your execs are taking big Federal bux, and setting them on fire instead of increasing overall throughput...
The problem I have is the lack of transparency, obtuse terms and the huge amount of funds taken in by the bigger operations that instead of being transformed into moar infra, get turned into exec bonuses/lobbying to not have to build infra/marketing to convince people the infra that was built is the best that's possible, yes sirree Bob.
When all you have is a string, and tin cans, you do the best you can. When your execs are taking big Federal bux, and setting them on fire instead of increasing overall throughput...
ATT home fiber is typically 32 ONT sharing 1 OLT, doesn't come with the SLA, and doesn't let you assign your own IP space to the connection (along with some other knobs BGP lets you turn).
Of all of that the bulk of the price increase is them running the dedicated fiber so you can go from 32:1 to 1:1. It costs a lot to not have the same physical setup as the rest of your neighbors.
Bear in mind these are entirely different classes of products... it's not just about bandwidth.
The monitoring and SLAs alone are probably the most significant differentiators, with these kinds of services you would usually expect an engineer on site within 4 hours to fix problems, and service credits for anything breaching SLA.
I once accidentally snagged a single mode fibre pigtail while moving stuff in a rack. Everything back up after the provider engineers had re-spliced a new pigtail on in less than two hours.
And that's before you consider the dynamic routing side of things.
(And the 3 x /24's that AS54316 have are also not cheap these days, that would cost something like $30-40k these days).
I currently have a BGP session with one of my ISPs on the 15 euro plan (1 Gbps), and another local ISP is giving me 10 Gbps for free because I'm a test user. Both of these ISPs support BGP, and it's amazing that I can have BGP sessions as a home user. However, it can be challenging to speak with regular sales reps who may not be familiar with BGP. It takes persistence and some effort to get to the network administrator.
It's frustrating that some ISPs make it artificially difficult for home users to obtain a BGP session, even when they're not asking for SLAs or dedicated connections. Technically, you can even run a BGP session over dial-up. It seems like these restrictions are in place to squeeze more money from customers.
It's no secret that we've run out of IPv4 addresses, and I happen to have a /24 subnet just for myself. Even with all my usage, I barely use half of it. There's no rule that says ISPs should only accept /24 or shorter prefixes, and this is contributing to the depletion of IPv4 addresses. I would happily announce a /25 and return the other /25 to the world, but I must have a /24 to do it. This practice of requiring /24 subnets may have made sense in the past to preserve router memory, but with today's technology, it's nonsensical and only serves to artificially deplete available IPv4 addresses.