Generally internet goes into the global routing table and is used for VPNv4 and VPNv6 peering, as well as label distribution.
So this means you have to run BGP everywhere then, yes? Isn't that quite a downside?
I don't know any ISP that doesn't use BGP. BGP runs the internet. Why would that be a downside? MPLS is basically BGP and label switching.
We use IS-IS in most of our core. But mostly BGP everywhere else.
The key phrase here isn't run BGP as much as it is run BGP everywhere. If you put the internet routing table in a VRF you could have a BGP-free core and only run it on the edge routers.
You can still have BGP free core using OSPF or IS-IS in the global table. BGP on the PEs to exchange VRF routes and IGP to distribute labels and global routes for BGP peering.
That makes sense I suppose. Is it a very common setup or do carriers still tend to run BGP on their backbone as well?
BGP-free core is extremely common. BGP is very slow to converge, so ECMP with an IGP and BFD make for quick failover while maintaining BGP peering across the network.
I see. Thank you.
Its not common to run on the backbone. Usually, you'll run it on your customer agg and edge routers (your LERs). You need to make sure that the packets know where to go when they leave the LSP at the penultimate hop.
You want to use MPLS LSPs because you want to be able to traffic engineer in the core. BGP becomes redundant, if you have an IGP there.
I think most carriers use IS-IS.
This is correct
IS-IS for the backbone.
MPLS can provide transit for traffic in the global routing table too. So effectively most carriers do what you say, just the “global” table is used not a specific VRF.
On some hardware platforms there are things you can do in the global table you can’t do in a VRF, which is the reason why.
No. Carriers have a public internet and private internet. If you have an MPLS circuit the VRF starts at the provider edge and goes through the carrier's private network back to their other edge and then to your other location where the VRF stops. Loosely speaking, the VRF contains BGP specific to you in this case. MPLS is the Carrier's "routing" protocol on their private network.
If you don't have an MPLS connection, then the traffic just stays on the public internet, all BGP.
Does that help?
I mean... I don't know. Are you saying you believe carriers put MPLS on a completely separate network than their internet infrastructure? Because that doesn't sound correct to me.
We basically do exactly that.
Top five globally.
If you have the money you can do this for sure.
The upside is that any DDoS and other evilness that exists on the internet is physically segmented away from the enterprise/office networks being used by your WAN-service.
Another upside is if/when for example an internet related router goes poff this online affects your internet traffic and not your WAN-service (and the other way around, if something goes poff with your WAN-service either its a hardware malfunction, cablecut or just bad config then your internet-service is most likely unaffected (except for the cablecut scenario).
A possible upside can also be if you got physically separated networks you can then also upgrade the gear for your WAN-service without affecting the stuff used for internet. Today its not uncommon that other services than just pushing ip-packets between sites can be provided - for example VPLS that can push ethernet-frames. Or other specific requests from the customers such as being able to push jumboframes (9k) site to site where with internet you can just say "nope, its 1500 bytes MTU and thats it".
The downside is not only double the cost for gear, darkfiber and wavelengths (and power, cooling and space renting at the site) between sites but also double the amount of management/maintenance since you now have twice the amount of gear to take care of.
No. But also, kind of, yeah. It's a mix of the two. For MPLS, the customers connect directly to the MPLS routers. For public internet, the customers connect to edge routers, which then connect to MPLS routers. The MPLS connections are then routed back to the private side of the core, and the public connections are routed to the public side of the core.
So the access layer is often separate, then sometimes meet at a shared aggregation layer, then to the appropriate public/private core routers.
It depends on many factors, some technical, some business reasons. Some SPs float internet on L3VPN, some carriers use inet.0 (global routing table).
In our case, we operate two completely separate layers of networks. IP transit (single AS) division runs internet on global routing table, but our regional area networks (different ASNs) float internet on L3VPN.
Bottom line, there is no technical justification on which method is superior. You can just as easily protect your core from DDoS and outside attacks while running internet on inet.0 instead of on L3VPN (infrastructure ACLs, proper numbering schemes, etc). It really comes down to the service models you're offering and business case.
You doesnt have to but it has become some sort of "best practice" on how to run an ISP.
The downside is the complexity and the cost (the gear must support MPLS) but the upside is that you can separate internet (or whatever vrf) from any other networks you maintain - if not to at least separate MGMT from PROD where PROD in this case is INTERNET.
And once you have MPLS up and running its a small task to also start to sell WAN-services and not just internet-connections to your customers.
Once you go the WAN-path you will for sure end up with customers running the same ipranges (if not so at least RFC1918) and here is vrf very handy to be able to deal with this.
The other option would be that you always tunnel the customers traffic end to end (think of this as a traditional vpn), this way whatever the customer choose to do or run wont affect your core or distribution network because the customer traffic is encapsulated. I mean the customer can still choose to run vpn on their own to protect the traffic from eavesdropping but then wont have to do it if they just want to move packets from site A to B (and doesnt care about eavesdropping along the road).
I would imagine that the reason any carrier chooses to do this is because the lack of supernets for routing purposes. Currently, we don't have that problem, yet. But I don't think it will be long before there are serious issues with anyone not natively supporting V6 on their buildouts. We run a 100% dual-stack so not really an issue for V6, but IPv4, won't last forever.
Because you generally can't resolve as much in VRF as you can in the master routing table.
It scales better to run your internet in a VRF instance, rather than running it in your GRT
Erythang is MPLS(or SR). Internet is just another VRF. The global table only contains the /32 loopbacks of the PE's
Routers, switches and firewalls. Network blogs, news and network management articles. Cisco, Juniper, Brocade and more all welcome.