Press J to jump to the feed. Press question mark to learn the rest of the keyboard shortcuts
0

Bgp edge devices recommendation

To date we've used 4x Juniper mx5 as our bgp edge. They are the right price point, have all the features we need, and day to day performance is fantastic.

However, we've hit 3 big issues with them:

  1. An unexpected major bgp change from the outside (a drop or similar) takes a long enough time to propagate through the krt queue that a blackhole or loop outage is inevitable for, if its a full table rebuild for example, close to 20 minutes. This is a known design flaw with Juniper

  2. Their 10g interfaces require a very expensive license to use and we are approaching >1gbps 95th%.

  3. Their support is awful for replacements, and we've had a total of 3 mx5s die on us over the course of 5 years.

So I'm looking for alternatives. The price of an mx5 (~10k Inc 5yr warranty) is about what we'd like to spend. Any suggestions appreciated!

17 comments
50% Upvoted
What are your thoughts? Log in or Sign uplog insign up
level 1
CCNP2 points · 2 months ago

Based on your very meagre amount of information. Juniper mx104 Cisco asr1k Huawei ne20

Maybe Arista might have something that fills your needs. Not use their kit though, so unsure about a full bgp feed on those.

If you want cheap and cheerful there is always mikrotik and their ilk, or a white box running cumulus or similar.

level 2

While I haven't used any personally, my understanding is that the MX104 also has a woefully inadequate control plane. Perhaps consider the new MX204.

level 1
2 points · 2 months ago

ASR1001X

level 1

If your MX5 is on JunOS 15.1F6 or newer, this helps with the slow propagation to FIB: https://www.juniper.net/documentation/en_US/junos/topics/reference/configuration-statement/delay-route-advertisements-edit-protocols-group-family-unicast.html

I’m using the MX80-48T because the 4-port 10G does not require additional license, so works well for us.

level 2
Original Poster2 points · 2 months ago

OMG, game changer!

level 1

So what are your requirements?

Number of 10G holes? Number of 1G holes? Features? Etc.

With your starting requirements of "something better than an MX5 for the same price" (also assuming usd) probably not a bunch of good options.

level 2
It places the packet on the wire or else it gets the hose again.4 points · 2 months ago · edited 2 months ago

Number of 10G holes? Number of 1G holes?

I got a laugh out of that. It sounds like something my girlfriend would say (minus the 10G / 1G).

EDIT: For the downvoters, this wasn't meant negatively. She just has her own way of phrasing technical things that I get a kick out of.

level 1
J1 point · 2 months ago · edited 2 months ago

Look at MX150. It's basically a commodity x86 box running vMX, but is supported and certified by Juniper for 10+Gbps throughput. They're fairly inexpensive (around $10k for the full routing featureset), and since it's a fairly beefy x86 box with no slow PFE CPUs involved, it has a fast control plane and fast flushing of routing changes.

Much better than MX5 for edge, provided the throughput / port complement hits your requirements, and you've got redundancy elsewhere, since they're single PSU boxes.

level 1

If you are looking for Juniper, consider MX204 -- good price, excellent performance, great value for $.

If you are looking for Cisco, consider ASR-9901. ASR-9901-120G is entry level (most comparable to MX5) so price is attractive. Excellent performance like on Juniper's MX204. License costs to upgrade from 120G to 256G or full 456G (eh hem, more like 400G due to box architecture) is reasonable (a bit premium overhead price to upgrade, but nothing ridiculous like with MX5/MX104 port licensing schemes).

level 1
It places the packet on the wire or else it gets the hose again.1 point · 2 months ago

I'm a Juniper man through and through but as your budget is on the low side, you might look at Mikrotik's CCR lineup. Their 72 core CCR1072-1G-8S+ has eight SFP+ sockets that don't require any special licensing to use and you can pick them up for around $3K USD apiece. I've experimented with a similar model CCR in my lab and although it took some getting used to the OS, performance was surprisingly excellent.

level 2
Original Poster2 points · 2 months ago

And they can handle full table bgp?

level 3

Yes but Mikrotik still has single threaded BGP so don't expect it to compete with more serious routing hardware for crunching a 700k table really fast. Still they are good devices but like anything else they have quirks. Also an option on the low-end is the Ubiquiti Edgerouter Infinity with a Juniper like CLI but again these are still not on par with the big players so it's on you to figure out your levels of cost vs. risk.

level 4
Original Poster1 point · 2 months ago

The issue for me isn't so much the time it takes to process, but the time it takes to get itself in sync. In junos the route table gets processed in software really fast and therfore routes get announced downstream via ospf etc. Problem happens because asynchronous to that it has the krt queue where its pushing routes from software to the ASICs and so you end up with routers requesting traffic that they don't know how to route.

level 3
It places the packet on the wire or else it gets the hose again.2 points · 2 months ago

Yes, multiple full feeds. In this 2015 article using an older less optimized version of code and back when the DFZ was around the 500K mark, it took about a 90 seconds for the model I referenced to process a single full feed. They go on to demonstrate numbers using multiple full feeds.

http://www.stubarea51.net/2015/07/25/mikrotik-ccr1072-1g-8s-review-part-2-bgp-performance/

Pretty impressive stuff at that price point.

level 3

I'm a MikroTik user and have been working with them for years mainly (well basically only) in Edge Networking. Our Setup is 2x Full BGP feeds, plus 1x 20k or so Public IX feed, plus around 26 internal prefixes, connecting as well to a scrubber provider, we re-route and RTBH on a daily basis thanks to lovely DDoS.

The 1072s have been working great so far we used to have 1036s for around 2.5 years with no issues (one of them was not even updated or rebooted for that time), our usage is around 2 Gbps in and 3 Gbps out. We use a lot of filtering with multilple long regex to achieve good routing (this is in APAC kind of tricky with some paths). And basically full convergence takes around 1 - 3 minutes from 0. We get a lot of changes daily and I have yet to see a situation that made the router hang with BGP for 20 minutes. Those routers are also our edge filters for potentially bad things (Memcache for instance).

This being said, MikroTik is a brand that lacks proper support (you can't just call them and expect a fix), and some features may have problems, for instance if you want MPLS this likely won't be a good core device mainly because of the lack of ECMP when MPLS is enabled.

I'd advice you to make a list of real current and future needs and check the MKs pros and cons with said needs, get a consultant or go to the forums or just buy one and test (they are quite cheap, and you can also test for free using CHR virtual routers). Then make a decision, our operations team love these things mainly for Winbox (its GUI) which gives a ton of info that makes their life easier when tshooting.

Engineering loves and hate them (for the situation where we can't use them properly) and finances loves them... do most of what we need for a really cheap price and we really have had no unit fail on us at all (hardware wise, software wise 1 - 2 times fixed after updating).

However we had someone around that knew a lot about those devices that made our "transition" easier, and made us realize where to use and where not to use them.

level 4
Original Poster1 point · 2 months ago

Thanks, that's really helpful! We have around 200 CCRs deployed but they are currently only used as internal routers and firewalls. What sort of pps and mbps you getting through the 1072s?

level 5

Most of our units have conntrack disabled and or selectively disabled. I say this because with conntrack the numbers will be likely way different, although we use the firewall and QoS we do it stateless.

On the highest usage one, it's around 1.2 gbps in and 1.5 gbps out, around 200 - 450 kpps (in+out), typical CPU usage is around 0-3%

Community Details

131k

Subscribers

973

Online

###Enterprise Networking Routers, switches and firewalls. Network blogs, news and network management articles. Cisco, Juniper, Brocade and more all welcome.

Create Post
r/networking Rules
1.
Rule #1: No Home Networking.
2.
Rule #2: No Certification Brain Dumps / Cheating.
3.
Rule #3: No BlogSpam / Traffic re-direction.
4.
Rule #4: No Low Quality Posts.
5.
Rule #5: No Early Career Advice.
6.
Rule #6: Educational Questions must show effort.
Cookies help us deliver our Services. By using our Services or clicking I agree, you agree to our use of cookies. Learn More.