Has anyone found a good way to run RFC 6349 tests over WAN links in an overall solution that scales easily? We have 100+ sites and really could use a server-based solution, something that we could deploy fairly easily.
I wouldn't get too hung up on trying to implement something that claims compliance with RFC6349, since that RFC is a little out of date and mostly just says to run a tool such as iperf. Also, you have to be a little careful with making a TCP throughput server generally available, since such tests consume (or at least attempt to consume) all available bandwidth for the duration of the test, which can amount to a self-inflicted denial of service. One option you may want to look into is the BWCTL (Bandwidth Test Controller), since it seems close to what you want to do and supports a few different test tools, such as iperf and nuttcp.
My impression was that 6349 provides better information than iperf (path mtu discovery, RTT, etc) for each application stream simultaneously. We use iperf now but it doesn't provide the full story. I'll check out that BWCTL - have you used it?
No, I have not used it, mostly because I don't have access to an environment that could benefit from it, but I have used most of the tools it can manage.
Regarding the statistics mentioned in RFC 6349, while many of them sound useful they don't tell the whole story. For example, one of the biggest throughput-related parameters I see involves the configuration of the traffic policers usually used by providers to enforce the customer's subscribed rate. Short of examining the device configs of the equipment in the path this is a non-trivial thing to quantify, but it can have a huge impact on performance, especially when TCP is used. RFC 6349 doesn't mention this at all, which is why I mentioned it being a little dated. I'm all for quantifying things, especially when it comes to SLA's, but some stats are more meaningful than others.
Are you saying you want to test the internet between your end sites?
We want to test metro Ethernet links that we lease from a service provider. We often find that they provision the links incorrectly and give us less bandwidth than we pay for.
So, then the question to ask is....do you ask for their testing of the service before your company accepts it from them?
Also if you want to do proper testing then get yourself a two test sets from like EXFO, JDSU, Xena Networks, Spirent, Ixia. That generally would be the best ways to do that.
Those test sets don't scale well when you have hundreds of remote sites. We also would like a solution that doesn't require a human on the other side of the link. Service providers usually can't test TCP throughput, they usually just test layers 2/3.
Those test sets don't scale well when you have hundreds of remote sites.
Yes, that is true. But proper testing of a link isn't meant to be distributed. It's meant to test just that link and that's it. There's a reason why those test sets and test suites work like that. The reason is because the sensitivity needed to prove something out isn't trivial.
We also would like a solution that doesn't require a human on the other side of the link.
I understand, but do you want to be able to verify that the link truly works? If the business is willing to accept sub-par results then that is something that has to be discussed.
Service providers usually can't test TCP throughput, they usually just test layers 2/3.
True, and that's because it's not what they are selling (TCP throughput). They are selling connectivity, and bandwidth. They don't touch TCP/UDP traffic. They only punt it. It's up to the end points to tune their TCP implementations. If your company is requiring for TCP throughput to be a certain amount then they will need to invest in proper testing tools to validate that said level of service.
What /u/djdawson said is probably going to be the best thing to use in this case.
The amount of "engineers" that don't understand TCP is really frustrating.
Especially when they suggest that service providers are selling them "layer 7 throughput."
To hell with those costly tools, use T-REX instead:
TBH I more use that for testing networks where there are middle-boxes, firewalls, NATs etc.
For raw testing of a metro ethernet type service I would probably just use iperf3 in UDP mode instead. Right tool for the right job.
Routers, switches and firewalls. Network blogs, news and network management articles. Cisco, Juniper, Brocade and more all welcome.