I have a weird (and very frustrating) issue I'm troubleshooting and I'm hoping someone can point me in the right direction...
I have a cluster of two ASA 5585-S SSP-60 firewalls in transparent mode. Upstream are two Nexus 7Ks. As you may know, in cluster mode the upstream N7Ks will see the ASAs as a single unit. The N7Ks have a vPC that spans the ASA cluster. Each N7K has two 10GE links to each ASA in the vPC, providing 40Gbps to each ASA. It’s a bit overkill IMO, but it is what it is.
Each ASA should be able to handle between 20Gbps to 40Gbps, depending on traffic type/protocols/etc.
The firewalls have several firewalled networks (e.g. 10.63.100.0/24, 10.63.253.0/24, etc). The firewalled networks are separated via subinterfaces (e.g. Network 10.63.100.0/24 is on Po10.14, Network 10.63.253.0/24 is on Po10.15, etc).
And as you may know, in transparent mode each firewalled network gets two subinterfaces (one as the inside interface and one as the outside interface), two VLANs (one for the inside interface and one for the outside), one bridge-group to bridge the two VLANs, and one BVI interface for the bridge-group
I’m only seeing no more than 2Gbps throughput when going from either a non-firewalled network to a firewalled network, or a firewalled network to another firewalled network. So if a host on a firewalled network (e.g. 10.63.100.x) tries to do a file transfer with a host on another firewalled network (e.g. 10.63.253.x), they see no more than 2Gbps. The hosts are VMs living Cisco UCS chassis. The chassis connect to Fabric Interconnects (FIs), then FIs to Nexus 5Ks, N5Ks to N7Ks, then N7Ks to ASAs. I did the throughput tests using iperf (with 50M window scaling) and using FTP file transfers. Running the same tests from non-firewalled hosts to non-firewalled hosts over the same switching infrastructure and I get 9Gbps (the maximum limit of the virtual servers I was testing). This tells me the issue isn't on the UCS, FI, N5K, N7K, but instead
I see two things that could be causing this issue.
I see incrementing packet drops on the pertinent port-channel subinterfaces that I’m transferring data through on the ASA. However, I do NOT see errors/drops on the physical interfaces (the port-channel doesn’t show me counters).
Using the command “show asp drop,” I see "Dispatch queue tail drops (dispatch-queue-limit)" increase while I transfer the traffic, but the google results are vague. My findings suggest the packet loss (or poor performance) is software related within the ASA and NOT a layer 1 issue…but I don’t know how to troubleshoot this further.
Have any of you see something similar?
How can subinterfaces incur errors, but not the physical interfaces?
I've posted the output of "show interface" for one of the subinterfaces and the physical links here