[FROG] TCP retransmits

Travis Garrison tgarrison at netviscom.com
Mon Dec 30 15:24:16 EST 2019


We are getting sporadic speed issues and other weirdness after we installed our 2 new cores running FRR. The routers are HP DL380G8 servers with 64Gb of ram, dual Intel Xeon E5-2690 V2 3GHz 10 core CPUs. We disabled hyperthreading. The nics are Mellanox ConnectX3 Pros 40Gb using the native drivers from Debian 10. All updates and firmware have been applied. We show no errors in the frr.log or messages. We did notice that in a packet capture we are getting a lot of TCP retransmits and TCP out of orders on the internal side. Looks to be about 40 to 50% of our traffic is retransmits. Anyone here have any ideas?

We have 2 10Gb internet feeds coming into 2 Mikrotik CRS326-24S+2Q+RM core switches. These switches are then connected with 40Gb DAC cables to the core routers on interface eno1. The switches are not connected together. The 2 core routers also have a 40Gb DAC cable connecting both together on interface eno1d1. Also we have 2 10Gb fiber feeds running to all the edge routers which are Mikrotik RB4011 or CCR1009s that are connected to the core switches. I can draw a diagram out but was unsure if I could attach it to this email or not. The core routers are running an MTU size of 2000 (with the exception of the ISP feed which is running 1500). We have verified that the edge routers and core switches are also running at 2000 MTU. The fiber network supports up to 8096 MTU. ISP1 is giving us full tables while ISP2 is currently giving us a default route only. They will be adding the capability of BGP and full routes in the next month. Through ISP1 we are also joining the local IXC and will be peering with Netflix. Do we need to redistribute the routes between the 2 cores?

Core router
/etc/network/interfaces
auto eno1
iface eno1 inet manual
        mtu 2000
#ISP
auto eno1.4003
iface eno1.4003 inet static
        address 1x.x.58.114/30
        gateway 1x.x.58.113
        dns-nameservers x.x.253.138 x.x.2.3
        mtu 1500
#RTR
auto eno1.3202
iface eno1.3202 inet static
        address 10.77.129.4/31
#Crossconnect to Core 2
auto eno1d1
iface eno1d1 inet static
        address 10.77.129.0/31
        mtu 2000

#FRR
interface eno1.3202
 ip ospf network point-to-point
!
interface eno1d1
 ip ospf cost 1
 ip ospf network point-to-point
!
router-id 10.77.131.0
!
router bgp 1234
 neighbor 10.77.131.1 remote-as 1234
 neighbor 10.77.131.1 update-source 10.77.131.0
 neighbor 1x.x.58.113 remote-as 4321
 neighbor 1x.x.58.113 update-source 1x.x.58.114
 !
 address-family ipv4 unicast
  network 10.x.20.0/24
  neighbor 10.77.131.5 default-originate route-map GW
  neighbor 10.77.131.5 prefix-list defaultonly out
  neighbor 1x.x.58.113 soft-reconfiguration inbound
  neighbor 1x.x.58.113 prefix-list infilter in
  neighbor 1x.x.58.113 prefix-list outfilter out
 exit-address-family
!
router ospf
 ospf router-id 10.77.131.0
 passive-interface eno1.4003
 network 10.77.129.0/31 area 0.0.0.0
!
ip prefix-list defaultonly seq 100 permit 0.0.0.0/0
ip prefix-list infilter seq 100 permit 0.0.0.0/0 le 24
ip prefix-list infilter seq 6 deny 10.77.128.0/22 le 32
ip prefix-list outfilter seq 5 permit 10.x.20.0/24 le 24
!
route-map GW permit 10
 set metric 1

#ip l
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether x:x:1b:80:c4:01 brd ff:ff:ff:ff:ff:ff
3: eno1d1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2000 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether x:x:1b:80:c4:02 brd ff:ff:ff:ff:ff:ff
5: eno1.4003 at eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether x:x:1b:80:c4:01 brd ff:ff:ff:ff:ff:ff
8: eno1.3202 at eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether x:x:1b:80:c4:01 brd ff:ff:ff:ff:ff:ff


Thanks
Travis Garrison




More information about the frog mailing list