I’m trying to debug a rather strange issue I’m seeing with bgpd from FRR 7.5.1 (latest from FreeBSD 13.0 packages) used as a route server. The single peer for this test case is a VRF on a Cisco 6509 (although the problem also exists with Brocade and a couple of flavors of Junos). Peering is over a tagged (VLAN) subnet, ie, a subinterface on a physical Ethernet.
The motivation is a smooth (only some minor syntactic changes needed) migration from the Euro-IX fork of quagga to FRR. The alternatives (BIRD or OpenBGPD) would probably require semantic changes in addition.
[Note: IP addresses and ASNs have been obfuscated so our security folks won’t yell]
The test configuration is straightforward:
==========
Current configuration:
!
frr version 7.5.1
frr defaults traditional
!
hostname rs-dev-bgpd
password password
enable password password
log syslog
!
debug bgp updates in
debug bgp updates out
!
!
!
router bgp 64512 view TESTING
bgp router-id 198.51.100.247
bgp log-neighbor-changes
no bgp ebgp-requires-policy
no bgp default ipv4-unicast
no bgp network import-check
neighbor 198.51.100.248 remote-as 64536
neighbor 198.51.100.248 description Peering with TESTING (VRF)
neighbor 2001:db8:0:fe1f::18 remote-as 64536
neighbor 2001:db8:0:fe1f::18 description Peering with TESTING (VRF)
!
address-family ipv4 unicast
neighbor 198.51.100.248 activate
neighbor 198.51.100.248 soft-reconfiguration inbound
neighbor 198.51.100.248 route-server-client
neighbor 198.51.100.248 route-map PERMIT_ANY in
neighbor 198.51.100.248 route-map DENY_ANY out
neighbor 198.51.100.248 attribute-unchanged next-hop
exit-address-family
!
address-family ipv6 unicast
neighbor 2001:db8:0:fe1f::18 activate
neighbor 2001:db8:0:fe1f::18 soft-reconfiguration inbound
neighbor 2001:db8:0:fe1f::18 route-server-client
neighbor 2001:db8:0:fe1f::18 route-map PERMIT_ANY in
neighbor 2001:db8:0:fe1f::18 route-map DENY_ANY out
neighbor 2001:db8:0:fe1f::18 attribute-unchanged next-hop
exit-address-family
!
!
!
route-map DENY_ANY deny 100
!
route-map PERMIT_ANY permit 100
!
!
line vty
exec-timeout 60 0
!
end
==========
I’m checking IPv4 first. The Cisco thinks its announcements are
cisco#show bgp vrf TESTING vpnv4 unicast neighbors 198.51.100.247 advertised-routes
BGP table version is 1867871, local router ID is 198.51.100.249
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 64536:1 (default for vrf TESTING) VRF Router ID 198.51.100.249
*> 192.0.2.0/24 0.0.0.0 0 32768 ?
*> 198.51.100.224/27 0.0.0.0 0 32768 ?
Total number of prefixes 2
==========
The BGP summary on FRR/bgpd shows the expected two routes:
==========
rs-dev-bgpd# show bgp view TESTING ipv4 unicast summary
BGP router identifier 198.51.100.247, local AS number 64512 vrf-id 1
BGP table version 2
RIB entries 3, using 576 bytes of memory
Peers 1, using 14 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
198.51.100.248 4 64536 16 12 0 0 0 00:10:05 2 0
Total number of neighbors 1
==========
And they are also the expected routes, matching what the Cisco thinks it is sending.
==========
rs-dev-bgpd# show bgp view TESTING ipv4 unicast
BGP table version is 2, local router ID is 198.51.100.247, vrf id 1
Default local pref 100, local AS 64512
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,
i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes: i - IGP, e - EGP, ? - incomplete
Network Next Hop Metric LocPrf Weight Path
*> 192.0.2.0/24 198.51.100.248 0 0 64536 ?
*> 198.51.100.224/27 198.51.100.248 0 0 64536 ?
Displayed 2 routes and 2 total paths
==========
From more extensive testing, IPv4 seems to be behaving as I think it should.
Now I turn my attention to IPv6. Again, the Cisco seems happy:
==========
cisco#show bgp vrf TESTING vpnv6 unicast neighbors 2001:DB8:0:FE1F::17 advertised-routes
BGP table version is 1642349, local router ID is 198.51.100.249
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 64536:1 (default for vrf TESTING) VRF Router ID 198.51.100.249
*> 2001:DB8:0:FE1F::/64
:: 0 32768 ?
*> 2001:DB8:FFFF:9F::/64
:: 0 32768 ?
Total number of prefixes 2
==========
Two prefixes are being announced over the established BGP session.
Indeed, bgpd sees the session as established, but it’s not receiving prefixes:
rs-dev-bgpd# show bgp view TESTING ipv6 unicast summary
BGP router identifier 198.51.100.247, local AS number 64512 vrf-id 1
BGP table version 0
RIB entries 0, using 0 bytes of memory
Peers 1, using 14 KiB of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt
2001:db8:0:fe1f::18 4 64536 16 12 0 0 0 00:10:16 0 0
Total number of neighbors 1
rs-dev-bgpd# show bgp view TESTING ipv6 unicast
No BGP prefixes displayed, 0 exist
==========
So what’s going on here?
In syslog (daemon.log), I see a few lines, two of which seem pretty strange:
==========
Oct 25 14:29:50 rs-dev bgpd[59414]: [EC 33554503] 2001:db8:0:fe1f::18 unrecognized capability code: 70 - ignored
Oct 25 14:29:50 rs-dev bgpd[59414]: %ADJCHANGE: neighbor 2001:db8:0:fe1f::18(Unknown) in vrf Up
Oct 25 14:29:50 rs-dev bgpd[59414]: 2001:db8:0:fe1f::18 sent a v6 LL next-hop and there's no peer interface information. Hence, withdrawing
Oct 25 14:29:50 rs-dev bgpd[59414]: [EC 33554487] 2001:db8:0:fe1f::18: Attribute MP_REACH_NLRI, parse error - treating as withdrawal
Oct 25 14:29:50 rs-dev bgpd[59414]: [EC 33554454] 2001:db8:0:fe1f::18 rcvd UPDATE with errors in attr(s)!! Withdrawing route.
Oct 25 14:29:50 rs-dev bgpd[59414]: 2001:db8:0:fe1f::18 [Info] UPDATE for non-enabled AFI/SAFI 1/1
Oct 25 14:29:50 rs-dev bgpd[59414]: bgp_update_receive: rcvd End-of-RIB for IPv6 Unicast from 2001:db8:0:fe1f::18 in vrf default
==========
Several comments/observations:
1) The link-local next-hop and AFI/SAFI messages are present whether the peer is the Cisco or a Juniper or Brocade/Extreme. These are particularly befuddling, since they have no trouble talking to quagga or to each other.
2) I tried assigning a different MAC address to the subinterface than that which was on the physical interface with the same results. I thought this might take care of link-local addresses and neighbor discovery issues.
3) On Friday, I built the code from the head of the git master and tested it, again with the same results.
4) If I mock this up on VMs under VirtualBox, using (emulated) physical interfaces (not tagged subinterfaces) and OpenBSD/OpenBGPD speakers instead of vendor routers, IPv4 and IPv6 both work as expected. I will note that I’m using ULAs and not global unicast addresses. In this case, it’s the same version of FreeBSD and FRR as in the physical test.
At this point, my best guess is that I’m running into an interaction issue between FRR/bgpd and tagged subinterfaces. If that is the case, is it across all platforms? Restricted to FreeBSD? Specific to FreeBSD 13? Am I just missing another knob on bgpd for running as a route server? Do I need to run vtysh to have the interfaces handled properly? Do I need to do something with zebra even though there is no forwarding involved, just routing?
Any insights would be most appreciated.
Michael
-----
Michael H Lambert, GigaPoP Manager Phone: +1 412 268-4960
Pittsburgh Supercomputing Center/3ROX FAX: +1 412 268-5832
300 S Craig St, Pittsburgh, PA 15213 USA lambert(a)psc.edu
