<div dir="ltr">Are you using zebra? <div><br></div><div>BGP has received a v6 LL address and it has absolutely no way to know what interface it has come in on as that it is depending on zebra for this data.</div><div><br></div><div>donald</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 25, 2021 at 7:15 PM Michael H Lambert <<a href="mailto:lambert@psc.edu">lambert@psc.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I’m trying to debug a rather strange issue I’m seeing with bgpd from FRR 7.5.1 (latest from FreeBSD 13.0 packages) used as a route server. The single peer for this test case is a VRF on a Cisco 6509 (although the problem also exists with Brocade and a couple of flavors of Junos). Peering is over a tagged (VLAN) subnet, ie, a subinterface on a physical Ethernet.<br>
<br>
The motivation is a smooth (only some minor syntactic changes needed) migration from the Euro-IX fork of quagga to FRR. The alternatives (BIRD or OpenBGPD) would probably require semantic changes in addition.<br>
<br>
[Note: IP addresses and ASNs have been obfuscated so our security folks won’t yell]<br>
<br>
The test configuration is straightforward:<br>
<br>
==========<br>
<br>
Current configuration:<br>
!<br>
frr version 7.5.1<br>
frr defaults traditional<br>
!<br>
hostname rs-dev-bgpd<br>
password password<br>
enable password password<br>
log syslog<br>
!<br>
debug bgp updates in<br>
debug bgp updates out<br>
!<br>
!<br>
!<br>
router bgp 64512 view TESTING<br>
bgp router-id 198.51.100.247<br>
bgp log-neighbor-changes<br>
no bgp ebgp-requires-policy<br>
no bgp default ipv4-unicast<br>
no bgp network import-check<br>
neighbor 198.51.100.248 remote-as 64536<br>
neighbor 198.51.100.248 description Peering with TESTING (VRF)<br>
neighbor 2001:db8:0:fe1f::18 remote-as 64536<br>
neighbor 2001:db8:0:fe1f::18 description Peering with TESTING (VRF)<br>
!<br>
address-family ipv4 unicast<br>
neighbor 198.51.100.248 activate<br>
neighbor 198.51.100.248 soft-reconfiguration inbound<br>
neighbor 198.51.100.248 route-server-client<br>
neighbor 198.51.100.248 route-map PERMIT_ANY in<br>
neighbor 198.51.100.248 route-map DENY_ANY out<br>
neighbor 198.51.100.248 attribute-unchanged next-hop<br>
exit-address-family<br>
!<br>
address-family ipv6 unicast<br>
neighbor 2001:db8:0:fe1f::18 activate<br>
neighbor 2001:db8:0:fe1f::18 soft-reconfiguration inbound<br>
neighbor 2001:db8:0:fe1f::18 route-server-client<br>
neighbor 2001:db8:0:fe1f::18 route-map PERMIT_ANY in<br>
neighbor 2001:db8:0:fe1f::18 route-map DENY_ANY out<br>
neighbor 2001:db8:0:fe1f::18 attribute-unchanged next-hop<br>
exit-address-family<br>
!<br>
!<br>
!<br>
route-map DENY_ANY deny 100<br>
!<br>
route-map PERMIT_ANY permit 100<br>
!<br>
!<br>
line vty<br>
exec-timeout 60 0<br>
!<br>
end<br>
<br>
==========<br>
<br>
I’m checking IPv4 first. The Cisco thinks its announcements are<br>
<br>
cisco#show bgp vrf TESTING vpnv4 unicast neighbors 198.51.100.247 advertised-routes<br>
BGP table version is 1867871, local router ID is 198.51.100.249<br>
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, <br>
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, <br>
x best-external, a additional-path, c RIB-compressed, <br>
Origin codes: i - IGP, e - EGP, ? - incomplete<br>
RPKI validation codes: V valid, I invalid, N Not found<br>
<br>
Network Next Hop Metric LocPrf Weight Path<br>
Route Distinguisher: 64536:1 (default for vrf TESTING) VRF Router ID 198.51.100.249<br>
*> <a href="http://192.0.2.0/24" rel="noreferrer" target="_blank">192.0.2.0/24</a> 0.0.0.0 0 32768 ?<br>
*> <a href="http://198.51.100.224/27" rel="noreferrer" target="_blank">198.51.100.224/27</a> 0.0.0.0 0 32768 ?<br>
<br>
Total number of prefixes 2 <br>
<br>
==========<br>
<br>
The BGP summary on FRR/bgpd shows the expected two routes:<br>
<br>
==========<br>
<br>
rs-dev-bgpd# show bgp view TESTING ipv4 unicast summary <br>
BGP router identifier 198.51.100.247, local AS number 64512 vrf-id 1<br>
BGP table version 2<br>
RIB entries 3, using 576 bytes of memory<br>
Peers 1, using 14 KiB of memory<br>
<br>
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt<br>
198.51.100.248 4 64536 16 12 0 0 0 00:10:05 2 0<br>
<br>
Total number of neighbors 1<br>
<br>
==========<br>
<br>
And they are also the expected routes, matching what the Cisco thinks it is sending.<br>
<br>
==========<br>
<br>
rs-dev-bgpd# show bgp view TESTING ipv4 unicast <br>
BGP table version is 2, local router ID is 198.51.100.247, vrf id 1<br>
Default local pref 100, local AS 64512<br>
Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,<br>
i internal, r RIB-failure, S Stale, R Removed<br>
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self<br>
Origin codes: i - IGP, e - EGP, ? - incomplete<br>
<br>
Network Next Hop Metric LocPrf Weight Path<br>
*> <a href="http://192.0.2.0/24" rel="noreferrer" target="_blank">192.0.2.0/24</a> 198.51.100.248 0 0 64536 ?<br>
*> <a href="http://198.51.100.224/27" rel="noreferrer" target="_blank">198.51.100.224/27</a> 198.51.100.248 0 0 64536 ?<br>
<br>
Displayed 2 routes and 2 total paths<br>
<br>
==========<br>
<br>
>From more extensive testing, IPv4 seems to be behaving as I think it should.<br>
<br>
Now I turn my attention to IPv6. Again, the Cisco seems happy:<br>
<br>
==========<br>
<br>
cisco#show bgp vrf TESTING vpnv6 unicast neighbors 2001:DB8:0:FE1F::17 advertised-routes<br>
BGP table version is 1642349, local router ID is 198.51.100.249<br>
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, <br>
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter, <br>
x best-external, a additional-path, c RIB-compressed, <br>
Origin codes: i - IGP, e - EGP, ? - incomplete<br>
RPKI validation codes: V valid, I invalid, N Not found<br>
<br>
Network Next Hop Metric LocPrf Weight Path<br>
Route Distinguisher: 64536:1 (default for vrf TESTING) VRF Router ID 198.51.100.249<br>
*> 2001:DB8:0:FE1F::/64<br>
:: 0 32768 ?<br>
*> 2001:DB8:FFFF:9F::/64<br>
:: 0 32768 ?<br>
<br>
Total number of prefixes 2 <br>
<br>
==========<br>
<br>
Two prefixes are being announced over the established BGP session.<br>
<br>
Indeed, bgpd sees the session as established, but it’s not receiving prefixes:<br>
<br>
rs-dev-bgpd# show bgp view TESTING ipv6 unicast summary <br>
BGP router identifier 198.51.100.247, local AS number 64512 vrf-id 1<br>
BGP table version 0<br>
RIB entries 0, using 0 bytes of memory<br>
Peers 1, using 14 KiB of memory<br>
<br>
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt<br>
2001:db8:0:fe1f::18 4 64536 16 12 0 0 0 00:10:16 0 0<br>
<br>
Total number of neighbors 1<br>
rs-dev-bgpd# show bgp view TESTING ipv6 unicast<br>
No BGP prefixes displayed, 0 exist<br>
<br>
==========<br>
<br>
So what’s going on here?<br>
<br>
In syslog (daemon.log), I see a few lines, two of which seem pretty strange:<br>
<br>
==========<br>
<br>
Oct 25 14:29:50 rs-dev bgpd[59414]: [EC 33554503] 2001:db8:0:fe1f::18 unrecognized capability code: 70 - ignored<br>
Oct 25 14:29:50 rs-dev bgpd[59414]: %ADJCHANGE: neighbor 2001:db8:0:fe1f::18(Unknown) in vrf Up<br>
Oct 25 14:29:50 rs-dev bgpd[59414]: 2001:db8:0:fe1f::18 sent a v6 LL next-hop and there's no peer interface information. Hence, withdrawing<br>
Oct 25 14:29:50 rs-dev bgpd[59414]: [EC 33554487] 2001:db8:0:fe1f::18: Attribute MP_REACH_NLRI, parse error - treating as withdrawal<br>
Oct 25 14:29:50 rs-dev bgpd[59414]: [EC 33554454] 2001:db8:0:fe1f::18 rcvd UPDATE with errors in attr(s)!! Withdrawing route.<br>
Oct 25 14:29:50 rs-dev bgpd[59414]: 2001:db8:0:fe1f::18 [Info] UPDATE for non-enabled AFI/SAFI 1/1<br>
Oct 25 14:29:50 rs-dev bgpd[59414]: bgp_update_receive: rcvd End-of-RIB for IPv6 Unicast from 2001:db8:0:fe1f::18 in vrf default<br>
<br>
==========<br>
<br>
Several comments/observations:<br>
<br>
1) The link-local next-hop and AFI/SAFI messages are present whether the peer is the Cisco or a Juniper or Brocade/Extreme. These are particularly befuddling, since they have no trouble talking to quagga or to each other.<br>
<br>
2) I tried assigning a different MAC address to the subinterface than that which was on the physical interface with the same results. I thought this might take care of link-local addresses and neighbor discovery issues.<br>
<br>
3) On Friday, I built the code from the head of the git master and tested it, again with the same results.<br>
<br>
4) If I mock this up on VMs under VirtualBox, using (emulated) physical interfaces (not tagged subinterfaces) and OpenBSD/OpenBGPD speakers instead of vendor routers, IPv4 and IPv6 both work as expected. I will note that I’m using ULAs and not global unicast addresses. In this case, it’s the same version of FreeBSD and FRR as in the physical test.<br>
<br>
At this point, my best guess is that I’m running into an interaction issue between FRR/bgpd and tagged subinterfaces. If that is the case, is it across all platforms? Restricted to FreeBSD? Specific to FreeBSD 13? Am I just missing another knob on bgpd for running as a route server? Do I need to run vtysh to have the interfaces handled properly? Do I need to do something with zebra even though there is no forwarding involved, just routing?<br>
<br>
Any insights would be most appreciated.<br>
<br>
Michael<br>
<br>
-----<br>
Michael H Lambert, GigaPoP Manager Phone: +1 412 268-4960<br>
Pittsburgh Supercomputing Center/3ROX FAX: +1 412 268-5832<br>
300 S Craig St, Pittsburgh, PA 15213 USA <a href="mailto:lambert@psc.edu" target="_blank">lambert@psc.edu</a><br>
<br>
<br>
<br>
_______________________________________________<br>
frog mailing list<br>
<a href="mailto:frog@lists.frrouting.org" target="_blank">frog@lists.frrouting.org</a><br>
<a href="https://lists.frrouting.org/listinfo/frog" rel="noreferrer" target="_blank">https://lists.frrouting.org/listinfo/frog</a><br>
</blockquote></div>