<div dir="ltr"><span style="text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Hi Donald,</span><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial">Thank you for your reply. I hope this isn't too jarring but we had to reproduce the issue on a slightly different host connected to different TORs. The issue is the same but the IPs are a bit different. Hopefully this is enough data. The network engineer with whom I'm working said to reproduce it all he had to do was "disable interface on switch towards eth0".</div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial">Carl</div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial">First, the host routes are a bit different here</div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial"><div> lab1r1u05# show ip bgp neighbor 10.112.97.1 received-routes</div><div> BGP table version is 0, local router ID is 10.112.128.1</div><span class="gmail-im" style="color:rgb(80,0,80)"><div> Status codes: s suppressed, d damped, h history, * valid, > best, = multipath,</div><div> i internal, r RIB-failure, S Stale, R Removed</div><div> Origin codes: i - IGP, e - EGP, ? - incomplete</div><div><br></div><div> Network Next Hop Metric LocPrf Weight Path</div></span><div> *><span> </span><a href="http://10.224.12.20/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.20/32</a> 10.112.128.2 0 4206900001 4206909999 i</div><div> *><span> </span><a href="http://10.224.12.25/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.25/32</a> 10.112.128.2 0 4206900001 4206909999 i</div><div> *><span> </span><a href="http://10.224.12.80/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.80/32</a> 10.112.128.2 0 4206900001 4206909999 i</div><div> *><span> </span><a href="http://10.224.12.85/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.85/32</a> 10.112.128.2 0 4206900001 4206909999 i</div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><div> Total number of prefixes 4</div></span></div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial">This is `show ip route` when the issue occurs. Note the difference in MPLS labels between the route to `<a href="http://10.112.128.2/32" target="_blank" style="color:rgb(17,85,204)">10.112.128.2/32`</a><span> </span>and the one to `<a href="http://10.224.12.25/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.25/32`</a>.</div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial"><div> lab1r1u05# show ip route</div><div> Codes: K - kernel route, C - connected, S - static, R - RIP,</div><div> O - OSPF, I - IS-IS, B - BGP, P - PIM, E - EIGRP, N - NHRP,</div><div> T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,</div><div> > - selected route, * - FIB route</div><div><br></div><div> K>*<span> </span><a href="http://0.0.0.0/0" target="_blank" style="color:rgb(17,85,204)">0.0.0.0/0</a><span> </span>[0/0] via 10.112.2.4, eth2, 04:27:00</div><div> C>*<span> </span><a href="http://10.112.2.0/25" target="_blank" style="color:rgb(17,85,204)">10.112.2.0/25</a><span> </span>is directly connected, eth2, 04:27:00</div><div> B>*<span> </span><a href="http://10.112.97.1/32" target="_blank" style="color:rgb(17,85,204)">10.112.97.1/32</a><span> </span>[20/0] via 10.112.129.1, eth1, label 696097, 04:26:57</div><div> * via 10.112.129.5, eth0, label 46, 04:26:57</div><div> C>*<span> </span><a href="http://10.112.128.1/32" target="_blank" style="color:rgb(17,85,204)">10.112.128.1/32</a><span> </span>is directly connected, lo, 04:27:00</div><div> B>*<span> </span><a href="http://10.112.128.2/32" target="_blank" style="color:rgb(17,85,204)">10.112.128.2/32</a><span> </span>[20/0] via 10.112.129.1, eth1, label 702241, 00:00:59</div><div> * via 10.112.129.5, eth0, label 564, 00:00:59</div><div> C>*<span> </span><a href="http://10.112.128.10/32" target="_blank" style="color:rgb(17,85,204)">10.112.128.10/32</a><span> </span>is directly connected, lo, 04:27:00</div><div> C>*<span> </span><a href="http://10.112.128.100/32" target="_blank" style="color:rgb(17,85,204)">10.112.128.100/32</a><span> </span>is directly connected, lo, 04:27:00</div><div> C>*<span> </span><a href="http://10.112.129.0/30" target="_blank" style="color:rgb(17,85,204)">10.112.129.0/30</a><span> </span>is directly connected, eth1, 04:27:00</div><div> C>*<span> </span><a href="http://10.112.129.4/30" target="_blank" style="color:rgb(17,85,204)">10.112.129.4/30</a><span> </span>is directly connected, eth0, 04:27:00</div><div> K>*<span> </span><a href="http://10.112.129.8/30" target="_blank" style="color:rgb(17,85,204)">10.112.129.8/30</a><span> </span>[0/0] via 10.112.129.1, eth1, 04:27:00</div><div> K>*<span> </span><a href="http://10.224.12.10/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.10/32</a><span> </span>[0/0] is directly connected, br0, 04:27:00</div><div> K>*<span> </span><a href="http://10.224.12.15/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.15/32</a><span> </span>[0/0] is directly connected, br0, 04:27:00</div><div> B> <span> </span><a href="http://10.224.12.20/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.20/32</a><span> </span>[20/0] via 10.112.128.2 (recursive), 04:26:51</div><div> * via 10.112.129.1, eth1, label 702161, 04:26:51</div><div> * via 10.112.129.5, eth0, label 560, 04:26:51</div><div> B> <span> </span><a href="http://10.224.12.25/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.25/32</a><span> </span>[20/0] via 10.112.128.2 (recursive), 04:26:51</div><div> * via 10.112.129.1, eth1, label 702161, 04:26:51</div><div> * via 10.112.129.5, eth0, label 560, 04:26:51</div><div> K>*<span> </span><a href="http://10.224.12.70/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.70/32</a><span> </span>[0/0] is directly connected, br0, 04:27:00</div><div> K>*<span> </span><a href="http://10.224.12.75/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.75/32</a><span> </span>[0/0] is directly connected, br0, 04:27:00</div><div> B> <span> </span><a href="http://10.224.12.80/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.80/32</a><span> </span>[20/0] via 10.112.128.2 (recursive), 04:26:51</div><div> * via 10.112.129.1, eth1, label 702161, 04:26:51</div><div> * via 10.112.129.5, eth0, label 560, 04:26:51</div><div> B> <span> </span><a href="http://10.224.12.85/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.85/32</a><span> </span>[20/0] via 10.112.128.2 (recursive), 04:26:51</div><div> * via 10.112.129.1, eth1, label 702161, 04:26:51</div><div> * via 10.112.129.5, eth0, label 560, 04:26:51</div><div> C>*<span> </span><a href="http://100.64.0.0/24" target="_blank" style="color:rgb(17,85,204)">100.64.0.0/24</a><span> </span>is directly connected, br0, 04:27:00</div></div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial">Here is the route to the two TORs to which FRR is connected directly.</div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial"><div> lab1r1u05# show ip route 10.112.129.1</div><div> Routing entry for<span> </span><a href="http://10.112.129.0/30" target="_blank" style="color:rgb(17,85,204)">10.112.129.0/30</a></div><div> Known via "connected", distance 0, metric 0, best</div><div> Last update 04:28:52 ago</div><div> * directly connected, eth1</div></div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial"><div> lab1r1u05# show ip route 10.112.129.5</div><div> Routing entry for<span> </span><a href="http://10.112.129.4/30" target="_blank" style="color:rgb(17,85,204)">10.112.129.4/30</a></div><div> Known via "connected", distance 0, metric 0, best</div><div> Last update 04:28:55 ago</div><div> * directly connected, eth0</div></div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial">The MPLS fec</div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial"><div> lab1r1u05# show mpls fec</div><div> <span> </span><a href="http://10.112.97.1/32" target="_blank" style="color:rgb(17,85,204)">10.112.97.1/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.112.128.2/32" target="_blank" style="color:rgb(17,85,204)">10.112.128.2/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.112.128.100/32" target="_blank" style="color:rgb(17,85,204)">10.112.128.100/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.224.12.10/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.10/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.224.12.15/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.15/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.224.12.20/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.20/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.224.12.25/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.25/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.224.12.70/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.70/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.224.12.75/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.75/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.224.12.80/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.80/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> <span> </span><a href="http://10.224.12.85/32" target="_blank" style="color:rgb(17,85,204)">10.224.12.85/32</a></div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:201::1/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:202::2/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:203::10/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:203::15/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:203::20/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:203::25/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:204::70/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:204::75/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:204::80/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div><div> 2604:a880:801:204::85/128</div><div> Label: 4294836223</div><div> Client list: bgp(fd 11)</div></div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial">... and the MPLS table.</div><div style="text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="text-decoration-style:initial;text-decoration-color:initial"><div> lab1r1u05# show mpls table (yes, it is empty)</div><div><br></div><div> Inbound Outbound</div><div> Label Type Nexthop Label</div><div> -------- ------- --------------- --------</div></div><br></div><br><div class="gmail_quote"><div dir="ltr">On Tue, Jun 26, 2018 at 6:11 AM Donald Sharp <<a href="mailto:sharpd@cumulusnetworks.com">sharpd@cumulusnetworks.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Can we get the output of `show ip route`, `show ip route<br>
10.112.129.9`, `show mpls fec`, and `show mpls table`?<br>
<br>
donald<br>
<br>
On Mon, Jun 25, 2018 at 3:45 PM, Carl Baldwin <<a href="mailto:carl@ecbaldwin.net" target="_blank">carl@ecbaldwin.net</a>> wrote:<br>
> Hi,<br>
><br>
> We're experimenting with BGP-LU and frr in our lab. We have two host<br>
> machines running frr. Each is connected to two TOR switches through their<br>
> eth0 and eth1 links (four TORs total). Those are connected through another<br>
> pair of switches. We have configured BGP-LU to distribute MPLS labels over<br>
> ebpg throughout. Connectivity between loopback addresses on the two hosts<br>
> works well at this point.<br>
><br>
> Then, I created some namespaces in the hosts and gave them addresses. We use<br>
> BGP to announce those addresses as /32 routes with the loopback address as<br>
> the next hop. The route between namespaces on the two machines is<br>
> recursively resolved so that they push the same MPLS label as the path to<br>
> the other loopback. This all works well to start off with and we did some<br>
> iperf runs that showed pretty good results. ECMP was working because the<br>
> bandwidth was higher than any single link.<br>
><br>
> After some link state changes, we seemed to lose the connection. However,<br>
> pings between the loopback addresses still worked. After some time, we<br>
> noticed that the MPLS labels in the routes to the namespace addresses (/32s)<br>
> were different than the label in the route to the loopback. Since the former<br>
> routes are resolved recursively using the latter, the labels should always<br>
> be the same. Could this be a bug in FRR? Shouldn't the routes to the<br>
> namespaces be invalidated or updated as soon as the route it was based on<br>
> changed? The traffic between namespaces is getting dropped because the<br>
> switch doesn't know about the label being pushed by the host.<br>
><br>
> Any insight would be very helpful.<br>
><br>
> Thanks!<br>
> Carl Baldwin<br>
><br>
> Here are /32 routes received. 10.112.128.1 is the loopback on the other<br>
> host. The four routes are to four namespaces on the other host.<br>
><br>
> lab1r2u05# show ip bgp neighbor 10.112.97.1 received-routes<br>
> BGP table version is 0, local router ID is 10.112.128.2<br>
> Status codes: s suppressed, d damped, h history, * valid, > best, =<br>
> multipath,<br>
> i internal, r RIB-failure, S Stale, R Removed<br>
> Origin codes: i - IGP, e - EGP, ? - incomplete<br>
><br>
> Network Next Hop Metric LocPrf Weight Path<br>
> *> <a href="http://10.224.12.10/32" rel="noreferrer" target="_blank">10.224.12.10/32</a> 10.112.128.1 0 4206900001<br>
> 4206909998 i<br>
> *> <a href="http://10.224.12.15/32" rel="noreferrer" target="_blank">10.224.12.15/32</a> 10.112.128.1 0 4206900001<br>
> 4206909998 i<br>
> *> <a href="http://10.224.12.70/32" rel="noreferrer" target="_blank">10.224.12.70/32</a> 10.112.128.1 0 4206900001<br>
> 4206909998 i<br>
> *> <a href="http://10.224.12.75/32" rel="noreferrer" target="_blank">10.224.12.75/32</a> 10.112.128.1 0 4206900001<br>
> 4206909998 i<br>
><br>
> Total number of prefixes 4<br>
><br>
> Below is the routing table as it looked when we lost connectivity. Notice<br>
> that the mpls label for the loopback route is 306592 via eth1 but the label<br>
> for the four namespace addresses is 306576.<br>
><br>
> root@lab1r2u05:~/ovs-droplets# ip route<br>
> default via 10.112.2.132 dev eth2<br>
> <a href="http://10.112.2.128/25" rel="noreferrer" target="_blank">10.112.2.128/25</a> dev eth2 proto kernel scope link src 10.112.2.145<br>
> 10.112.128.1 encap mpls 306592 via 10.112.129.9 dev eth1 proto 186<br>
> metric 20<br>
> <a href="http://10.112.129.8/30" rel="noreferrer" target="_blank">10.112.129.8/30</a> dev eth1 proto kernel scope link src 10.112.129.10<br>
> <a href="http://10.112.129.12/30" rel="noreferrer" target="_blank">10.112.129.12/30</a> dev eth0 proto kernel scope link src 10.112.129.14<br>
> 10.224.12.10 encap mpls 306576 via 10.112.129.9 dev eth1 proto 186<br>
> metric 20<br>
> 10.224.12.15 encap mpls 306576 via 10.112.129.9 dev eth1 proto 186<br>
> metric 20<br>
> 10.224.12.70 encap mpls 306576 via 10.112.129.9 dev eth1 proto 186<br>
> metric 20<br>
> 10.224.12.75 encap mpls 306576 via 10.112.129.9 dev eth1 proto 186<br>
> metric 20<br>
> 10.224.12.20 dev br0 scope link<br>
> 10.224.12.25 dev br0 scope link<br>
> 10.224.12.80 dev br0 scope link<br>
> 10.224.12.85 dev br0 scope link<br>
><br>
><br>
> _______________________________________________<br>
> dev mailing list<br>
> <a href="mailto:dev@lists.frrouting.org" target="_blank">dev@lists.frrouting.org</a><br>
> <a href="https://lists.frrouting.org/listinfo/dev" rel="noreferrer" target="_blank">https://lists.frrouting.org/listinfo/dev</a><br>
><br>
</blockquote></div>