<div dir="ltr"><div>Hi.</div><div><br></div><div>Please can someone help me, I am a bit stuck at the moment and not winning?</div><div><br></div><div>The image is the lab I am testing. PE1,P1,P2 and PE2 are Junos devices. The leaf/spines are Cumulus implementations on Mellanox switches.</div><div><br></div><div>                                           +-------+                                                +-------+                                                       </div><div>                                           |       |                                                |       |                                                       </div><div>                      |-------------------->   P1  |<----------------------------------------------->  P2   |                                                       </div><div>                      |                    |       |                                                |       |                                                       </div><div>                      |                    |       |                                                |       |                                                       </div><div>                      |                    +-------+                                                +-------+                                                       </div><div>  PODA                |                                                                               ^                                                             </div><div>+--------------------------------------------+                                                        |                                                             </div><div>|                 +-------+                  |                                                        |                                                             </div><div>| ^--------------->       |<--------------^  |                                                        |                                                             </div><div>| |      y.y.y.y  |  PE1  |               |  |                                                        |                                                             </div><div>| |               |z.z.z.z|               |  |                                                        |                                                             </div><div>| |               |       |               |  |                                                        |                                                             </div><div>| |               +-------+               |  |                             PODB                       |                                                             </div><div>| |                                       |  |                            +--------------------------------------+                                                  </div><div>| |                                       |  |                            |                           |          |                                                  </div><div>| |  +-------+                +-------+   |  |                            |    +-------+              |          |                                                  </div><div>| |  |       |                |       |   |  |                            |    |       |              |          |                                                  </div><div>| |  | Spine1|                | Spine2|   |  |                            |    | Spine1|              |          |                                                  </div><div>| |  |       |                |       |   |  |            +               |    |       |              |          |                                                  </div><div>| |  |       |                |       |   |  |                            |    |       |              |          |                                                  </div><div>| |  +-------+--              +-------+   |  |                            |    +-------+              |          |                                                  </div><div>| |      |      \--         --    |       |  |                            |        |                  |          |                                                  </div><div>| |      |         \-   ---/      |       |  |                            |        |                  |          |                                                  </div><div>| |      |          ---/          |       |  |                            |        |                  |          |                                                  </div><div>| |      |      ---/    \--       |       |  |                            |        |                  |          |                                                  </div><div>| |      v   <-/           \>     v       |  |                            |        v                  v          |                                                  </div><div>| |  +------------+           +-------+   |  |                            |    +-------+           +-------+     |                                                  </div><div>| |  |            |           |       |   |  |                            |    |       |           |       |     |                                                  </div><div>| |  |            |           |Leaf2  |   |  |                            |    |Leaf1  |           | PE2   |     |                                                  </div><div>| v--|  Leaf1     |           |       |---v  |                            |    |       |<--------> |       |     |                                                  </div><div>|    | b.b.b.b    |           |c.c.c.c|      |                            |    |x.x.x.x|  a.a.a.a  |g.g.g.g|     |                                                  </div><div>|    +------------+           +-------+      |                            |    +-------+           +-------+     |                                                  </div><div>|         <--                        <-      |                            |     ->                               |                                                  </div><div>|            \----                     \--   |                            |  --/                                 |                                                  </div><div>+--------------------------------------------+                           -|-/                                    |                                                  </div><div>                       \---                   \> Port2                --/ +--------------------------------------+                                                  </div><div>                           \----          +------------------------+</                                                                                              </div><div>                                \----     |                        | Port3                                                                                          </div><div>                                     \->  |                        |                                                                                                </div><div>                                          |      Testing Device    |                                                                                                </div><div>                                     Port1|                        |                                                                                                </div><div>                                          +------------------------+                                                                                                </div><div>                                                                                                                      </div><div><br></div><div>Internally in both pods, the architecture is L3 and using eBGP by means of unnumbered interfaces, advertising connected routes. In PODA PE1 is connected to both leafs, over separate links with labeled-unicast enabled, using implicit-null. The testing device is a Juniper SRX, with each interface setup in a virtual router, but part of the same subnet. On the switch side, the port facing the testing device is in a bridge, with a VNI setup and the local tunnel endpoint the loopback address. I am also using an SVI on the same subnet. Each port can reach the SVI locally. and testing between Port1 and Port2 is successful via the uplinks to PE1, label switching seems to be working correctly. There is reachability between the loopbacks of all the leafs as well. It seems there is a problem with the route-map since switching to labeled-unicast routes after testing ospf and ldp. I still need to confirm that, in order to test the reachability over the leaf/spine network, as it was working. </div><div><br></div><div>I am running into an issue with testing between Port1/Port2 and Port3. It seems like all routes are present, but with the eBGP architecture, the standard operation is to change the next-hop on external routes, but it changes the remote vtep as well. The mac/ip from Port3 is advertised by PODB leaf 1 with the VTEP as x.x.x.x, however displaying it on PODA Leaf1, it reports as y.y.y.y, where y.y.y.y is the link local address of PE1.</div><div><br></div><div>root@poda-leaf1:~# net show evpn mac vni 1001 </div><div>Number of MACs (local and remote) known for this VNI: 3</div><div>MAC               Type   Intf/Remote VTEP      VLAN </div><div>54:4b:8c:51:1c:a9 local  swp13                 1001 </div><div>54:4b:8c:51:1c:ad remote y.y.y.y</div><div><br></div><div>root@poda-leaf1:~# net show bgp evpn route vni 1001 mac 54:4b:8c:51:1c:ad</div><div>BGP routing table entry for [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]</div><div>Paths: (1 available, best #1)</div><div>  Not advertised to any peer</div><div>  Route [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad] VNI 993</div><div>  Imported from x.x.x.x:2:[2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]</div><div>  11111 65202</div><div>    y.y.y.y from y.y.y.y (z.z.z.z)</div><div>      Origin IGP, metric 200, localpref 100, valid, external, bestpath-from-AS 11111, best</div><div>      Extended Community: RT:65202:1001 ET:8</div><div>      AddPath ID: RX 0, TX 56</div><div>      Last update: Wed Aug  1 11:54:59 2018</div><div><br></div><div>I am also not understanding why the route above outputs VNI 993, it does however seem to import correctly into 1001. </div><div><br></div><div>root@poda-leaf1:~# net show bgp evpn route vni 1001 vtep y.y.y.y</div><div>BGP table version is 33, local router ID is b.b.b.b</div><div>Status codes: s suppressed, d damped, h history, * valid, > best, i - internal</div><div>Origin codes: i - IGP, e - EGP, ? - incomplete</div><div>EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]</div><div>EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]</div><div>EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]</div><div><br></div><div>   Network          Next Hop            Metric LocPrf Weight Path</div><div>*> [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]</div><div>                    y.y.y.y           200             0 11111 65202 i</div><div>*> [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]:[32]:[10.2.0.200]</div><div>                    y.y.y.y           200             0 11111 65202 i</div><div>*> [3]:[0]:[32]:[x.x.x.x]</div><div>                    y.y.y.y           200             0 11111 65202 i</div><div><br></div><div><br></div><div>Checking in anything is present for the loopback of podb-leaf1</div><div>root@poda-leaf1:~# net show bgp evpn route vni 1001 vtep x.x.x.x</div><div>BGP table version is 33, local router ID is b.b.b.b</div><div>Status codes: s suppressed, d damped, h history, * valid, > best, i - internal</div><div>Origin codes: i - IGP, e - EGP, ? - incomplete</div><div>EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]</div><div>EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]</div><div>EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]</div><div><br></div><div>   Network          Next Hop            Metric LocPrf Weight Path</div><div><br></div><div>Displayed 9 prefixes (0 paths) </div><div><br></div><div><br></div><div>Below is the same command on the originating leaf.</div><div>         </div><div>root@podB-leaf-01:~# net show bgp evpn route vni 1001 mac 54:4b:8c:51:1c:ad</div><div>BGP routing table entry for [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]</div><div>Paths: (1 available, best #1)</div><div>  Not advertised to any peer</div><div>  Route [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad] VNI 1001</div><div>  Local</div><div>    x.x.x.x from 0.0.0.0 (x.x.x.x)</div><div>      Origin IGP, localpref 100, weight 32768, valid, sourced, local, bestpath-from-AS Local, best</div><div>      Extended Community: ET:8 RT:65202:1001</div><div>      AddPath ID: RX 0, TX 63</div><div>      Last update: Wed Aug  1 11:51:44 2018</div><div><br></div><div><br></div><div>Spanning the uplink port to another port on the same switch, allowed me to look at the dataplane, and it does confirm that it is sending the traffic to the wrong destination.</div><div><br></div><div>tcpdump on the uplink interface:</div><div>12:21:53.724226 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 134)</div><div>    b.b.b.b.20496 > y.y.y.y.4789: [no cksum] VXLAN, flags [I] (0x08), vni 1001</div><div>IP (tos 0x0, ttl 64, id 45715, offset 0, flags [none], proto ICMP (1), length 84)</div><div>    10.2.0.1 > <a href="http://10.2.0.200">10.2.0.200</a>: ICMP echo request, id 20365, seq 231, length 64</div><div><span class="gmail-Apple-tab-span" style="white-space:pre"> </span>0x0000:  4500 0086 0000 4000 4011 a2ec 29c1 77ea  E.....@.@...).w.</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">   </span>0x0010:  d1cb 2404 5010 12b5 0072 0000 0800 0000  ..$.P....r......</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">   </span>0x0020:  0003 e900 544b 8c51 1cad 544b 8c51 1ca9  ....TK.Q..TK.Q..</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">   </span>0x0030:  0800 4500 0054 b293 0000 4001 b349 0a02  ..E..T....@..I..</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">   </span>0x0040:  0001 0a02 00c8 0800 ae01 4f8d 00e7 5b61  ..........O...[a</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">   </span>0x0050:  f7a0 0009 bb7b 0809 0a0b 0c0d 0e0f 1011  .....{..........</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">   </span>0x0060:  1213 1415 1617 1819 1a1b 1c1d 1e1f 2021  ...............!</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">   </span>0x0070:  2223 2425 2627 2829 2a2b 2c2d 2e2f 3031  "#$%&'()*+,-./01</div><div><span class="gmail-Apple-tab-span" style="white-space:pre">      </span>0x0080:  3233 3435 3637                           234567</div><div><br></div><div>It is showing the VXLAN packet and not MPLS as there is no static entry for it as I am using labeled-unicast distribution.</div><div><br></div><div>I would appreciate any assistance as I have spend a lot of hours on this deployment and just keep on failing to get the 2 pods to talk to each other. If you can also confirm if this type of architecture is suppose to work? Below are the config files.</div><div><br></div><div>poda-leaf1 interface file:</div><div><br></div><div>auto lo</div><div>iface lo inet loopback</div><div>    address b.b.b.b/32</div><div>auto swp3</div><div>iface swp3</div><div>    address y.y.y.z/31</div><div>    mpls-enable yes</div><div>    mtu 9178</div><div><br></div><div>auto bridge</div><div>iface bridge</div><div>    bridge-ports swp13 vni1001</div><div>    bridge-pvid 1</div><div>    bridge-vids 1001</div><div>    bridge-vlan-aware yes</div><div><br></div><div>auto vlan1001</div><div>iface vlan1001</div><div>    #hwaddress 44:39:39:FF:40:94</div><div>    address <a href="http://10.2.0.150/24">10.2.0.150/24</a></div><div>    vlan-id 1001</div><div>    vlan-raw-device bridge</div><div><br></div><div>auto vni1001</div><div>iface vni1001</div><div>    bridge-access 1001</div><div>    bridge-arp-nd-suppress on</div><div>    bridge-learning off</div><div>    mstpctl-bpduguard yes</div><div>    mstpctl-portbpdufilter yes</div><div>    vxlan-id 1001</div><div>    vxlan-local-tunnelip b.b.b.b</div><div><br></div><div>poda-leaf1 frr.conf</div><div><br></div><div>router bgp 65200</div><div> bgp router-id b.b.b.b</div><div> coalesce-time 1000</div><div> bgp bestpath as-path multipath-relax</div><div> bgp bestpath compare-routerid</div><div> neighbor fabric peer-group</div><div> neighbor fabric remote-as external</div><div> neighbor fabric description Internal Fabric Network</div><div> neighbor fabric capability extended-nexthop</div><div> neighbor swp47 interface peer-group fabric</div><div> neighbor swp48 interface peer-group fabric</div><div> neighbor y.y.y.y remote-as 11111</div><div> neighbor y.y.y.y ebgp-multihop 3</div><div> !</div><div> address-family ipv4 unicast</div><div>  network b.b.b.b/32</div><div>  redistribute connected</div><div>  no neighbor y.y.y.y activate</div><div>  export vpn</div><div> exit-address-family</div><div> !</div><div> address-family ipv4 labeled-unicast</div><div>  neighbor y.y.y.y activate</div><div>  neighbor y.y.y.y route-map HigherMetric in</div><div> exit-address-family</div><div> !</div><div> address-family l2vpn evpn</div><div>  neighbor fabric activate</div><div>  neighbor y.y.y.y activate</div><div>  neighbor y.y.y.y route-map HigherMetric in</div><div>  advertise-all-vni</div><div> exit-address-family</div><div>!</div><div>route-map HigherMetric permit 10</div><div> set metric 200</div><div>!</div><div>ip route z.z.z.z/32 y.y.y.y</div><div>!</div><div>mpls label global-block 16 1000</div><div>mpls label bind b.b.b.b/32 implicit-null</div><div>mpls label bind c.c.c.c/32 102</div><div>mpls label bind x.x.x.x/32 103</div><div><br></div><div><br></div><div>podb-leaf01 interface file</div><div>auto lo</div><div>iface lo inet loopback</div><div>    address x.x.x.x/32</div><div>auto swp3</div><div>iface swp3</div><div>    address a.a.a.b/31</div><div>    mpls-enable yes</div><div>    mtu 9178</div><div>auto bridge</div><div>iface bridge</div><div>    bridge-ports swp5 swp13 vni1001</div><div>    bridge-pvid 1</div><div>    bridge-vids 1001</div><div>    bridge-vlan-aware yes</div><div><br></div><div>auto vlan1001</div><div>iface vlan1001</div><div>    address <a href="http://10.2.0.152/24">10.2.0.152/24</a></div><div>    vlan-id 1001</div><div>    vlan-raw-device bridge</div><div><br></div><div>auto vni1001</div><div>iface vni1001</div><div>    bridge-access 1001</div><div>    bridge-arp-nd-suppress on</div><div>    bridge-learning off</div><div>    mstpctl-bpduguard yes</div><div>    mstpctl-portbpdufilter yes</div><div>    vxlan-id 1001</div><div>    vxlan-local-tunnelip x.x.x.x</div><div><br></div><div>podb-leaf1 frr.conf</div><div><br></div><div>router bgp 65202</div><div> bgp router-id x.x.x.x</div><div> coalesce-time 1000</div><div> bgp bestpath as-path multipath-relax</div><div> bgp bestpath compare-routerid</div><div> neighbor fabric peer-group</div><div> neighbor fabric remote-as external</div><div> neighbor fabric description Internal Fabric Network</div><div> neighbor fabric capability extended-nexthop</div><div> neighbor swp47 interface peer-group fabric</div><div> neighbor swp48 interface peer-group fabric</div><div> neighbor a.a.a.a remote-as 11111</div><div> neighbor a.a.a.a ebgp-multihop 3</div><div> !</div><div> address-family ipv4 unicast</div><div>  network x.x.x.x/32</div><div>  redistribute connected</div><div>  no neighbor a.a.a.a activate</div><div>  export vpn</div><div> exit-address-family</div><div> !</div><div> address-family ipv4 labeled-unicast</div><div>  neighbor a.a.a.a activate</div><div> exit-address-family</div><div> !</div><div> address-family l2vpn evpn</div><div>  neighbor fabric activate</div><div>  neighbor a.a.a.a activate</div><div>  advertise-all-vni</div><div> exit-address-family</div><div>!</div><div>ip route g.g.g.g/32 a.a.a.a</div><div>!</div><div>mpls label global-block 16 1000</div><div>mpls label bind b.b.b.b/32 301</div><div>mpls label bind c.c.c.c/32 302</div><div>mpls label bind x.x.x.x/32 implicit-null</div><div><br></div></div>