Hi.
Please can someone help me, I am a bit stuck at the moment and not winning?
The image is the lab I am testing. PE1,P1,P2 and PE2 are Junos devices. The
leaf/spines are Cumulus implementations on Mellanox switches.
+-------+
+-------+
| |
| |
|--------------------> P1
|<-----------------------------------------------> P2 |
| | |
| |
| | |
| |
| +-------+
+-------+
PODA |
^
+--------------------------------------------+
|
| +-------+ |
|
| ^---------------> |<--------------^ |
|
| | y.y.y.y | PE1 | | |
|
| | |z.z.z.z| | |
|
| | | | | |
|
| | +-------+ | |
PODB |
| | | |
+--------------------------------------+
| | | | |
| |
| | +-------+ +-------+ | | |
+-------+ | |
| | | | | | | | |
| | | |
| | | Spine1| | Spine2| | | |
| Spine1| | |
| | | | | | | | + |
| | | |
| | | | | | | | |
| | | |
| | +-------+-- +-------+ | | |
+-------+ | |
| | | \-- -- | | | |
| | |
| | | \- ---/ | | | |
| | |
| | | ---/ | | | |
| | |
| | | ---/ \-- | | | |
| | |
| | v <-/ \> v | | |
v v |
| | +------------+ +-------+ | | |
+-------+ +-------+ |
| | | | | | | | |
| | | | |
| | | | |Leaf2 | | | |
|Leaf1 | | PE2 | |
| v--| Leaf1 | | |---v | |
| |<--------> | | |
| | b.b.b.b | |c.c.c.c| | |
|x.x.x.x| a.a.a.a |g.g.g.g| |
| +------------+ +-------+ | |
+-------+ +-------+ |
| <-- <- | |
-> |
| \---- \-- | |
--/ |
+--------------------------------------------+
-|-/ |
\--- \> Port2 --/
+--------------------------------------+
\---- +------------------------+</
\---- | | Port3
\-> | |
| Testing Device |
Port1| |
+------------------------+
Internally in both pods, the architecture is L3 and using eBGP by means of
unnumbered interfaces, advertising connected routes. In PODA PE1 is
connected to both leafs, over separate links with labeled-unicast enabled,
using implicit-null. The testing device is a Juniper SRX, with each
interface setup in a virtual router, but part of the same subnet. On the
switch side, the port facing the testing device is in a bridge, with a VNI
setup and the local tunnel endpoint the loopback address. I am also using
an SVI on the same subnet. Each port can reach the SVI locally. and testing
between Port1 and Port2 is successful via the uplinks to PE1, label
switching seems to be working correctly. There is reachability between the
loopbacks of all the leafs as well. It seems there is a problem with the
route-map since switching to labeled-unicast routes after testing ospf and
ldp. I still need to confirm that, in order to test the reachability over
the leaf/spine network, as it was working.
I am running into an issue with testing between Port1/Port2 and Port3. It
seems like all routes are present, but with the eBGP architecture, the
standard operation is to change the next-hop on external routes, but it
changes the remote vtep as well. The mac/ip from Port3 is advertised by
PODB leaf 1 with the VTEP as x.x.x.x, however displaying it on PODA Leaf1,
it reports as y.y.y.y, where y.y.y.y is the link local address of PE1.
root@poda-leaf1:~# net show evpn mac vni 1001
Number of MACs (local and remote) known for this VNI: 3
MAC Type Intf/Remote VTEP VLAN
54:4b:8c:51:1c:a9 local swp13 1001
54:4b:8c:51:1c:ad remote y.y.y.y
root@poda-leaf1:~# net show bgp evpn route vni 1001 mac 54:4b:8c:51:1c:ad
BGP routing table entry for [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]
Paths: (1 available, best #1)
Not advertised to any peer
Route [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad] VNI 993
Imported from x.x.x.x:2:[2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]
11111 65202
y.y.y.y from y.y.y.y (z.z.z.z)
Origin IGP, metric 200, localpref 100, valid, external,
bestpath-from-AS 11111, best
Extended Community: RT:65202:1001 ET:8
AddPath ID: RX 0, TX 56
Last update: Wed Aug 1 11:54:59 2018
I am also not understanding why the route above outputs VNI 993, it does
however seem to import correctly into 1001.
root@poda-leaf1:~# net show bgp evpn route vni 1001 vtep y.y.y.y
BGP table version is 33, local router ID is b.b.b.b
Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
*> [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]
y.y.y.y 200 0 11111 65202 i
*> [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]:[32]:[10.2.0.200]
y.y.y.y 200 0 11111 65202 i
*> [3]:[0]:[32]:[x.x.x.x]
y.y.y.y 200 0 11111 65202 i
Checking in anything is present for the loopback of podb-leaf1
root@poda-leaf1:~# net show bgp evpn route vni 1001 vtep x.x.x.x
BGP table version is 33, local router ID is b.b.b.b
Status codes: s suppressed, d damped, h history, * valid, > best, i -
internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
Displayed 9 prefixes (0 paths)
Below is the same command on the originating leaf.
root@podB-leaf-01:~# net show bgp evpn route vni 1001 mac 54:4b:8c:51:1c:ad
BGP routing table entry for [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad]
Paths: (1 available, best #1)
Not advertised to any peer
Route [2]:[0]:[0]:[48]:[54:4b:8c:51:1c:ad] VNI 1001
Local
x.x.x.x from 0.0.0.0 (x.x.x.x)
Origin IGP, localpref 100, weight 32768, valid, sourced, local,
bestpath-from-AS Local, best
Extended Community: ET:8 RT:65202:1001
AddPath ID: RX 0, TX 63
Last update: Wed Aug 1 11:51:44 2018
Spanning the uplink port to another port on the same switch, allowed me to
look at the dataplane, and it does confirm that it is sending the traffic
to the wrong destination.
tcpdump on the uplink interface:
12:21:53.724226 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto UDP
(17), length 134)
b.b.b.b.20496 > y.y.y.y.4789: [no cksum] VXLAN, flags [I] (0x08), vni
1001
IP (tos 0x0, ttl 64, id 45715, offset 0, flags [none], proto ICMP (1),
length 84)
10.2.0.1 > 10.2.0.200: ICMP echo request, id 20365, seq 231, length 64
0x0000: 4500 0086 0000 4000 4011 a2ec 29c1 77ea E.....@.@...).w.
0x0010: d1cb 2404 5010 12b5 0072 0000 0800 0000 ..$.P....r......
0x0020: 0003 e900 544b 8c51 1cad 544b 8c51 1ca9 ....TK.Q..TK.Q..
0x0030: 0800 4500 0054 b293 0000 4001 b349 0a02 ..E..T....@..I..
0x0040: 0001 0a02 00c8 0800 ae01 4f8d 00e7 5b61 ..........O...[a
0x0050: f7a0 0009 bb7b 0809 0a0b 0c0d 0e0f 1011 .....{..........
0x0060: 1213 1415 1617 1819 1a1b 1c1d 1e1f 2021 ...............!
0x0070: 2223 2425 2627 2829 2a2b 2c2d 2e2f 3031 "#$%&'()*+,-./01
0x0080: 3233 3435 3637 234567
It is showing the VXLAN packet and not MPLS as there is no static entry for
it as I am using labeled-unicast distribution.
I would appreciate any assistance as I have spend a lot of hours on this
deployment and just keep on failing to get the 2 pods to talk to each
other. If you can also confirm if this type of architecture is suppose to
work? Below are the config files.
poda-leaf1 interface file:
auto lo
iface lo inet loopback
address b.b.b.b/32
auto swp3
iface swp3
address y.y.y.z/31
mpls-enable yes
mtu 9178
auto bridge
iface bridge
bridge-ports swp13 vni1001
bridge-pvid 1
bridge-vids 1001
bridge-vlan-aware yes
auto vlan1001
iface vlan1001
#hwaddress 44:39:39:FF:40:94
address 10.2.0.150/24
vlan-id 1001
vlan-raw-device bridge
auto vni1001
iface vni1001
bridge-access 1001
bridge-arp-nd-suppress on
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
vxlan-id 1001
vxlan-local-tunnelip b.b.b.b
poda-leaf1 frr.conf
router bgp 65200
bgp router-id b.b.b.b
coalesce-time 1000
bgp bestpath as-path multipath-relax
bgp bestpath compare-routerid
neighbor fabric peer-group
neighbor fabric remote-as external
neighbor fabric description Internal Fabric Network
neighbor fabric capability extended-nexthop
neighbor swp47 interface peer-group fabric
neighbor swp48 interface peer-group fabric
neighbor y.y.y.y remote-as 11111
neighbor y.y.y.y ebgp-multihop 3
!
address-family ipv4 unicast
network b.b.b.b/32
redistribute connected
no neighbor y.y.y.y activate
export vpn
exit-address-family
!
address-family ipv4 labeled-unicast
neighbor y.y.y.y activate
neighbor y.y.y.y route-map HigherMetric in
exit-address-family
!
address-family l2vpn evpn
neighbor fabric activate
neighbor y.y.y.y activate
neighbor y.y.y.y route-map HigherMetric in
advertise-all-vni
exit-address-family
!
route-map HigherMetric permit 10
set metric 200
!
ip route z.z.z.z/32 y.y.y.y
!
mpls label global-block 16 1000
mpls label bind b.b.b.b/32 implicit-null
mpls label bind c.c.c.c/32 102
mpls label bind x.x.x.x/32 103
podb-leaf01 interface file
auto lo
iface lo inet loopback
address x.x.x.x/32
auto swp3
iface swp3
address a.a.a.b/31
mpls-enable yes
mtu 9178
auto bridge
iface bridge
bridge-ports swp5 swp13 vni1001
bridge-pvid 1
bridge-vids 1001
bridge-vlan-aware yes
auto vlan1001
iface vlan1001
address 10.2.0.152/24
vlan-id 1001
vlan-raw-device bridge
auto vni1001
iface vni1001
bridge-access 1001
bridge-arp-nd-suppress on
bridge-learning off
mstpctl-bpduguard yes
mstpctl-portbpdufilter yes
vxlan-id 1001
vxlan-local-tunnelip x.x.x.x
podb-leaf1 frr.conf
router bgp 65202
bgp router-id x.x.x.x
coalesce-time 1000
bgp bestpath as-path multipath-relax
bgp bestpath compare-routerid
neighbor fabric peer-group
neighbor fabric remote-as external
neighbor fabric description Internal Fabric Network
neighbor fabric capability extended-nexthop
neighbor swp47 interface peer-group fabric
neighbor swp48 interface peer-group fabric
neighbor a.a.a.a remote-as 11111
neighbor a.a.a.a ebgp-multihop 3
!
address-family ipv4 unicast
network x.x.x.x/32
redistribute connected
no neighbor a.a.a.a activate
export vpn
exit-address-family
!
address-family ipv4 labeled-unicast
neighbor a.a.a.a activate
exit-address-family
!
address-family l2vpn evpn
neighbor fabric activate
neighbor a.a.a.a activate
advertise-all-vni
exit-address-family
!
ip route g.g.g.g/32 a.a.a.a
!
mpls label global-block 16 1000
mpls label bind b.b.b.b/32 301
mpls label bind c.c.c.c/32 302
mpls label bind x.x.x.x/32 implicit-null
