Diagram of network attached since it is not getting correctly displayed: [cid:image002.jpg@01D62AB2.42A4F320] From: Gaurav Goyal Sent: Wednesday, 13 May 2020 6:14 pm To: 'frog@lists.frrouting.org' <frog@lists.frrouting.org<mailto:frog@lists.frrouting.org>> Subject: Unable to get spoke to spoke communication for private networks behind two spoke. Testing on frr version: 7.4 from https://github.com/FRRouting/frr.git The network is following: ------------ ------------ -------------- | | gre1: 22.22.22.2/32 | Switch | 22.22.22.3/32: gre1 | |--- LO: 192.168.6.1/32 192.168.4.1/24:eth0 | SPOKE1|--eth1: 169.254.50.50--------| |------ -- 169.254.50.52 :eth0| SPOKE2 |--- eth1: 192.168.5.1/24 ------------ ------------ ------------ | | | gre1: 22.22.22.1/32 eth0: 169.254.50.51 --------------- | UBUNTU | | HUB | --------------- Issue: Spoke to Spoke communication between spoke1 and spoke2 works for gre interface IPs 22.22.22.2 and 22.22.22.3 respectively. But does not work between 192.168.5.1 (eth1 IP on Spoke2) and 192.168.4.1 (eth0 IP on Spoke1), this communication happens via the HUB. When pinging 192.168.4.1 from Spoke2, the routes which get installed on Spoke2 after this are the following. As you can see the route to reach 192.168.4.0/24 network is via 22.22.22.1. It should have been via 22.22.22.2. show ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued route, r - rejected route K>* 0.0.0.0/0 [0/0] via 192.168.20.1, wlan0, 02:19:32 N>* 22.22.22.1/32 [10/0] is directly connected, gre1, weight 1, 02:19:26 N>* 22.22.22.2/32 [10/0] is directly connected, gre1, weight 1, 00:01:16 B 22.22.22.2/32 [200/0] via 22.22.22.1 (recursive), weight 1, 02:19:25 via 22.22.22.1, gre1 onlink, weight 1, 02:19:25 B 22.22.22.3/32 [200/0] via 22.22.22.1 (recursive), weight 1, 02:19:25 via 22.22.22.1, gre1 onlink, weight 1, 02:19:25 C>* 22.22.22.3/32 is directly connected, gre1, 02:19:32 C>* 169.254.50.0/24 is directly connected, eth0, 02:19:32 B> 192.168.4.0/24 [200/0] via 22.22.22.1 (recursive), weight 1, 02:19:25 * via 22.22.22.1, gre1 onlink, weight 1, 02:19:25 C>* 192.168.5.0/24 is directly connected, eth1, 02:19:32 C>* 192.168.20.0/24 is directly connected, wlan0, 02:19:32 HUB Configuration: ---------------------------------------------------------------------------------------------------------------- sudo ip tunnel add gre1 mode gre key 42 ttl 64 sudo ip addr add 22.22.22.1/32 dev gre1 sudo ip link set gre1 up sudo iptables -A FORWARD -i gre1 -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128 sudo iptables -A OUTPUT -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128 ThinkPad-T400# show run Building configuration... Current configuration: ! frr version 7.4-dev-MyOwnFRRVersion frr defaults traditional hostname ThinkPad-T400 log syslog informational nhrp nflog-group 1 service integrated-vtysh-config ! interface gre1 ip nhrp network-id 1 ip nhrp redirect tunnel source eth0 ! router bgp 65000 bgp deterministic-med neighbor spokes-ibgp peer-group neighbor spokes-ibgp remote-as 65000 neighbor spokes-ibgp disable-connected-check neighbor spokes-ibgp advertisement-interval 1 neighbor 22.22.22.2 peer-group spokes-ibgp neighbor 22.22.22.3 peer-group spokes-ibgp ! address-family ipv4 unicast network 22.22.22.0/24 redistribute nhrp neighbor spokes-ibgp route-reflector-client neighbor spokes-ibgp next-hop-self force neighbor spokes-ibgp soft-reconfiguration inbound exit-address-family ! line vty ! end SPOKE1 Configuration: ------------------------------------------------------------------------------------------------------------------ ip tunnel add gre1 mode gre key 42 ttl 64 ip addr add 22.22.22.2/32 dev gre1 ip link set gre1 up iptables -A FORWARD -i gre1 -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128 iptables -A OUTPUT -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128 show running-config Building configuration... Current configuration: ! frr version 7.3 frr defaults traditional hostname AprisaLTE log stdout informational log syslog service integrated-vtysh-config ! debug nhrp all debug zebra events debug zebra kernel debug zebra nexthop debug zebra nht ! password zebra ! interface gre1 ip nhrp holdtime 600 ip nhrp mtu opennhrp ip nhrp network-id 1 ip nhrp nhs dynamic nbma 169.254.50.51 ip nhrp shortcut tunnel source br-lan169 ! router bgp 65000 neighbor spokes-ibgp peer-group neighbor spokes-ibgp remote-as 65000 neighbor spokes-ibgp disable-connected-check neighbor spokes-ibgp advertisement-interval 1 neighbor 22.22.22.1 peer-group spokes-ibgp ! address-family ipv4 unicast network 192.168.4.0/24 neighbor spokes-ibgp next-hop-self neighbor spokes-ibgp soft-reconfiguration inbound exit-address-family ! access-list vty seq 10 deny any access-list vty seq 5 permit 127.0.0.0/8 ! line vty access-class vty ! end SPOKE2 Configuration: ------------------------------------------------------------------------------------------------------------------ ip tunnel add gre1 mode gre key 42 ttl 64 ip addr add 22.22.22.3/32 dev gre1 ip link set gre1 up sudo nft list ruleset table ip filter { chain input { type filter hook input priority 0; policy accept; } chain forward { type filter hook forward priority 0; policy accept; iifname "gre1" oifname "gre1" meter loglimit-1 size 65535 { ip daddr & 255.255.255.0 . ip saddr & 255.255.255.0 timeout 1m limit rate 4/minute burst 1 packets} counter packets 0 bytes 0 log group 1 snaplen 128 } chain output { type filter hook output priority 0; policy accept; oifname "gre1" meter loglimit-0 size 65535 { ip daddr & 255.255.255.0 . ip saddr & 255.255.255.0 timeout 1m limit rate 4/minute burst 1 packets} counter packets 4322 bytes 291697 log group 1 snaplen 128 } } raspberrypi# show run Building configuration... Current configuration: ! frr version 7.4-dev-MyOwnFRRVersion frr defaults traditional hostname raspberrypi log stdout informational log syslog nhrp nflog-group 1 service integrated-vtysh-config ! debug nhrp all debug zebra events debug zebra kernel debug zebra nexthop debug zebra nht ! interface gre1 ip nhrp holdtime 600 ip nhrp mtu opennhrp ip nhrp network-id 1 ip nhrp nhs dynamic nbma 169.254.50.51 ip nhrp shortcut tunnel source eth0 ! router bgp 65000 bgp router-id 192.168.6.1 no bgp network import-check neighbor spokes-ibgp peer-group neighbor spokes-ibgp remote-as 65000 neighbor spokes-ibgp disable-connected-check neighbor spokes-ibgp advertisement-interval 1 neighbor 22.22.22.1 peer-group spokes-ibgp ! address-family ipv4 unicast network 192.168.5.0/24 network 192.168.6.1/32 neighbor spokes-ibgp next-hop-self neighbor spokes-ibgp soft-reconfiguration inbound exit-address-family ! line vty ! end ------------------------------------------------------------------------------------------------------ When pinging 192.168.4.1 from Spoke2 following logs are seen at spoke2: pi@raspberrypi:~ $ tail -f /var/log/syslog | grep -v vici_reconnect May 12 15:59:44 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0 May 12 15:59:44 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring May 12 16:00:22 raspberrypi nhrpd[15616]: Netlink-log: Received msg_type 1024, msg_flags 0 May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Recv 169.254.50.51 -> 169.254.50.52 May 12 16:00:22 raspberrypi nhrpd[15616]: Recv Traffic-Indication(8) 22.22.22.1 -> 22.22.22.3 May 12 16:00:22 raspberrypi nhrpd[15616]: !LOCAL Traffic-Indication(8) 22.22.22.1 -> 22.22.22.3 May 12 16:00:22 raspberrypi nhrpd[15616]: Traffic Indication from 22.22.22.1 about packet to 192.168.4.1: trying shortcut May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.1/32 created May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.4.1/32: zebra route dev (none) May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.1/32: nhrp_if=gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut res_req: set cie ht to 600 and mtu to 0. shortcut ht is 0 May 12 16:00:22 raspberrypi nhrpd[15616]: Send Resolution-Request(1) 22.22.22.3 -> 192.168.4.1 May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Send 169.254.50.52 -> 169.254.50.51 May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Recv 169.254.50.51 -> 169.254.50.52 May 12 16:00:22 raspberrypi nhrpd[15616]: Recv Resolution-Request(1) 22.22.22.2 -> 22.22.22.3 May 12 16:00:22 raspberrypi nhrpd[15616]: !LOCAL Resolution-Request(1) 22.22.22.2 -> 22.22.22.3 May 12 16:00:22 raspberrypi nhrpd[15616]: Parsing and replying to Resolution Req May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.2/32: zebra route dev (none) May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.1/32: nhrp_if=gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: shortcut res_rep: paylen 0 May 12 16:00:22 raspberrypi nhrpd[15616]: Send Resolution-Reply(2) 22.22.22.3 -> 22.22.22.2 May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Send 169.254.50.52 -> 169.254.50.51 May 12 16:00:22 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=72, seq=0, pid=15615 May 12 16:00:22 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Recv 169.254.50.51 -> 169.254.50.52 May 12 16:00:22 raspberrypi nhrpd[15616]: Recv Resolution-Reply(2) 192.168.4.1 -> 22.22.22.3 May 12 16:00:22 raspberrypi nhrpd[15616]: !LOCAL Resolution-Reply(2) 192.168.4.1 -> 22.22.22.3 May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.4.1/32: zebra route dev (none) May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.1/32: nhrp_if=gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: 192.168.4.1/24 is at proto 22.22.22.2 dst_proto 192.168.4.1 cie-nbma 169.254.50.50 nat-nbma (unspec) cie-holdtime 600 May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: cache found, update binding May 12 16:00:22 raspberrypi nhrpd[15616]: cache: new type 0/4, or peer diff, or mtu 0/0, nbma (unspec) --> 169.254.50.50 (map 0) May 12 16:00:22 raspberrypi nhrpd[15616]: cache: gre1 22.22.22.2: accept May 12 16:00:22 raspberrypi nhrpd[15616]: cache: update binding for 22.22.22.2 dev gre1 from (deleted) peer.vc.nbma 169.254.50.50 to 169.254.50.50 May 12 16:00:22 raspberrypi nhrpd[15616]: cache (remote_nbma_natoa set): Update binding for 22.22.22.2 dev gre1 from (deleted) peer.vc.nbma 169.254.50.50 to 169.254.50.50 May 12 16:00:22 raspberrypi nhrpd[15616]: Zebra send: route add 22.22.22.2/32 nexthop <onlink> metric 0 count 1 dev gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.0/24 created May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: calling update_binding May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: forcing renewal of zebra announce on prefix change peer 22.22.22.2 ht 600 cur nbma 169.254.50.50 dev gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: route install 192.168.4.0/24 nh (unspec) dev gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: Zebra send: route add 192.168.4.0/24 nexthop <onlink> metric 0 count 1 dev gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: Resolution reply handled May 12 16:00:22 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0 May 12 16:00:22 raspberrypi nhrpd[15616]: Netlink: new-neigh 22.22.22.2 dev gre1 lladdr 169.254.50.50 nud 0x2 cache used 0 type 4 May 12 16:00:22 raspberrypi zebra[15602]: netlink_route_multipath: RTM_NEWROUTE 22.22.22.2/32 vrf 0(254) May 12 16:00:22 raspberrypi zebra[15602]: _netlink_route_build_singlepath: (single-path): 22.22.22.2/32 nexthop via if 10 vrf default(0) May 12 16:00:22 raspberrypi zebra[15602]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=52 seq=23 flags 0x501 May 12 16:00:22 raspberrypi zebra[15602]: netlink_route_multipath: RTM_NEWROUTE 192.168.4.0/24 vrf 0(254) May 12 16:00:22 raspberrypi zebra[15602]: _netlink_route_build_singlepath: (single-path): 192.168.4.0/24 nexthop via if 10 vrf default(0) May 12 16:00:22 raspberrypi zebra[15602]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=52 seq=25 flags 0x501 May 12 16:00:22 raspberrypi nhrpd[15616]: if-route-add: 22.22.22.2/32 via 0.0.0.0 dev gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: if-route-add: 192.168.4.0/24 via 0.0.0.0 dev gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.0/24 used and expiring May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.4.1/32: zebra route dev gre1 May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 0.0.0.0/32: zebra route dev wlan0 May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.20.1/32: zebra route dev wlan0 May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 0.0.0.0/32: zebra route dev wlan0 May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.1/32 purged May 12 16:00:23 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=72, seq=0, pid=0 May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0 May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: new-neigh 22.22.22.1 dev gre1 lladdr 169.254.50.51 nud 0x4 cache used 1 type 5 May 12 16:00:23 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring May 12 16:00:23 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_GETNEIGH(30), len=64, seq=0, pid=0 May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: Received msg_type 30, msg_flags 1 May 12 16:00:23 raspberrypi zebra[15602]: Received RTM_GETNEIGH, ignoring May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0 May 12 16:00:23 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0 May 12 16:00:23 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring May 12 16:00:24 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=64, seq=0, pid=0 May 12 16:00:24 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0 May 12 16:00:24 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring May 12 16:00:24 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_GETNEIGH(30), len=64, seq=0, pid=0 May 12 16:00:24 raspberrypi zebra[15602]: Received RTM_GETNEIGH, ignoring May 12 16:00:24 raspberrypi nhrpd[15616]: Netlink: Received msg_type 30, msg_flags 1 May 12 16:00:25 raspberrypi nhrpd[15616]: Shortcut 192.168.4.0/24 purged May 12 16:00:25 raspberrypi nhrpd[15616]: Shortcut: notify cache down because cache?no or ri?n/a May 12 16:00:25 raspberrypi nhrpd[15616]: Zebra send: route del 192.168.4.0/24 nexthop <onlink> metric 0 count 1 dev none May 12 16:00:25 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0 May 12 16:00:25 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=64, seq=0, pid=0 May 12 16:00:25 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring May 12 16:00:25 raspberrypi nhrpd[15616]: Netlink: Received msg_type 30, msg_flags 1 May 12 16:00:25 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_GETNEIGH(30), len=64, seq=0, pid=0 May 12 16:00:25 raspberrypi zebra[15602]: Received RTM_GETNEIGH, ignoring May 12 16:00:25 raspberrypi zebra[15602]: netlink_route_multipath: RTM_NEWROUTE 192.168.4.0/24 vrf 0(254) May 12 16:00:25 raspberrypi zebra[15602]: _netlink_route_build_singlepath: (recursive, single-path): 192.168.4.0/24 nexthop via 22.22.22.1 if 10 vrf default(0) May 12 16:00:25 raspberrypi zebra[15602]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=60 seq=27 flags 0x501 May 12 16:00:25 raspberrypi nhrpd[15616]: if-route-add: 192.168.4.0/24 via 22.22.22.1 dev (none) May 12 16:00:26 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=64, seq=0, pid=0 May 12 16:00:26 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0 May 12 16:00:26 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring May 12 16:00:30 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0 ----------------------------------------------------------------------------------------------------------------------- ________________________________ The information in this email communication (inclusive of attachments) is confidential to 4RF Limited and the intended recipient(s). If you are not the intended recipient(s), please note that any use, disclosure, distribution or copying of this information or any part thereof is strictly prohibited and that the author accepts no liability for the consequences of any action taken on the basis of the information provided. If you have received this email in error, please notify the sender immediately by return email and then delete all instances of this email from your system. 4RF Limited will not accept responsibility for any consequences associated with the use of this email (including, but not limited to, damages sustained as a result of any viruses and/or any action or lack of action taken in reliance on it).