[FROG] Unable to get spoke to spoke communication for private networks behind two spoke.
Gaurav Goyal
Gaurav.Goyal at 4rf.com
Wed May 13 02:14:06 EDT 2020
Testing on frr version: 7.4 from https://github.com/FRRouting/frr.git
The network is following:
------------ ------------ --------------
| | gre1: 22.22.22.2/32 | Switch | 22.22.22.3/32: gre1 | |--- LO: 192.168.6.1/32
192.168.4.1/24:eth0 | SPOKE1|--eth1: 169.254.50.50--------| |------ -- 169.254.50.52 :eth0| SPOKE2 |--- eth1: 192.168.5.1/24
------------ ------------ ------------
|
|
|
gre1: 22.22.22.1/32
eth0: 169.254.50.51
---------------
| UBUNTU |
| HUB |
---------------
Issue: Spoke to Spoke communication between spoke1 and spoke2 works for gre interface IPs 22.22.22.2 and 22.22.22.3 respectively. But does not work between 192.168.5.1 (eth1 IP on Spoke2) and 192.168.4.1 (eth0 IP on Spoke1), this communication happens via the HUB.
When pinging 192.168.4.1 from Spoke2, the routes which get installed on Spoke2 after this are the following. As you can see the route to reach 192.168.4.0/24 network is via 22.22.22.1. It should have been via 22.22.22.2.
show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued route, r - rejected route
K>* 0.0.0.0/0 [0/0] via 192.168.20.1, wlan0, 02:19:32
N>* 22.22.22.1/32 [10/0] is directly connected, gre1, weight 1, 02:19:26
N>* 22.22.22.2/32 [10/0] is directly connected, gre1, weight 1, 00:01:16
B 22.22.22.2/32 [200/0] via 22.22.22.1 (recursive), weight 1, 02:19:25
via 22.22.22.1, gre1 onlink, weight 1, 02:19:25
B 22.22.22.3/32 [200/0] via 22.22.22.1 (recursive), weight 1, 02:19:25
via 22.22.22.1, gre1 onlink, weight 1, 02:19:25
C>* 22.22.22.3/32 is directly connected, gre1, 02:19:32
C>* 169.254.50.0/24 is directly connected, eth0, 02:19:32
B> 192.168.4.0/24 [200/0] via 22.22.22.1 (recursive), weight 1, 02:19:25
* via 22.22.22.1, gre1 onlink, weight 1, 02:19:25
C>* 192.168.5.0/24 is directly connected, eth1, 02:19:32
C>* 192.168.20.0/24 is directly connected, wlan0, 02:19:32
HUB Configuration:
----------------------------------------------------------------------------------------------------------------
sudo ip tunnel add gre1 mode gre key 42 ttl 64
sudo ip addr add 22.22.22.1/32 dev gre1
sudo ip link set gre1 up
sudo iptables -A FORWARD -i gre1 -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128
sudo iptables -A OUTPUT -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128
ThinkPad-T400# show run
Building configuration...
Current configuration:
!
frr version 7.4-dev-MyOwnFRRVersion
frr defaults traditional
hostname ThinkPad-T400
log syslog informational
nhrp nflog-group 1
service integrated-vtysh-config
!
interface gre1
ip nhrp network-id 1
ip nhrp redirect
tunnel source eth0
!
router bgp 65000
bgp deterministic-med
neighbor spokes-ibgp peer-group
neighbor spokes-ibgp remote-as 65000
neighbor spokes-ibgp disable-connected-check
neighbor spokes-ibgp advertisement-interval 1
neighbor 22.22.22.2 peer-group spokes-ibgp
neighbor 22.22.22.3 peer-group spokes-ibgp
!
address-family ipv4 unicast
network 22.22.22.0/24
redistribute nhrp
neighbor spokes-ibgp route-reflector-client
neighbor spokes-ibgp next-hop-self force
neighbor spokes-ibgp soft-reconfiguration inbound
exit-address-family
!
line vty
!
end
SPOKE1 Configuration:
------------------------------------------------------------------------------------------------------------------
ip tunnel add gre1 mode gre key 42 ttl 64
ip addr add 22.22.22.2/32 dev gre1
ip link set gre1 up
iptables -A FORWARD -i gre1 -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128
iptables -A OUTPUT -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128
show running-config
Building configuration...
Current configuration:
!
frr version 7.3
frr defaults traditional
hostname AprisaLTE
log stdout informational
log syslog
service integrated-vtysh-config
!
debug nhrp all
debug zebra events
debug zebra kernel
debug zebra nexthop
debug zebra nht
!
password zebra
!
interface gre1
ip nhrp holdtime 600
ip nhrp mtu opennhrp
ip nhrp network-id 1
ip nhrp nhs dynamic nbma 169.254.50.51
ip nhrp shortcut
tunnel source br-lan169
!
router bgp 65000
neighbor spokes-ibgp peer-group
neighbor spokes-ibgp remote-as 65000
neighbor spokes-ibgp disable-connected-check
neighbor spokes-ibgp advertisement-interval 1
neighbor 22.22.22.1 peer-group spokes-ibgp
!
address-family ipv4 unicast
network 192.168.4.0/24
neighbor spokes-ibgp next-hop-self
neighbor spokes-ibgp soft-reconfiguration inbound
exit-address-family
!
access-list vty seq 10 deny any
access-list vty seq 5 permit 127.0.0.0/8
!
line vty
access-class vty
!
end
SPOKE2 Configuration:
------------------------------------------------------------------------------------------------------------------
ip tunnel add gre1 mode gre key 42 ttl 64
ip addr add 22.22.22.3/32 dev gre1
ip link set gre1 up
sudo nft list ruleset
table ip filter {
chain input {
type filter hook input priority 0; policy accept;
}
chain forward {
type filter hook forward priority 0; policy accept;
iifname "gre1" oifname "gre1" meter loglimit-1 size 65535 { ip daddr & 255.255.255.0 . ip saddr & 255.255.255.0 timeout 1m limit rate 4/minute burst 1 packets} counter packets 0 bytes 0 log group 1 snaplen 128
}
chain output {
type filter hook output priority 0; policy accept;
oifname "gre1" meter loglimit-0 size 65535 { ip daddr & 255.255.255.0 . ip saddr & 255.255.255.0 timeout 1m limit rate 4/minute burst 1 packets} counter packets 4322 bytes 291697 log group 1 snaplen 128
}
}
raspberrypi# show run
Building configuration...
Current configuration:
!
frr version 7.4-dev-MyOwnFRRVersion
frr defaults traditional
hostname raspberrypi
log stdout informational
log syslog
nhrp nflog-group 1
service integrated-vtysh-config
!
debug nhrp all
debug zebra events
debug zebra kernel
debug zebra nexthop
debug zebra nht
!
interface gre1
ip nhrp holdtime 600
ip nhrp mtu opennhrp
ip nhrp network-id 1
ip nhrp nhs dynamic nbma 169.254.50.51
ip nhrp shortcut
tunnel source eth0
!
router bgp 65000
bgp router-id 192.168.6.1
no bgp network import-check
neighbor spokes-ibgp peer-group
neighbor spokes-ibgp remote-as 65000
neighbor spokes-ibgp disable-connected-check
neighbor spokes-ibgp advertisement-interval 1
neighbor 22.22.22.1 peer-group spokes-ibgp
!
address-family ipv4 unicast
network 192.168.5.0/24
network 192.168.6.1/32
neighbor spokes-ibgp next-hop-self
neighbor spokes-ibgp soft-reconfiguration inbound
exit-address-family
!
line vty
!
end
------------------------------------------------------------------------------------------------------
When pinging 192.168.4.1 from Spoke2 following logs are seen at spoke2:
pi at raspberrypi:~ $ tail -f /var/log/syslog | grep -v vici_reconnect
May 12 15:59:44 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
May 12 15:59:44 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
May 12 16:00:22 raspberrypi nhrpd[15616]: Netlink-log: Received msg_type 1024, msg_flags 0
May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Recv 169.254.50.51 -> 169.254.50.52
May 12 16:00:22 raspberrypi nhrpd[15616]: Recv Traffic-Indication(8) 22.22.22.1 -> 22.22.22.3
May 12 16:00:22 raspberrypi nhrpd[15616]: !LOCAL Traffic-Indication(8) 22.22.22.1 -> 22.22.22.3
May 12 16:00:22 raspberrypi nhrpd[15616]: Traffic Indication from 22.22.22.1 about packet to 192.168.4.1: trying shortcut
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.1/32 created
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.4.1/32: zebra route dev (none)
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.1/32: nhrp_if=gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut res_req: set cie ht to 600 and mtu to 0. shortcut ht is 0
May 12 16:00:22 raspberrypi nhrpd[15616]: Send Resolution-Request(1) 22.22.22.3 -> 192.168.4.1
May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Send 169.254.50.52 -> 169.254.50.51
May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Recv 169.254.50.51 -> 169.254.50.52
May 12 16:00:22 raspberrypi nhrpd[15616]: Recv Resolution-Request(1) 22.22.22.2 -> 22.22.22.3
May 12 16:00:22 raspberrypi nhrpd[15616]: !LOCAL Resolution-Request(1) 22.22.22.2 -> 22.22.22.3
May 12 16:00:22 raspberrypi nhrpd[15616]: Parsing and replying to Resolution Req
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.2/32: zebra route dev (none)
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.1/32: nhrp_if=gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: shortcut res_rep: paylen 0
May 12 16:00:22 raspberrypi nhrpd[15616]: Send Resolution-Reply(2) 22.22.22.3 -> 22.22.22.2
May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Send 169.254.50.52 -> 169.254.50.51
May 12 16:00:22 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=72, seq=0, pid=15615
May 12 16:00:22 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Recv 169.254.50.51 -> 169.254.50.52
May 12 16:00:22 raspberrypi nhrpd[15616]: Recv Resolution-Reply(2) 192.168.4.1 -> 22.22.22.3
May 12 16:00:22 raspberrypi nhrpd[15616]: !LOCAL Resolution-Reply(2) 192.168.4.1 -> 22.22.22.3
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.4.1/32: zebra route dev (none)
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.1/32: nhrp_if=gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: 192.168.4.1/24 is at proto 22.22.22.2 dst_proto 192.168.4.1 cie-nbma 169.254.50.50 nat-nbma (unspec) cie-holdtime 600
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: cache found, update binding
May 12 16:00:22 raspberrypi nhrpd[15616]: cache: new type 0/4, or peer diff, or mtu 0/0, nbma (unspec) --> 169.254.50.50 (map 0)
May 12 16:00:22 raspberrypi nhrpd[15616]: cache: gre1 22.22.22.2: accept
May 12 16:00:22 raspberrypi nhrpd[15616]: cache: update binding for 22.22.22.2 dev gre1 from (deleted) peer.vc.nbma 169.254.50.50 to 169.254.50.50
May 12 16:00:22 raspberrypi nhrpd[15616]: cache (remote_nbma_natoa set): Update binding for 22.22.22.2 dev gre1 from (deleted) peer.vc.nbma 169.254.50.50 to 169.254.50.50
May 12 16:00:22 raspberrypi nhrpd[15616]: Zebra send: route add 22.22.22.2/32 nexthop <onlink> metric 0 count 1 dev gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.0/24 created
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: calling update_binding
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: forcing renewal of zebra announce on prefix change peer 22.22.22.2 ht 600 cur nbma 169.254.50.50 dev gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: route install 192.168.4.0/24 nh (unspec) dev gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: Zebra send: route add 192.168.4.0/24 nexthop <onlink> metric 0 count 1 dev gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: Resolution reply handled
May 12 16:00:22 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0
May 12 16:00:22 raspberrypi nhrpd[15616]: Netlink: new-neigh 22.22.22.2 dev gre1 lladdr 169.254.50.50 nud 0x2 cache used 0 type 4
May 12 16:00:22 raspberrypi zebra[15602]: netlink_route_multipath: RTM_NEWROUTE 22.22.22.2/32 vrf 0(254)
May 12 16:00:22 raspberrypi zebra[15602]: _netlink_route_build_singlepath: (single-path): 22.22.22.2/32 nexthop via if 10 vrf default(0)
May 12 16:00:22 raspberrypi zebra[15602]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=52 seq=23 flags 0x501
May 12 16:00:22 raspberrypi zebra[15602]: netlink_route_multipath: RTM_NEWROUTE 192.168.4.0/24 vrf 0(254)
May 12 16:00:22 raspberrypi zebra[15602]: _netlink_route_build_singlepath: (single-path): 192.168.4.0/24 nexthop via if 10 vrf default(0)
May 12 16:00:22 raspberrypi zebra[15602]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=52 seq=25 flags 0x501
May 12 16:00:22 raspberrypi nhrpd[15616]: if-route-add: 22.22.22.2/32 via 0.0.0.0 dev gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: if-route-add: 192.168.4.0/24 via 0.0.0.0 dev gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.0/24 used and expiring
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.4.1/32: zebra route dev gre1
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 0.0.0.0/32: zebra route dev wlan0
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.20.1/32: zebra route dev wlan0
May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 0.0.0.0/32: zebra route dev wlan0
May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.1/32 purged
May 12 16:00:23 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=72, seq=0, pid=0
May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0
May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: new-neigh 22.22.22.1 dev gre1 lladdr 169.254.50.51 nud 0x4 cache used 1 type 5
May 12 16:00:23 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
May 12 16:00:23 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_GETNEIGH(30), len=64, seq=0, pid=0
May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: Received msg_type 30, msg_flags 1
May 12 16:00:23 raspberrypi zebra[15602]: Received RTM_GETNEIGH, ignoring
May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0
May 12 16:00:23 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0
May 12 16:00:23 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
May 12 16:00:24 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=64, seq=0, pid=0
May 12 16:00:24 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0
May 12 16:00:24 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
May 12 16:00:24 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_GETNEIGH(30), len=64, seq=0, pid=0
May 12 16:00:24 raspberrypi zebra[15602]: Received RTM_GETNEIGH, ignoring
May 12 16:00:24 raspberrypi nhrpd[15616]: Netlink: Received msg_type 30, msg_flags 1
May 12 16:00:25 raspberrypi nhrpd[15616]: Shortcut 192.168.4.0/24 purged
May 12 16:00:25 raspberrypi nhrpd[15616]: Shortcut: notify cache down because cache?no or ri?n/a
May 12 16:00:25 raspberrypi nhrpd[15616]: Zebra send: route del 192.168.4.0/24 nexthop <onlink> metric 0 count 1 dev none
May 12 16:00:25 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0
May 12 16:00:25 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=64, seq=0, pid=0
May 12 16:00:25 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
May 12 16:00:25 raspberrypi nhrpd[15616]: Netlink: Received msg_type 30, msg_flags 1
May 12 16:00:25 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_GETNEIGH(30), len=64, seq=0, pid=0
May 12 16:00:25 raspberrypi zebra[15602]: Received RTM_GETNEIGH, ignoring
May 12 16:00:25 raspberrypi zebra[15602]: netlink_route_multipath: RTM_NEWROUTE 192.168.4.0/24 vrf 0(254)
May 12 16:00:25 raspberrypi zebra[15602]: _netlink_route_build_singlepath: (recursive, single-path): 192.168.4.0/24 nexthop via 22.22.22.1 if 10 vrf default(0)
May 12 16:00:25 raspberrypi zebra[15602]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=60 seq=27 flags 0x501
May 12 16:00:25 raspberrypi nhrpd[15616]: if-route-add: 192.168.4.0/24 via 22.22.22.1 dev (none)
May 12 16:00:26 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=64, seq=0, pid=0
May 12 16:00:26 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0
May 12 16:00:26 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring
May 12 16:00:30 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0
-----------------------------------------------------------------------------------------------------------------------
________________________________
The information in this email communication (inclusive of attachments) is confidential to 4RF Limited and the intended recipient(s). If you are not the intended recipient(s), please note that any use, disclosure, distribution or copying of this information or any part thereof is strictly prohibited and that the author accepts no liability for the consequences of any action taken on the basis of the information provided. If you have received this email in error, please notify the sender immediately by return email and then delete all instances of this email from your system. 4RF Limited will not accept responsibility for any consequences associated with the use of this email (including, but not limited to, damages sustained as a result of any viruses and/or any action or lack of action taken in reliance on it).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.frrouting.org/pipermail/frog/attachments/20200513/0d9cd9d3/attachment-0001.htm>
More information about the frog
mailing list