Testing on frr version: 7.4 from https://github.com/FRRouting/frr.git

 

The network is following:

                                                        ------------                                                    ------------                                                   --------------

                                                        |               |  gre1: 22.22.22.2/32            | Switch |             22.22.22.3/32: gre1 |                |--- LO:   192.168.6.1/32

                 192.168.4.1/24:eth0 | SPOKE1|--eth1: 169.254.50.50--------|               |------ -- 169.254.50.52 :eth0| SPOKE2 |--- eth1: 192.168.5.1/24

                                                         ------------                                                    ------------                                                   ------------

                                                                                                                                   |

                                                                                                                                   |

                                                                                                                                   |

                                                                                                                  gre1: 22.22.22.1/32         

                                                                                                                  eth0: 169.254.50.51

                                                                                                                             ---------------

                                                                                                                            | UBUNTU |

                                                                                                                            |   HUB      |

                                                                                                                             ---------------

 

Issue: Spoke to Spoke communication between spoke1 and spoke2 works for gre interface IPs 22.22.22.2 and 22.22.22.3 respectively. But does not work between 192.168.5.1 (eth1 IP on Spoke2) and 192.168.4.1 (eth0 IP on Spoke1), this communication happens via the HUB.

 

When pinging 192.168.4.1 from Spoke2, the routes which get installed on Spoke2 after this are the following. As  you can see the route to reach 192.168.4.0/24 network is via 22.22.22.1. It should have been via 22.22.22.2.

 

show ip route

Codes: K - kernel route, C - connected, S - static, R - RIP,

       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,

       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,

       F - PBR, f - OpenFabric,

       > - selected route, * - FIB route, q - queued route, r - rejected route

 

K>* 0.0.0.0/0 [0/0] via 192.168.20.1, wlan0, 02:19:32

N>* 22.22.22.1/32 [10/0] is directly connected, gre1, weight 1, 02:19:26

N>* 22.22.22.2/32 [10/0] is directly connected, gre1, weight 1, 00:01:16

B   22.22.22.2/32 [200/0] via 22.22.22.1 (recursive), weight 1, 02:19:25

                            via 22.22.22.1, gre1 onlink, weight 1, 02:19:25

B   22.22.22.3/32 [200/0] via 22.22.22.1 (recursive), weight 1, 02:19:25

                            via 22.22.22.1, gre1 onlink, weight 1, 02:19:25

C>* 22.22.22.3/32 is directly connected, gre1, 02:19:32

C>* 169.254.50.0/24 is directly connected, eth0, 02:19:32

B>  192.168.4.0/24 [200/0] via 22.22.22.1 (recursive), weight 1, 02:19:25

  *                          via 22.22.22.1, gre1 onlink, weight 1, 02:19:25

C>* 192.168.5.0/24 is directly connected, eth1, 02:19:32

C>* 192.168.20.0/24 is directly connected, wlan0, 02:19:32

 

 

HUB Configuration:

----------------------------------------------------------------------------------------------------------------

sudo ip tunnel add gre1 mode gre key 42 ttl 64

sudo ip addr add 22.22.22.1/32 dev gre1

sudo ip link set gre1 up

 

sudo iptables -A FORWARD -i gre1 -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128

 

sudo iptables -A OUTPUT -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128

 

 

ThinkPad-T400# show run

Building configuration...

 

Current configuration:

!

frr version 7.4-dev-MyOwnFRRVersion

frr defaults traditional

hostname ThinkPad-T400

log syslog informational

nhrp nflog-group 1

service integrated-vtysh-config

!

interface gre1

ip nhrp network-id 1

ip nhrp redirect

tunnel source eth0

!

router bgp 65000

bgp deterministic-med

neighbor spokes-ibgp peer-group

neighbor spokes-ibgp remote-as 65000

neighbor spokes-ibgp disable-connected-check

neighbor spokes-ibgp advertisement-interval 1

neighbor 22.22.22.2 peer-group spokes-ibgp

neighbor 22.22.22.3 peer-group spokes-ibgp

!

address-family ipv4 unicast

  network 22.22.22.0/24

  redistribute nhrp

  neighbor spokes-ibgp route-reflector-client

  neighbor spokes-ibgp next-hop-self force

  neighbor spokes-ibgp soft-reconfiguration inbound

exit-address-family

!

line vty

!

end

 

SPOKE1 Configuration:

------------------------------------------------------------------------------------------------------------------

ip tunnel add gre1 mode gre key 42 ttl 64

ip addr add 22.22.22.2/32 dev gre1

ip link set gre1 up

 

iptables -A FORWARD -i gre1 -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128

 

iptables -A OUTPUT -o gre1 -m hashlimit --hashlimit-upto 4/minute --hashlimit-burst 1 --hashlimit-mode srcip,dstip --hashlimit-srcmask 24 --hashlimit-dstmask 24 --hashlimit-name loglimit-0 -j NFLOG --nflog-group 1 --nflog-size 128

 

show running-config

Building configuration...

 

Current configuration:

!

frr version 7.3

frr defaults traditional

hostname AprisaLTE

log stdout informational

log syslog

service integrated-vtysh-config

!

debug nhrp all

debug zebra events

debug zebra kernel

debug zebra nexthop

debug zebra nht

!

password zebra

!

interface gre1

ip nhrp holdtime 600

ip nhrp mtu opennhrp

ip nhrp network-id 1

ip nhrp nhs dynamic nbma 169.254.50.51

ip nhrp shortcut

tunnel source br-lan169

!

router bgp 65000

neighbor spokes-ibgp peer-group

neighbor spokes-ibgp remote-as 65000

neighbor spokes-ibgp disable-connected-check

neighbor spokes-ibgp advertisement-interval 1

neighbor 22.22.22.1 peer-group spokes-ibgp

!

address-family ipv4 unicast

  network 192.168.4.0/24

  neighbor spokes-ibgp next-hop-self

  neighbor spokes-ibgp soft-reconfiguration inbound

exit-address-family

!

access-list vty seq 10 deny any

access-list vty seq 5 permit 127.0.0.0/8

!

line vty

access-class vty

!

end

 

SPOKE2 Configuration:

------------------------------------------------------------------------------------------------------------------

ip tunnel add gre1 mode gre key 42 ttl 64

ip addr add 22.22.22.3/32 dev gre1

ip link set gre1 up

 

sudo nft list ruleset

table ip filter {

        chain input {

                type filter hook input priority 0; policy accept;

        }

 

        chain forward {

                type filter hook forward priority 0; policy accept;

                iifname "gre1" oifname "gre1" meter loglimit-1 size 65535 { ip daddr & 255.255.255.0 . ip saddr & 255.255.255.0 timeout 1m limit rate 4/minute burst 1 packets}  counter packets 0 bytes 0 log group 1 snaplen 128

        }

 

        chain output {

                type filter hook output priority 0; policy accept;

                oifname "gre1" meter loglimit-0 size 65535 { ip daddr & 255.255.255.0 . ip saddr & 255.255.255.0 timeout 1m limit rate 4/minute burst 1 packets}  counter packets 4322 bytes 291697 log group 1 snaplen 128

        }

}

 

raspberrypi# show run

Building configuration...

 

Current configuration:

!

frr version 7.4-dev-MyOwnFRRVersion

frr defaults traditional

hostname raspberrypi

log stdout informational

log syslog

nhrp nflog-group 1

service integrated-vtysh-config

!

debug nhrp all

debug zebra events

debug zebra kernel

debug zebra nexthop

debug zebra nht

!

interface gre1

ip nhrp holdtime 600

ip nhrp mtu opennhrp

ip nhrp network-id 1

ip nhrp nhs dynamic nbma 169.254.50.51

ip nhrp shortcut

tunnel source eth0

!

router bgp 65000

bgp router-id 192.168.6.1

no bgp network import-check

neighbor spokes-ibgp peer-group

neighbor spokes-ibgp remote-as 65000

neighbor spokes-ibgp disable-connected-check

neighbor spokes-ibgp advertisement-interval 1

neighbor 22.22.22.1 peer-group spokes-ibgp

!

address-family ipv4 unicast

  network 192.168.5.0/24

  network 192.168.6.1/32

  neighbor spokes-ibgp next-hop-self

  neighbor spokes-ibgp soft-reconfiguration inbound

exit-address-family

!

line vty

!

end

------------------------------------------------------------------------------------------------------

 

When pinging 192.168.4.1 from Spoke2 following logs are seen at spoke2:

 

pi@raspberrypi:~ $ tail -f /var/log/syslog | grep -v vici_reconnect

May 12 15:59:44 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0

May 12 15:59:44 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring

May 12 16:00:22 raspberrypi nhrpd[15616]: Netlink-log: Received msg_type 1024, msg_flags 0

May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Recv 169.254.50.51 -> 169.254.50.52

May 12 16:00:22 raspberrypi nhrpd[15616]: Recv Traffic-Indication(8) 22.22.22.1 -> 22.22.22.3

May 12 16:00:22 raspberrypi nhrpd[15616]: !LOCAL Traffic-Indication(8) 22.22.22.1 -> 22.22.22.3

May 12 16:00:22 raspberrypi nhrpd[15616]: Traffic Indication from 22.22.22.1 about packet to 192.168.4.1: trying shortcut

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.1/32 created

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.4.1/32: zebra route dev (none)

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.1/32: nhrp_if=gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut res_req: set cie ht to 600 and mtu to 0. shortcut ht is 0

May 12 16:00:22 raspberrypi nhrpd[15616]: Send Resolution-Request(1) 22.22.22.3 -> 192.168.4.1

May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Send 169.254.50.52 -> 169.254.50.51

May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Recv 169.254.50.51 -> 169.254.50.52

May 12 16:00:22 raspberrypi nhrpd[15616]: Recv Resolution-Request(1) 22.22.22.2 -> 22.22.22.3

May 12 16:00:22 raspberrypi nhrpd[15616]: !LOCAL Resolution-Request(1) 22.22.22.2 -> 22.22.22.3

May 12 16:00:22 raspberrypi nhrpd[15616]: Parsing and replying to Resolution Req

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.2/32: zebra route dev (none)

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.1/32: nhrp_if=gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: shortcut res_rep: paylen 0

May 12 16:00:22 raspberrypi nhrpd[15616]: Send Resolution-Reply(2) 22.22.22.3 -> 22.22.22.2

May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Send 169.254.50.52 -> 169.254.50.51

May 12 16:00:22 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=72, seq=0, pid=15615

May 12 16:00:22 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring

May 12 16:00:22 raspberrypi nhrpd[15616]: PACKET: Recv 169.254.50.51 -> 169.254.50.52

May 12 16:00:22 raspberrypi nhrpd[15616]: Recv Resolution-Reply(2) 192.168.4.1 -> 22.22.22.3

May 12 16:00:22 raspberrypi nhrpd[15616]: !LOCAL Resolution-Reply(2) 192.168.4.1 -> 22.22.22.3

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.4.1/32: zebra route dev (none)

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 22.22.22.1/32: nhrp_if=gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: 192.168.4.1/24 is at proto 22.22.22.2 dst_proto 192.168.4.1 cie-nbma 169.254.50.50 nat-nbma (unspec) cie-holdtime 600

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: cache found, update binding

May 12 16:00:22 raspberrypi nhrpd[15616]: cache: new type 0/4, or peer diff, or mtu 0/0, nbma (unspec) --> 169.254.50.50 (map 0)

May 12 16:00:22 raspberrypi nhrpd[15616]: cache: gre1 22.22.22.2: accept

May 12 16:00:22 raspberrypi nhrpd[15616]: cache: update binding for 22.22.22.2 dev gre1 from (deleted) peer.vc.nbma 169.254.50.50 to 169.254.50.50

May 12 16:00:22 raspberrypi nhrpd[15616]: cache (remote_nbma_natoa set): Update binding for 22.22.22.2 dev gre1 from (deleted) peer.vc.nbma 169.254.50.50 to 169.254.50.50

May 12 16:00:22 raspberrypi nhrpd[15616]: Zebra send: route add 22.22.22.2/32 nexthop <onlink> metric 0 count 1 dev gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.0/24 created

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: calling update_binding

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: forcing renewal of zebra announce on prefix change peer 22.22.22.2 ht 600 cur nbma 169.254.50.50 dev gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: route install 192.168.4.0/24 nh (unspec) dev gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: Zebra send: route add 192.168.4.0/24 nexthop <onlink> metric 0 count 1 dev gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut: Resolution reply handled

May 12 16:00:22 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0

May 12 16:00:22 raspberrypi nhrpd[15616]: Netlink: new-neigh 22.22.22.2 dev gre1 lladdr 169.254.50.50 nud 0x2 cache used 0 type 4

May 12 16:00:22 raspberrypi zebra[15602]: netlink_route_multipath: RTM_NEWROUTE 22.22.22.2/32 vrf 0(254)

May 12 16:00:22 raspberrypi zebra[15602]: _netlink_route_build_singlepath: (single-path): 22.22.22.2/32 nexthop via if 10 vrf default(0)

May 12 16:00:22 raspberrypi zebra[15602]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=52 seq=23 flags 0x501

May 12 16:00:22 raspberrypi zebra[15602]: netlink_route_multipath: RTM_NEWROUTE 192.168.4.0/24 vrf 0(254)

May 12 16:00:22 raspberrypi zebra[15602]: _netlink_route_build_singlepath: (single-path): 192.168.4.0/24 nexthop via if 10 vrf default(0)

May 12 16:00:22 raspberrypi zebra[15602]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=52 seq=25 flags 0x501

May 12 16:00:22 raspberrypi nhrpd[15616]: if-route-add: 22.22.22.2/32 via 0.0.0.0 dev gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: if-route-add: 192.168.4.0/24 via 0.0.0.0 dev gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.0/24 used and expiring

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.4.1/32: zebra route dev gre1

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 0.0.0.0/32: zebra route dev wlan0

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 192.168.20.1/32: zebra route dev wlan0

May 12 16:00:22 raspberrypi nhrpd[15616]: lookup 0.0.0.0/32: zebra route dev wlan0

May 12 16:00:22 raspberrypi nhrpd[15616]: Shortcut 192.168.4.1/32 purged

May 12 16:00:23 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=72, seq=0, pid=0

May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0

May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: new-neigh 22.22.22.1 dev gre1 lladdr 169.254.50.51 nud 0x4 cache used 1 type 5

May 12 16:00:23 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring

May 12 16:00:23 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_GETNEIGH(30), len=64, seq=0, pid=0

May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: Received msg_type 30, msg_flags 1

May 12 16:00:23 raspberrypi zebra[15602]: Received RTM_GETNEIGH, ignoring

May 12 16:00:23 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0

May 12 16:00:23 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=76, seq=0, pid=0

May 12 16:00:23 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring

May 12 16:00:24 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=64, seq=0, pid=0

May 12 16:00:24 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0

May 12 16:00:24 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring

May 12 16:00:24 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_GETNEIGH(30), len=64, seq=0, pid=0

May 12 16:00:24 raspberrypi zebra[15602]: Received RTM_GETNEIGH, ignoring

May 12 16:00:24 raspberrypi nhrpd[15616]: Netlink: Received msg_type 30, msg_flags 1

May 12 16:00:25 raspberrypi nhrpd[15616]: Shortcut 192.168.4.0/24 purged

May 12 16:00:25 raspberrypi nhrpd[15616]: Shortcut: notify cache down because cache?no or ri?n/a

May 12 16:00:25 raspberrypi nhrpd[15616]: Zebra send: route del 192.168.4.0/24 nexthop <onlink> metric 0 count 1 dev none

May 12 16:00:25 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0

May 12 16:00:25 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=64, seq=0, pid=0

May 12 16:00:25 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring

May 12 16:00:25 raspberrypi nhrpd[15616]: Netlink: Received msg_type 30, msg_flags 1

May 12 16:00:25 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_GETNEIGH(30), len=64, seq=0, pid=0

May 12 16:00:25 raspberrypi zebra[15602]: Received RTM_GETNEIGH, ignoring

May 12 16:00:25 raspberrypi zebra[15602]: netlink_route_multipath: RTM_NEWROUTE 192.168.4.0/24 vrf 0(254)

May 12 16:00:25 raspberrypi zebra[15602]: _netlink_route_build_singlepath: (recursive, single-path): 192.168.4.0/24 nexthop via 22.22.22.1  if 10 vrf default(0)

May 12 16:00:25 raspberrypi zebra[15602]: netlink_talk: netlink-dp (NS 0) type RTM_NEWROUTE(24), len=60 seq=27 flags 0x501

May 12 16:00:25 raspberrypi nhrpd[15616]: if-route-add: 192.168.4.0/24 via 22.22.22.1 dev (none)

May 12 16:00:26 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWNEIGH(28), len=64, seq=0, pid=0

May 12 16:00:26 raspberrypi nhrpd[15616]: Netlink: Received msg_type 28, msg_flags 0

May 12 16:00:26 raspberrypi zebra[15602]: #011Neighbor Entry received is not on a VLAN or a BRIDGE, ignoring

May 12 16:00:30 raspberrypi zebra[15602]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=72, seq=0, pid=0

-----------------------------------------------------------------------------------------------------------------------

 

 

 

 

 

 

                                                                                                                                                                                                    

The information in this email communication (inclusive of attachments) is confidential to 4RF Limited and the intended recipient(s). If you are not the intended recipient(s), please note that any use, disclosure, distribution or copying of this information or any part thereof is strictly prohibited and that the author accepts no liability for the consequences of any action taken on the basis of the information provided. If you have received this email in error, please notify the sender immediately by return email and then delete all instances of this email from your system. 4RF Limited will not accept responsibility for any consequences associated with the use of this email (including, but not limited to, damages sustained as a result of any viruses and/or any action or lack of action taken in reliance on it).