[FROG] MAC Flapping With VRRP on Linux

Sun May 19 20:50:12 UTC 2024

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

VRRP is configured in FRR, as per the manual. Interface addressing is applied by
systemd-networkd.

Except for priority, the configuration is the same on both routers. VRRP
itself works fine and fails over as expected.

interface bond0
 description Bond to core switches
 vrrp 1
 vrrp 1 priority 110
 vrrp 1 ip 192.168.1.1
exit

The problem is Linux's behaviour replying to ARP who-has. Given that FRR's
VRRP implementation only works on Linux I'm assuming the correct combination
of sysctls to make ARP behave is known, but it's not in the manual that I can
see (yes, I checked the sysctl section too).

Keepalived doesn't use the same virtual MAC mechanism as FRR, and I want to
avoid it if at all possible. It does avoid this specific ARP problem but
it's a lot harder to inspect the state of than running "show vrrp" in vtysh
and I explicitly want an RFC compliant VRRP that uses a VRRP MAC. I also want
to keep all of the configuration in one place, which FRR does.

Hendrik Visage <hvjunk at gmail.com> writes:

> What are you using for doing VRRP, and the configurations you've setup?
>
> I've been using keepalived to provide VRRP VIPs without this issue
> before, so need more information on the configs etc.
>
> On Sun, May 19, 2024 at 12:33 PM Alasdair Muckart via frog
> <frog at lists.frrouting.org> wrote:
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Alasdair Muckart <alasdairmuckart at catalyst.net.nz>
>> To: frog at lists.frrouting.org
>> Cc:
>> Bcc:
>> Date: Sun, 19 May 2024 21:36:22 +1200
>> Subject: MAC Flapping With VRRP on Linux
>> Hello FRR folk,
>>
>> I'm having difficulty with traffic to linux (Ubuntu 22.04, kernel 5.15)
>> routers running VRRP.  The problem is MAC flapping between the VRRP MAC
>> and the underlying interface MAC. It's so bad traffic from the LAN to
>> the VIP is basically unusable. Every who-has for the VIP or the VRRP
>> primary's underlying interface IP gets multiple responses, and the mac
>> table on the switches is flailing.
>>
>> I've tried all the combinations of the various arp sysctl I can think of
>> and I can't get one that will only respond to requests for the VIP with
>> the VRRP MAC. Either I get duelling replies with both the VIP MAC and
>> the underlying interface MAC, or I get nothing at all.
>>
>> Can anyone tell me what I need to do to get the routers to only reply
>> with the VIP MAC when there's an arp who-has for the VIP?  I couldn't
>> see anything in the manual about this.
>>
>> TIA.
>>
>>
>> In case it's relevant, the topology as follows:
>>
>> A pair of core switches connected by an ERPS ring.
>>
>> Two routers, each connected to both switches with an active/passive bond
>> interface.
>>
>> VRRP running on the bond interface. The bond interfaces are .2 and .3,
>> the VIP is .1.
>>
>> The eth0 and eth1 interfaces are unnumbered children of the bond0.
>>
>>                  192.168.1.2/24
>>
>>
>>                 eth0       eth1
>>         +-----------+bond0+-X----------+
>>         |           |     |            |
>>         |        +--+-----+--+         |
>>         |        | vrrp4-1-1 |         |
>>         |        +-----------+         |
>>         |        192.168.1.1/24        |
>>         |                              |
>> +------+-----+                 +------+-----+
>> |            +-----------------+            |
>> |  SWITCH 1  |       ERPS      |  SWITCH 2  |
>> |            +---------------X-+            |
>> +------+-----+                 +------+-----+
>>         |                              |
>>         |                              |
>>         |        +-----------+         |
>>         |        | vrrp4-1-1 |         |
>>         |        +--+-----+--+         |
>>         |           |     |            |
>>         +-----------+bond0+-X----------+
>>                 eth0       eth1
>>
>>                  192.168.1.3/24
>>
>> I've got the bond0 interfaces because FRR doesn't seem to cope at all
>> with having two interfaces in the same VRRP on the same router, one of
>> them is permanently stuck 'initializing', but the MAC flapping is the
>> same with just one interface and no bond.
>>
>>
>>
>> --
>> Alasdair Muckart (he/him)
>> Network Infrastructure Architect
>> Catalyst.Net Limited - Expert Open Source Solutions
>>
>> Catalyst.Net Ltd - a Catalyst IT group company
>> DDI: +64 4 897 7794 | Mobile: +64 22 638 5141 | Tel: +64 4 499 2267 | www.catalyst.net.nz
>>
>> CONFIDENTIALITY NOTICE: This email is intended for the named
>> recipients only. It may contain privileged, confidential or copyright
>> information. If you are not the named recipient, any use, reliance
>> upon, disclosure or copying of this email or its attachments is
>> unauthorised. If you have received this email in error, please reply
>> via email or call +64 4 499 2267.
>>
>>
>>
>>
>> ---------- Forwarded message ----------
>> From: Alasdair Muckart via frog <frog at lists.frrouting.org>
>> To: frog at lists.frrouting.org
>> Cc:
>> Bcc:
>> Date: Sun, 19 May 2024 21:36:22 +1200
>> Subject: [FROG] MAC Flapping With VRRP on Linux
>> _______________________________________________
>> frog mailing list
>> frog at lists.frrouting.org
>> https://lists.frrouting.org/listinfo/frog

- -- 
Alasdair Muckart (he/him)
Network Infrastructure Architect
Catalyst.Net Limited - Expert Open Source Solutions

Catalyst.Net Ltd - a Catalyst IT group company
DDI: +64 4 897 7794 | Mobile: +64 22 638 5141 | Tel: +64 4 499 2267 | www.catalyst.net.nz

CONFIDENTIALITY NOTICE: This email is intended for the named
recipients only. It may contain privileged, confidential or copyright
information. If you are not the named recipient, any use, reliance
upon, disclosure or copying of this email or its attachments is
unauthorised. If you have received this email in error, please reply
via email or call +64 4 499 2267.
-----BEGIN PGP SIGNATURE-----

iQJUBAEBCgA+FiEEu4g3jwJ68cPCdgH9iBAgH4ERwwMFAmZKaJEgHGFsYXNkYWly
bXVja2FydEBjYXRhbHlzdC5uZXQubnoACgkQiBAgH4ERwwN4yBAAlUN+S3wogjAV
gTm69b4T3VLbEV1Fo+lHk0HFD+zhXgw2Ypb2yZigOkQfDkQFX7VPG8v7LsKljg5P
1ssb3kjie8VeoZ1yri3JSAvCxrov5CgtBeiI8vtftPnubhAzsh6IeiTu8IlqP72X
O7UWVZY+PF6pY7h8hxoahZa2BBitevr9cLHpI7EhLX8TJedY6/UCS6Xn/IO1mpHG
bqqjQnsanqBQKUq43GUHvdZylhVlHhrGqVyWuYeagoSqaDw5H7EtotQbfptyoVYu
8jeWwfg6Tfk5N+DViGSQ3qrxFKQFh8MtUksN2xU3oUaV99fmzR7HljYMJ8zhRS7O
Dkwd3ZcCSM2hasEJBLPGOPfRwPLgS+lB+iHQ6yFTRt6GCOzt+axloo4vDZHf3p8X
ysXOwO7Tq4L7zq5YqUoWevNcJZJ2qSXfbLzzZ/MB/+mfUgX5TqOsSgP5Be8rpoSq
cpdmY//Ygow4x5HJ+k6LskJfQH/84x4q6E4Tlc8BwfEOXIk1R4IpwV1CNs2zZ1TN
JelmHvlNfyxHz2PS/Cti8+L2mclr9Oj5kNCJVytJuIFqKLzRS7Cq+qEgKkI24+en
ufH0o55n9n3lRGWZdisvDS/dqKzXJEEEWasTr2rv+b0okXXRgqGP6uYOoKcXZajj
Ke6eg1BrEFTauPoOrv8MJN1x4ppfOjM=
=+dwk
-----END PGP SIGNATURE-----