[FROG] FRR8.4 BGPD crashes

[Philip Smith]

Hi everyone,

FRR8.4 updated on my FRR8.3.1 system yesterday. 8.3.1 has been running 
fine for many weeks.

FRR8.4's BGPD crashes every 25 minutes or so with this appearing in syslog:

********************
Nov  8 22:59:29 frr BGP[81651]: Received signal 6 at 1667948369 (si_addr 
0x7000013ef3, PC 0x7fe3e2e5ba7c); aborting...
Nov  8 22:59:29 frr BGP[81651]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x71) 
[0x7fe3e3184f01]
Nov  8 22:59:29 frr BGP[81651]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7fe3e3185105]
Nov  8 22:59:29 frr BGP[81651]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xc3b45) [0x7fe3e31afb45]
Nov  8 22:59:29 frr BGP[81651]: 
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fe3e2e07520]
Nov  8 22:59:29 frr BGP[81651]: 
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c) [0x7fe3e2e5ba7c]
Nov  8 22:59:29 frr BGP[81651]: 
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16) [0x7fe3e2e07476]
Nov  8 22:59:29 frr BGP[81651]: 
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7fe3e2ded7f3]
Nov  8 22:59:29 frr BGP[81651]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(_zlog_assert_failed+0xed) 
[0x7fe3e31d400d]
Nov  8 22:59:29 frr BGP[81651]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(route_node_delete+0x16e) 
[0x7fe3e31b940e]
Nov  8 22:59:29 frr BGP[81651]: 
/usr/lib/x86_64-linux-gnu/frr/modules/bgpd_rpki.so(+0x7a65) [0x7fe3e2bd0a65]
Nov  8 22:59:29 frr BGP[81651]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x81) [0x7fe3e31c3c01]
Nov  8 22:59:29 frr BGP[81651]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7fe3e3180508]
Nov  8 22:59:30 frr BGP[81651]: /usr/lib/frr/bgpd(main+0x37c) 
[0x55d34af51dec]
Nov  8 22:59:30 frr BGP[81651]: 
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fe3e2deed90]
Nov  8 22:59:30 frr BGP[81651]: 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fe3e2deee40]
Nov  8 22:59:30 frr BGP[81651]: /usr/lib/frr/bgpd(_start+0x25) 
[0x55d34af52895]
Nov  8 22:59:30 frr BGP[81651]: in thread bgpd_sync_callback scheduled 
from bgpd/bgp_rpki.c:403 bgpd_sync_callback()
...
Nov  8 23:24:44 frr BGP[84102]: Received signal 6 at 1667949884 (si_addr 
0x7000014886, PC 0x7ff44d243a7c); aborting...
Nov  8 23:24:44 frr BGP[84102]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x71) 
[0x7ff44d56cf01]
Nov  8 23:24:44 frr BGP[84102]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7ff44d56d105]
Nov  8 23:24:44 frr BGP[84102]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xc3b45) [0x7ff44d597b45]
Nov  8 23:24:44 frr BGP[84102]: 
/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff44d1ef520]
Nov  8 23:24:44 frr BGP[84102]: 
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c) [0x7ff44d243a7c]
Nov  8 23:24:44 frr BGP[84102]: 
/lib/x86_64-linux-gnu/libc.so.6(raise+0x16) [0x7ff44d1ef476]
Nov  8 23:24:44 frr BGP[84102]: 
/lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7ff44d1d57f3]
Nov  8 23:24:44 frr BGP[84102]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(_zlog_assert_failed+0xed) 
[0x7ff44d5bc00d]
Nov  8 23:24:44 frr BGP[84102]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(route_node_delete+0x16e) 
[0x7ff44d5a140e]
Nov  8 23:24:44 frr BGP[84102]: 
/usr/lib/x86_64-linux-gnu/frr/modules/bgpd_rpki.so(+0x7a65) [0x7ff44cfb8a65]
Nov  8 23:24:44 frr BGP[84102]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x81) [0x7ff44d5abc01]
Nov  8 23:24:44 frr BGP[84102]: 
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7ff44d568508]
Nov  8 23:24:44 frr BGP[84102]: /usr/lib/frr/bgpd(main+0x37c) 
[0x55750e7f8dec]
Nov  8 23:24:44 frr BGP[84102]: 
/lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ff44d1d6d90]
Nov  8 23:24:44 frr BGP[84102]: 
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7ff44d1d6e40]
Nov  8 23:24:44 frr BGP[84102]: /usr/lib/frr/bgpd(_start+0x25) 
[0x55750e7f9895]
Nov  8 23:24:44 frr BGP[84102]: in thread bgpd_sync_callback scheduled 
from bgpd/bgp_rpki.c:403 bgpd_sync_callback()
********************

This is with frr, frr-pythontools, and frr-rpki-rtrlib installed.

Looks like unhappiness in the rpki component.

I've got about 60 BGP feeds into this collector, some full table, but 
most are 30k routes.

I also noticed this new issue appearing with FRR8.4:

********************
2022/11/08 23:37:24 BGP: [ZM2F8-MV4BJ][EC 33554509] Interface: eth0 does 
not have a v6 LL address associated with it, waiting until one is 
created for it
********************

No issue in FRR8.3.1 before it and eth0 definitely does have a v6 LL 
address associated with it, as per below:

********************
56: eth0 at if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue 
state UP group default qlen 1000
     link/ether 00:16:3e:35:10:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0
     inet 128.xxx.xxx.xxx brd 128.xxx.xxx.xxx scope global eth0
        valid_lft forever preferred_lft forever
     inet6 2607:xxxx:xxxx::xxxx/64 scope global
        valid_lft forever preferred_lft forever
     inet6 fe80::216:3eff:fe35:10ab/64 scope link
        valid_lft forever preferred_lft forever
********************

If any other bits are need to help decipher any of this, please let me know.

I've had to revert to 8.3.1 unfortunately, as the instabilities were 
causing issues for the providers of the feeds I am receiving.

Thanks!

philip
--