[FROG] FRR8.4 BGPD crashes
Philip Smith
philip at nsrc.org
Wed Nov 9 12:14:47 UTC 2022
Hi Donald,
Donald Sharp wrote on 9/11/2022 21:28:
> I noticed the bgp crash in rpki myself this yesterday morning as well.
> https://github.com/FRRouting/frr/pull/12287/commits/31d0363ffc3c387204068d9b81c4f281a2116342 <https://github.com/FRRouting/frr/pull/12287/commits/31d0363ffc3c387204068d9b81c4f281a2116342>
> That is the commit you want to fix the issue. I'll make sure it gets
> backported to 8.4
Ah! :-) Ok will look out for the fix in 8.4. Thank you.
> For the eth0 issue. What do you have configured on that interface? and
> Can I see `show int eth0` inside of vtysh?
Looking at the logs and the timing, I think this is a feature of bgpd
starting up. The eth0 message only pops up right before the EBGP
sessions are re-established after the bgpd crash in 8.4. The IPv6
sessions come up, and the message about v6 LL never reappears after
that. The below is from 8.3.1 - but it is the same format as my other
8.4 instance (a testbed).
frr.nsrc.org# show interface eth0
Interface eth0 is up, line protocol is up
Link ups: 0 last: (never)
Link downs: 0 last: (never)
vrf: default
index 56 metric 0 mtu 1500 speed 10000
flags: <UP,BROADCAST,RUNNING,MULTICAST>
Type: Ethernet
HWaddr: 00:16:3e:35:10:ab
inet 128.xxx.xxx.xxx/25
inet6 2607:xxx:xxx::xxx/64
inet6 fe80::216:3eff:fe35:10ab/64
Interface Type VETH
Interface Slave Type None
protodown: off
Parent ifindex: 57
Looking back through older logs, I see the same in 8.3.1 each time I
start up frr after system reboot or "systemctl restart frr". Which is
not very often at all so I never noticed before. Perhaps nothing to
worry about, tbh.
philip
--
>
>
> On Tue, Nov 8, 2022 at 8:10 PM Philip Smith <philip at nsrc.org
> <mailto:philip at nsrc.org>> wrote:
>
> Hi everyone,
>
> FRR8.4 updated on my FRR8.3.1 system yesterday. 8.3.1 has been running
> fine for many weeks.
>
> FRR8.4's BGPD crashes every 25 minutes or so with this appearing in
> syslog:
>
> ********************
> Nov 8 22:59:29 frr BGP[81651]: Received signal 6 at 1667948369
> (si_addr
> 0x7000013ef3, PC 0x7fe3e2e5ba7c); aborting...
> Nov 8 22:59:29 frr BGP[81651]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x71)
> [0x7fe3e3184f01]
> Nov 8 22:59:29 frr BGP[81651]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5)
> [0x7fe3e3185105]
> Nov 8 22:59:29 frr BGP[81651]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xc3b45) [0x7fe3e31afb45]
> Nov 8 22:59:29 frr BGP[81651]:
> /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fe3e2e07520]
> Nov 8 22:59:29 frr BGP[81651]:
> /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c) [0x7fe3e2e5ba7c]
> Nov 8 22:59:29 frr BGP[81651]:
> /lib/x86_64-linux-gnu/libc.so.6(raise+0x16) [0x7fe3e2e07476]
> Nov 8 22:59:29 frr BGP[81651]:
> /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7fe3e2ded7f3]
> Nov 8 22:59:29 frr BGP[81651]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(_zlog_assert_failed+0xed)
> [0x7fe3e31d400d]
> Nov 8 22:59:29 frr BGP[81651]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(route_node_delete+0x16e)
> [0x7fe3e31b940e]
> Nov 8 22:59:29 frr BGP[81651]:
> /usr/lib/x86_64-linux-gnu/frr/modules/bgpd_rpki.so(+0x7a65)
> [0x7fe3e2bd0a65]
> Nov 8 22:59:29 frr BGP[81651]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x81)
> [0x7fe3e31c3c01]
> Nov 8 22:59:29 frr BGP[81651]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7fe3e3180508]
> Nov 8 22:59:30 frr BGP[81651]: /usr/lib/frr/bgpd(main+0x37c)
> [0x55d34af51dec]
> Nov 8 22:59:30 frr BGP[81651]:
> /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fe3e2deed90]
> Nov 8 22:59:30 frr BGP[81651]:
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7fe3e2deee40]
> Nov 8 22:59:30 frr BGP[81651]: /usr/lib/frr/bgpd(_start+0x25)
> [0x55d34af52895]
> Nov 8 22:59:30 frr BGP[81651]: in thread bgpd_sync_callback scheduled
> from bgpd/bgp_rpki.c:403 bgpd_sync_callback()
> ...
> Nov 8 23:24:44 frr BGP[84102]: Received signal 6 at 1667949884
> (si_addr
> 0x7000014886, PC 0x7ff44d243a7c); aborting...
> Nov 8 23:24:44 frr BGP[84102]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x71)
> [0x7ff44d56cf01]
> Nov 8 23:24:44 frr BGP[84102]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5)
> [0x7ff44d56d105]
> Nov 8 23:24:44 frr BGP[84102]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xc3b45) [0x7ff44d597b45]
> Nov 8 23:24:44 frr BGP[84102]:
> /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7ff44d1ef520]
> Nov 8 23:24:44 frr BGP[84102]:
> /lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x12c) [0x7ff44d243a7c]
> Nov 8 23:24:44 frr BGP[84102]:
> /lib/x86_64-linux-gnu/libc.so.6(raise+0x16) [0x7ff44d1ef476]
> Nov 8 23:24:44 frr BGP[84102]:
> /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7ff44d1d57f3]
> Nov 8 23:24:44 frr BGP[84102]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(_zlog_assert_failed+0xed)
> [0x7ff44d5bc00d]
> Nov 8 23:24:44 frr BGP[84102]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(route_node_delete+0x16e)
> [0x7ff44d5a140e]
> Nov 8 23:24:44 frr BGP[84102]:
> /usr/lib/x86_64-linux-gnu/frr/modules/bgpd_rpki.so(+0x7a65)
> [0x7ff44cfb8a65]
> Nov 8 23:24:44 frr BGP[84102]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(thread_call+0x81)
> [0x7ff44d5abc01]
> Nov 8 23:24:44 frr BGP[84102]:
> /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xe8) [0x7ff44d568508]
> Nov 8 23:24:44 frr BGP[84102]: /usr/lib/frr/bgpd(main+0x37c)
> [0x55750e7f8dec]
> Nov 8 23:24:44 frr BGP[84102]:
> /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7ff44d1d6d90]
> Nov 8 23:24:44 frr BGP[84102]:
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x80) [0x7ff44d1d6e40]
> Nov 8 23:24:44 frr BGP[84102]: /usr/lib/frr/bgpd(_start+0x25)
> [0x55750e7f9895]
> Nov 8 23:24:44 frr BGP[84102]: in thread bgpd_sync_callback scheduled
> from bgpd/bgp_rpki.c:403 bgpd_sync_callback()
> ********************
>
> This is with frr, frr-pythontools, and frr-rpki-rtrlib installed.
>
> Looks like unhappiness in the rpki component.
>
> I've got about 60 BGP feeds into this collector, some full table, but
> most are 30k routes.
>
> I also noticed this new issue appearing with FRR8.4:
>
> ********************
> 2022/11/08 23:37:24 BGP: [ZM2F8-MV4BJ][EC 33554509] Interface: eth0
> does
> not have a v6 LL address associated with it, waiting until one is
> created for it
> ********************
>
> No issue in FRR8.3.1 before it and eth0 definitely does have a v6 LL
> address associated with it, as per below:
>
> ********************
> 56: eth0 at if57: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue
> state UP group default qlen 1000
> link/ether 00:16:3e:35:10:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0
> inet 128.xxx.xxx.xxx brd 128.xxx.xxx.xxx scope global eth0
> valid_lft forever preferred_lft forever
> inet6 2607:xxxx:xxxx::xxxx/64 scope global
> valid_lft forever preferred_lft forever
> inet6 fe80::216:3eff:fe35:10ab/64 scope link
> valid_lft forever preferred_lft forever
> ********************
>
> If any other bits are need to help decipher any of this, please let
> me know.
>
> I've had to revert to 8.3.1 unfortunately, as the instabilities were
> causing issues for the providers of the feeds I am receiving.
>
> Thanks!
>
> philip
> --
>
>
>
> _______________________________________________
> frog mailing list
> frog at lists.frrouting.org <mailto:frog at lists.frrouting.org>
> https://lists.frrouting.org/listinfo/frog
> <https://lists.frrouting.org/listinfo/frog>
>
More information about the frog
mailing list