<html><body>
<div style="font-size:14pt;font-family:Tahoma">Reported here:<br></div>
<div style="font-size:14pt;font-family:Tahoma"><a href="https://github.com/FRRouting/frr/issues/10826">https://github.com/FRRouting/frr/issues/10826</a></div>
<div style="font-size:14pt;font-family:Tahoma" id="signature"><div>
<br></div></div>
<div><br></div><blockquote class="as91pFk8AX8sDPOUp66RUy8IoCRjefxNG8oLk8fg" style="PADDING-RIGHT: 0px; PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #000000 2px solid; MARGIN-RIGHT: 0px">
From: frog-request@lists.frrouting.org<br>
To: frog@lists.frrouting.org<br>
Date: Sun, 03 Apr 2022 12:00:02 +0000<br>
Subject: frog Digest, Vol 61, Issue 2<br>
<br>
<div style="font-family: monospace, courier new, courier">
Send frog mailing list submissions to<br>
<a href="mailto:frog%40lists.frrouting.org">frog@lists.frrouting.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="https://lists.frrouting.org/listinfo/frog" target="_blank">https://lists.frrouting.org/listinfo/frog</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:frog-request%40lists.frrouting.org">frog-request@lists.frrouting.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:frog-owner%40lists.frrouting.org">frog-owner@lists.frrouting.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of frog digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. BGPD hanging in FRR 8.2.2 (Philip Smith)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Sat, 2 Apr 2022 20:47:42 +0100<br>
From: Philip Smith <<a href="mailto:philip%40nsrc.org">philip@nsrc.org</a>><br>
To: <a href="mailto:frog%40lists.frrouting.org">frog@lists.frrouting.org</a><br>
Subject: [FROG] BGPD hanging in FRR 8.2.2<br>
Message-ID: <<a href="mailto:54869a9a-07db-2033-cc16-c0b8a6612060%40nsrc.org">54869a9a-07db-2033-cc16-c0b8a6612060@nsrc.org</a>><br>
Content-Type: text/plain; charset=UTF-8; format=flowed<br>
<br>
Hi everyone,<br>
<br>
Just following up on my previous note about BGPD hanging in FRR 8.2.2. I <br>
now have more info to share.<br>
<br>
As background, I've got around 60 BGP feeds total in 30 different <br>
"views", to form a route collector for analysis work I'm doing of the <br>
global R&E routing table.<br>
<br>
This hang seems to have a period of 5-7 days. Using FRR 8.2.2 on Ubuntu <br>
20.04. Not had any issue with FRR 8.1.0; this only started with FRR 8.2.2.<br>
<br>
The latest hang earlier today allowed a colleague to grab debug info <br>
which I hope will help.<br>
<br>
/var/log/frr/frr.log shows entries like this:<br>
<br>
Apr 2 11:46:42 frr watchfrr[52904]: [T58XM-TP956][EC 268435457] bgpd <br>
state -> unresponsive : no response yet to ping sent 90 seconds ago<br>
Apr 2 11:46:42 frr watchfrr[52904]: [YFT0P-5Q5YX] Forked background <br>
command [pid 1674696]: /usr/lib/frr/watchfrr.sh restart bgpd<br>
Apr 2 11:47:02 frr watchfrr[52904]: [ZE9RA-19PS5] restart bgpd child <br>
process 1674696 still running after 20 seconds, sending signal 15<br>
Apr 2 11:47:02 frr watchfrr[52904]: [SK7QP-A2GT9] restart bgpd process <br>
1674696 terminated due to signal 15<br>
<snip><br>
Apr 2 14:18:03 frr watchfrr[52904]: [YFT0P-5Q5YX] Forked background <br>
command [pid 1697956]: /usr/lib/frr/watchfrr.sh restart bgpd<br>
Apr 2 14:18:23 frr watchfrr[52904]: [ZE9RA-19PS5] restart bgpd child <br>
process 1697956 still running after 20 seconds, sending signal 15<br>
Apr 2 14:18:23 frr watchfrr[52904]: [SK7QP-A2GT9] restart bgpd process <br>
1697956 terminated due to signal 15<br>
<br>
which just repeat every 10 minutes or so.<br>
<br>
A few hours earlier I was getting:<br>
<br>
Apr 1 22:53:19 frr bgpd[52925]: [YZRX4-ZXG0C][EC 100663315] Thread <br>
Starvation: {(thread *)0x5566a35c01a0 arg=0x556682b31da0 timer r=-5.940 <br>
bgp_announce_route_timer_expired() &paf->t_announce_route from <br>
bgpd/bgp_route.c:4763} was scheduled to pop greater than 4s ago<br>
<br>
Apr 1 23:24:34 frr bgpd[52925]: [YZRX4-ZXG0C][EC 100663315] Thread <br>
Starvation: {(thread *)0x5567954b16c0 arg=0x556682f14870 timer r=-5.224 <br>
bgp_announce_route_timer_expired() &paf->t_announce_route from <br>
bgpd/bgp_route.c:4763} was scheduled to pop greater than 4s ago<br>
<br>
<br>
Trying to connect by vtysh prints message of day, but never a command <br>
prompt. Same if trying to connect via telnet.<br>
<br>
The only way out is a kill -9 of the BGPD process, followed by a <br>
"systemctl restart frr".<br>
<br>
The process stack for bgpd shows:<br>
<br>
root@frr:~# cat /proc/52925/stack<br>
[<0>] futex_wait_queue_me+0xbb/0x120<br>
[<0>] futex_wait+0x105/0x290<br>
[<0>] do_futex+0x157/0x4d0<br>
[<0>] __x64_sys_futex+0x13f/0x170<br>
[<0>] do_syscall_64+0x57/0x190<br>
[<0>] entry_SYSCALL_64_after_hwframe+0x44/0xa9<br>
<br>
Thread debugging shows:<br>
<br>
[Thread debugging using libthread_db enabled]<br>
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".<br>
__pthread_clockjoin_ex (threadid=139670697043712, thread_return=0x0, <br>
clockid=<optimized out>, abstime=<optimized out>,<br>
block=<optimized out>) at pthread_join_common.c:145<br>
145 pthread_join_common.c: No such file or directory.<br>
(gdb) bt<br>
#0 __pthread_clockjoin_ex (threadid=139670697043712, thread_return=0x0, <br>
clockid=<optimized out>, abstime=<optimized out>,<br>
block=<optimized out>) at pthread_join_common.c:145<br>
#1 0x00007f07b1f3d985 in ?? () from /lib/x86_64-linux-gnu/librtr.so.0<br>
#2 0x00007f07b1f38dc1 in rtr_mgr_stop () from <br>
/lib/x86_64-linux-gnu/librtr.so.0<br>
#3 0x00007f07b1f53ef0 in ?? () from <br>
/usr/lib/x86_64-linux-gnu/frr/modules/bgpd_rpki.so<br>
#4 0x00007f07b1f53f7d in ?? () from <br>
/usr/lib/x86_64-linux-gnu/frr/modules/bgpd_rpki.so<br>
#5 0x00007f07b1f543ca in ?? () from <br>
/usr/lib/x86_64-linux-gnu/frr/modules/bgpd_rpki.so<br>
#6 0x00007f07b2586621 in thread_call () from <br>
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0<br>
#7 0x00007f07b2540198 in frr_run () from <br>
/usr/lib/x86_64-linux-gnu/frr/libfrr.so.0<br>
#8 0x00005566800b6678 in main ()<br>
<br>
I've got about 2.5Mbytes of strace which I'll happily unicast to whoever <br>
would like to have a look at it. It looks very repetitive/boring to my <br>
non-developer eye, like something's got stuck waiting for something else.<br>
<br>
BTW, this is what's running (after I killed and restarted), including <br>
command line options:<br>
<br>
1707406 ? S<s 0:02 /usr/lib/frr/watchfrr -d -F traditional <br>
zebra bgpd staticd<br>
1707423 ? S<sl 0:01 /usr/lib/frr/zebra -d -F traditional -A <br>
127.0.0.1 -s 90000000<br>
1707428 ? S<sl 17:03 /usr/lib/frr/bgpd -d -F traditional -Z -M rpki<br>
1707435 ? S<s 0:00 /usr/lib/frr/staticd -d -F traditional -A <br>
127.0.0.1<br>
<br>
Any ideas? I'd hate to revert to 8.1 but...<br>
<br>
philip<br>
--<br>
<br>
<br>
<br>
------------------------------<br>
<br>
Subject: Digest Footer<br>
<br>
_______________________________________________<br>
frog mailing list<br>
<a href="mailto:frog%40lists.frrouting.org">frog@lists.frrouting.org</a><br>
<a href="https://lists.frrouting.org/listinfo/frog" target="_blank">https://lists.frrouting.org/listinfo/frog</a><br>
<br>
<br>
------------------------------<br>
<br>
End of frog Digest, Vol 61, Issue 2<br>
***********************************
</div>
</blockquote>
<br>*****************************************************<br>
Best Service and Trustworthy From Us<br>
<br>
Our Mail Server Support IPv6 & IPv4 Mail<br>
<br>
======================================================<br>
</body></html>