Hi Donald,
RPKI config literally is only (CloudFlare RPKI Server):
rpki
rpki cache 172.65.0.2 8282 preference 1
rpki cache 172.65.0.2 8283 preference 2
exit
sh memory bgpd:
System allocator statistics:
Total heap allocated: > 2GB
Holding block headers: 36 MiB
Used small blocks: 0 bytes
Used ordinary blocks: > 2GB
Free small blocks: 4960 bytes
Free ordinary blocks: 785 MiB
Ordinary blocks: 1400129
Small blocks: 122
Holding blocks: 3
(see system documentation for 'mallinfo' for meaning)
--- qmem logging subsystem ---
Type : Current# Size Total Max# MaxBytes
syslog target : 1 56 56 1 56
--- qmem libfrr ---
Type : Current# Size Total Max# MaxBytes
log thread-local buffer : 2 24608 49232 2 49232
Redistribution instance IDs : 6 2 144 6 144
Zclient : 2 3144 6288 2 6288
YANG data structure : 0 1032 0 1 1032
YANG module : 5 48 296 5 296
Work queue name string : 5 variable 136 6 176
Work queue item : 0 24 0 112 2688
Work queue : 5 144 760 6 912
VTY server : 2 32 80 2 80
VTY : 4 variable 61216 8 122432
VRF bit-map : 5 8 120 5 120
VRF : 1 216 216 1 216
Vector index : 31049 variable 986696 31053 986936
Vector : 31049 24 746440 31053 746536
Typed-heap array : 1 576 584 1 584
Typed-hash bucket : 21 variable 20982136 21 20982136
Thread stats : 33 96 3464 33 3464
Thread Poll Info : 6 8192 49200 6 49200
Thread master : 12 variable 50352 12 50352
Thread : 71 160 12136 358 63184
Route node : 14 120 1760 14 1760
Route table : 114 56 6464 114 6464
Stream FIFO : 50 64 3600 54 3888
Stream : 21 variable 752024 111502 20790624
Socket union : 36 112 4336 38 4576
Skiplist Counters : 2 68 144 2 144
Skip Node : 2 160 336 2 336
Skip List : 2 56 112 2 112
Route map dependency data : 15 16 408 15 408
Route map dependency : 9 24 232 9 232
Route map compiled : 103 variable 2488 103 2488
Route map rule str : 80 variable 1920 80 1920
Route map rule : 81 40 3304 81 3304
Route map index : 36 152 5600 36 5600
Route map name : 31 variable 744 32 768
Route map : 7 120 840 7 840
Ring buffer : 50 variable 2458800 54 2655504
Privilege information : 3 variable 136 3 136
Prefix : 3 56 168 3 168
Prefix List Trie Table : 1 4096 4104 1 4104
Prefix List Entry : 1 136 136 1 136
Prefix List Str : 1 16 24 1 24
Prefix List : 1 88 88 1 88
Northbound Configuration Entry: 127 1032 131064 127 131064
Northbound Configuration : 2 24 48 2 48
Northbound Node : 258 1192 307536 258 307536
Nexthop : 36 152 5536 38 5856
Module loading name : 1 5 24 1 24
Bitfield memory : 2 variable 10256 2 10256
Temporary memory : 134 variable 6816 142 11464
Link Node : 465 24 11192 597 14360
Link List : 62 40 2560 70 2888
Connected : 3 48 168 3 168
Interface : 2 272 560 2 560
ID Number Allocator Page : 2095 144 318760 2095 318760
ID Number Allocator Subdirectory: 17 1024 17544 17 17544
ID Number Allocator Directory : 2 1024 2064 2 2064
ID Number Allocator Name : 2 46 112 2 112
ID Number Allocator : 2 2072 4144 2 4144
Hash Index : 306 variable 32155448 309 33204032
Hash Bucket : 2231959 32 89416248 3116603 125184800
Hash : 611 variable 30008 615 30200
Graph Node : 15481 32 621368 15483 621448
Graph : 42 8 1008 43 1032
POSIX sync primitives : 4 variable 192 4 192
FRR POSIX Thread : 4 variable 320 4 320
RCU thread : 2 128 272 2 272
Lexer token (temporary) : 0 variable 0 2 64
Command Argument Name : 2294 variable 55088 2302 55280
Command Argument : 2 variable 48 10 272
Command Token Help : 9423 variable 227176 9431 227368
Command Token Text : 9423 variable 328904 9431 329320
Command Tokens : 13159 72 948184 13167 948776
Host config : 7 variable 280 7 280
Buffer data : 2 variable 8240 107564 443170128
Buffer : 6 24 144 10 240
--- qmem rfapi ---
Type : Current# Size Total Max# MaxBytes
RFAPI Import Table : 1 208 216 1 216
RFAPI Generic : 1 296 296 1 296
NVE Configuration : 1 2984 2984 1 2984
--- qmem bgpd ---
Type : Current# Size Total Max# MaxBytes
BGP EVPN instance information : 1 56 56 1 56
BGP interface context : 2 4 48 2 48
BGP PBR Context : 1 32 40 1 40
BGP Martian Addr Intf String : 2 7 48 2 48
Software Version : 2 16 48 2 48
BGP EVPN MH Information : 1 56 72 1 72
BGP Filter Information : 22 variable 528 23 552
BGP redistribution : 2 24 48 2 48
BGP own address : 2 64 144 2 144
BGP Dampening array : 3 variable 11928 3 11928
BGP nexthop : 26 184 4816 26 4816
BGP Process queue : 0 32 0 112 4480
Cluster list val : 1 4 24 1 24
Cluster list : 1 24 24 1 24
community-list handler : 1 120 136 1 136
extcommunity str : 3 66 216 3 216
extcommunity val : 3 8 72 4 96
extcommunity : 3 40 120 4 176
community str : 4 8192 32800 4 32800
community val : 31 variable 744 32 768
community : 31 40 1352 32 1408
BGP multipath info : 1886967 48 110401272 1886967 110401272
BGP adj out : 5485508 96 578004240 6599386 694406576
BGP adj in : 4287743 48 276474536 4287743 276474536
BGP synchronise : 8 48 496 10 608
BGP adv : 0 64 0 5086993 366586872
BGP adv attr : 0 24 0 888383 21587624
BGP connected : 2 4 48 2 48
BGP ancillary route info : 2143869 432 943368808 2143869 943368808
BGP route : 2143879 136 303155384 2143879 303155384
BGP node : 2056049 192 411343880 2056049 411343880
BGP table : 87 56 4872 87 4872
BGP aspath str : 263979 variable 17160632 263980 17161296
BGP aspath segment data : 264217 variable 6923800 264219 6924032
BGP aspath seg : 264217 24 6341224 264219 6341272
BGP aspath : 263979 40 10559384 263979 10559384
BGP attribute : 1965262 320 644785312 1965262 644785312
BGP packet : 8 56 448 200 11488
BGP update subgroup : 8 240 1984 10 2496
BGP update group : 8 104 832 8 832
BGP peer af : 24 80 2112 26 2288
Peer description : 24 variable 576 24 576
BGP peer hostname : 36 variable 1040 39 1128
BGP peer : 33 20864 688776 35 730536
BGP listen socket details : 2 144 304 2 304
BGP instance : 27 variable 10808 27 10808
Mac Hash Entry Intf String : 2 variable 48 3 72
Mac Hash Entry : 2 16 48 2 48
BGP Peer pthread Mutex : 1 40 40 1 40
BGP Peer pthread Conditional : 1 48 56 1 56
Peer KeepAlive Timer : 18 24 432 18 432
BGP RPKI Revalidation : 0 variable 0 294 7056
BGP RPKI RTRLib : 1253160 variable 50842592 1253222 60981208
BGP RPKI Cache server group : 0 48 0 1 56
BGP RPKI Cache server : 12 variable 640 12 640
ZA-CTN-RS01A# sh memory bgpd | grep RPKI
% Unknown action 'grep'
ZA-CTN-RS01A# sh memory bgpd | include RPKI
BGP RPKI Revalidation : 0 variable 0 294 7056
BGP RPKI RTRLib : 1253160 variable 50842592 1253222 60981208
BGP RPKI Cache server group : 0 48 0 1 56
BGP RPKI Cache server : 12 variable 640 12 640
Regards,
Chris.
From: Donald Sharp <donaldsharp72@gmail.com>
Sent: Thursday, 28 September 2023 18:37
To: Chris Knipe <savage@savage.za.org>
Cc: frog@lists.frrouting.org
Subject: Re: [FROG] RPKI, this is getting a bit frustrating
Can I see the output of `show memory` for bgpd? I'm not aware of any memory leaks in rpki( but that does not mean that they don't exist ). I'd also like to see the output of the rpki section of a show run.
donald
On Thu, Sep 28, 2023 at 12:33 PM Chris Knipe via frog <frog@lists.frrouting.org> wrote:
---------- Forwarded message ----------
From: Chris Knipe <savage@savage.za.org>
To: "frog@lists.frrouting.org" <frog@lists.frrouting.org>
Cc:
Bcc:
Date: Thu, 28 Sep 2023 15:24:21 +0000
Subject: RPKI, this is getting a bit frustratingHi All,
Been battling with this one now for a while, through a few different versions, and I am pulling my hair out here…. Is there any memory leaks in RPKI???
My RPKI config:
rpki
rpki cache 172.65.0.2 8282 preference 1
rpki cache 172.65.0.2 8283 preference 2
exit
Nothing more, nothing less. RPKI is (not yet) used in any route-maps, policies, etc.
RPKI works absolutely fine. However, when I start bgpd, it has a resident memory size of ~3.6G (fine, considering it’s loading multiple full tables for IPv4 & ipv6). When RPKI isn’t configured, the router is stable, and memory usage is consistent and constant at ~3.6GB.
However, as soon as RPKI is enabled, bgpd grows about 600MB in resident memory per day. Eventually, the system runs out of memory and bgpd is restarted (which, is obviously bad).
root@MY-LOC-RS01A:~# ps auxxw|grep bgpd
root 802 0.0 0.0 8784 2844 ? S<s Sep26 1:02 /usr/lib/frr/watchfrr -d -F traditional zebra mgmtd bgpd ospfd ospf6d staticd
frr 850 16.8 54.7 4656684 4429752 ? S<sl Sep26 547:23 /usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M rpki -e 10
root 12813 0.0 0.0 6608 2312 pts/2 S+ 17:21 0:00 grep --color=auto bgpd
We’re already at 4.6GB up from 3.6GB when BGP has initially started (2d uptime for the longest BGP peer – see below).
FRR version, and IPv4 BGP Peers (with prefixes):
ZA-CTN-RS01A# sh ver
FRRouting 9.0.1 (MY-LOC-RS01A) on Linux(5.15.0-84-generic).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
configured with:
'--build=x86_64-linux-gnu' '--prefix=/usr' '--includedir=${prefix}/include' '--mandir=${prefix}/share/man' '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disable-option-checking' '--disable-silent-rules' '--libdir=${prefix}/lib/x86_64-linux-gnu' '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode' '--localstatedir=/var/run/frr' '--sbindir=/usr/lib/frr' '--sysconfdir=/etc/frr' '--with-vtysh-pager=/usr/bin/pager' '--libdir=/usr/lib/x86_64-linux-gnu/frr' '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules' '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting' '--enable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html' '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq' '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256' '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty' '--enable-configfile-mask=0640' '--enable-logfile-mask=0640' 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'
IPv4 Unicast Summary (VRF default):
BGP router identifier aaa.bbb.112.164, local AS number xxxxxx vrf-id 0
BGP table version 8318219
RIB entries 1693671, using 310 MiB of memory
Peers 12, using 245 KiB of memory
Dampening enabled.
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd PfxSnt Desc
aaa.bbb.112.165 4 xxxxxx 723094 1164605 8318219 0 0 2d05h58m 4 1665344 MY-LOC-RS01B
aaa.bbb.112.240 4 xxxxxx 265406 743205 8318219 0 0 2d05h58m 835450 922653 MY-LOC-TR01A
aaa.bbb.112.241 4 xxxxxx 156145 763504 8318219 0 0 2d05h58m 603608 922653 MY-LOC-TR01B
aaa.bbb.112.242 4 xxxxxx 26776 761874 8318219 0 0 2d05h58m 113141 922653 MY-LOC-PR01A
aaa.bbb.112.243 4 xxxxxx 37609 761874 8318219 0 0 2d05h58m 113141 922653 MY-LOC-PR01B
aaa.bbb.112.244 4 xxxxxx 3241 745167 8318219 0 0 2d05h58m 0 922653 MY-LOC-CR01A
aaa.bbb.112.245 4 xxxxxx 3241 743205 8318219 0 0 2d05h58m 0 922653 MY-LOC-CR01B
aaa.bbb.112.248 4 xxxxxx 384 436702 8318219 0 0 06:16:02 0 922653 MY-LOC-IKE01A
aaa.bbb.112.249 4 xxxxxx 381 306758 8318219 0 0 06:18:32 0 922653 MY-LOC-IKE01B
aaa.bbb.113.151 4 xxxxxx 0 0 0 0 0 never Active 0 MY-LOC-RS01A
aaa.bbb.113.152 4 xxxxxx 0 0 0 0 0 never Active 0 MY-LOC-RS01B
Total number of neighbors 11
Thanks,
Chris.
---------- Forwarded message ----------
From: Chris Knipe via frog <frog@lists.frrouting.org>
To: "frog@lists.frrouting.org" <frog@lists.frrouting.org>
Cc:
Bcc:
Date: Thu, 28 Sep 2023 15:24:21 +0000
Subject: [FROG] RPKI, this is getting a bit frustrating
_______________________________________________
frog mailing list
frog@lists.frrouting.org
https://lists.frrouting.org/listinfo/frog