[FROG] RPKI, this is getting a bit frustrating
Donald Sharp
donaldsharp72 at gmail.com
Thu Sep 28 17:43:59 UTC 2023
Can we wait a couple hours and regrab the memory for bgp? I'd like to see
where it is growing. As of now I'm not convinced it's rpki. I just turned
on rpki (again!) on my home full feed. I'll keep an eye on it as well.
donald
On Thu, Sep 28, 2023 at 1:07 PM Chris Knipe <savage at savage.za.org> wrote:
> Hi Donald,
>
>
>
> RPKI config literally is only (CloudFlare RPKI Server):
>
> rpki
>
> rpki cache 172.65.0.2 8282 preference 1
>
> rpki cache 172.65.0.2 8283 preference 2
>
> exit
>
>
>
> sh memory bgpd:
>
> System allocator statistics:
>
> Total heap allocated: > 2GB
>
> Holding block headers: 36 MiB
>
> Used small blocks: 0 bytes
>
> Used ordinary blocks: > 2GB
>
> Free small blocks: 4960 bytes
>
> Free ordinary blocks: 785 MiB
>
> Ordinary blocks: 1400129
>
> Small blocks: 122
>
> Holding blocks: 3
>
> (see system documentation for 'mallinfo' for meaning)
>
> --- qmem logging subsystem ---
>
> Type : Current# Size Total Max#
> MaxBytes
>
> syslog target : 1 56 56
> 1 56
>
> --- qmem libfrr ---
>
> Type : Current# Size Total Max#
> MaxBytes
>
> log thread-local buffer : 2 24608 49232 2
> 49232
>
> Redistribution instance IDs : 6 2 144 6
> 144
>
> Zclient : 2 3144 6288 2
> 6288
>
> YANG data structure : 0 1032 0 1
> 1032
>
> YANG module : 5 48 296 5
> 296
>
> Work queue name string : 5 variable 136 6
> 176
>
> Work queue item : 0 24 0 112
> 2688
>
> Work queue : 5 144 760 6
> 912
>
> VTY server : 2 32 80
> 2 80
>
> VTY : 4 variable 61216 8
> 122432
>
> VRF bit-map : 5 8 120 5
> 120
>
> VRF : 1 216 216 1
> 216
>
> Vector index : 31049 variable 986696 31053
> 986936
>
> Vector : 31049 24 746440 31053
> 746536
>
> Typed-heap array : 1 576 584 1
> 584
>
> Typed-hash bucket : 21 variable 20982136 21
> 20982136
>
> Thread stats : 33 96 3464 33
> 3464
>
> Thread Poll Info : 6 8192 49200 6
> 49200
>
> Thread master : 12 variable 50352 12
> 50352
>
> Thread : 71 160 12136 358
> 63184
>
> Route node : 14 120 1760 14
> 1760
>
> Route table : 114 56 6464 114
> 6464
>
> Stream FIFO : 50 64 3600 54
> 3888
>
> Stream : 21 variable 752024 111502
> 20790624
>
> Socket union : 36 112 4336 38
> 4576
>
> Skiplist Counters : 2 68 144 2
> 144
>
> Skip Node : 2 160 336 2
> 336
>
> Skip List : 2 56 112 2
> 112
>
> Route map dependency data : 15 16 408 15
> 408
>
> Route map dependency : 9 24 232 9
> 232
>
> Route map compiled : 103 variable 2488 103
> 2488
>
> Route map rule str : 80 variable 1920 80
> 1920
>
> Route map rule : 81 40 3304 81
> 3304
>
> Route map index : 36 152 5600 36
> 5600
>
> Route map name : 31 variable 744 32
> 768
>
> Route map : 7 120 840 7
> 840
>
> Ring buffer : 50 variable 2458800 54
> 2655504
>
> Privilege information : 3 variable 136 3
> 136
>
> Prefix : 3 56 168 3
> 168
>
> Prefix List Trie Table : 1 4096 4104 1
> 4104
>
> Prefix List Entry : 1 136 136 1
> 136
>
> Prefix List Str : 1 16 24
> 1 24
>
> Prefix List : 1 88 88
> 1 88
>
> Northbound Configuration Entry: 127 1032 131064 127
> 131064
>
> Northbound Configuration : 2 24 48
> 2 48
>
> Northbound Node : 258 1192 307536 258
> 307536
>
> Nexthop : 36 152 5536 38
> 5856
>
> Module loading name : 1 5 24
> 1 24
>
> Bitfield memory : 2 variable 10256 2
> 10256
>
> Temporary memory : 134 variable 6816 142
> 11464
>
> Link Node : 465 24 11192 597
> 14360
>
> Link List : 62 40 2560 70
> 2888
>
> Connected : 3 48 168 3
> 168
>
> Interface : 2 272 560 2
> 560
>
> ID Number Allocator Page : 2095 144 318760 2095
> 318760
>
> ID Number Allocator Subdirectory: 17 1024 17544 17
> 17544
>
> ID Number Allocator Directory : 2 1024 2064 2
> 2064
>
> ID Number Allocator Name : 2 46 112 2
> 112
>
> ID Number Allocator : 2 2072 4144 2
> 4144
>
> Hash Index : 306 variable 32155448 309
> 33204032
>
> Hash Bucket : 2231959 32 89416248 3116603
> 125184800
>
> Hash : 611 variable 30008 615
> 30200
>
> Graph Node : 15481 32 621368 15483
> 621448
>
> Graph : 42 8 1008 43
> 1032
>
> POSIX sync primitives : 4 variable 192 4
> 192
>
> FRR POSIX Thread : 4 variable 320 4
> 320
>
> RCU thread : 2 128 272 2
> 272
>
> Lexer token (temporary) : 0 variable 0
> 2 64
>
> Command Argument Name : 2294 variable 55088 2302
> 55280
>
> Command Argument : 2 variable 48 10
> 272
>
> Command Token Help : 9423 variable 227176 9431
> 227368
>
> Command Token Text : 9423 variable 328904 9431
> 329320
>
> Command Tokens : 13159 72 948184 13167
> 948776
>
> Host config : 7 variable 280 7
> 280
>
> Buffer data : 2 variable 8240 107564
> 443170128
>
> Buffer : 6 24 144 10
> 240
>
> --- qmem rfapi ---
>
> Type : Current# Size Total Max#
> MaxBytes
>
> RFAPI Import Table : 1 208 216 1
> 216
>
> RFAPI Generic : 1 296 296 1
> 296
>
> NVE Configuration : 1 2984 2984 1
> 2984
>
> --- qmem bgpd ---
>
> Type : Current# Size Total Max#
> MaxBytes
>
> BGP EVPN instance information : 1 56 56
> 1 56
>
> BGP interface context : 2 4 48
> 2 48
>
> BGP PBR Context : 1 32 40
> 1 40
>
> BGP Martian Addr Intf String : 2 7 48
> 2 48
>
> Software Version : 2 16 48
> 2 48
>
> BGP EVPN MH Information : 1 56 72
> 1 72
>
> BGP Filter Information : 22 variable 528 23
> 552
>
> BGP redistribution : 2 24 48
> 2 48
>
> BGP own address : 2 64 144 2
> 144
>
> BGP Dampening array : 3 variable 11928 3
> 11928
>
> BGP nexthop : 26 184 4816 26
> 4816
>
> BGP Process queue : 0 32 0 112
> 4480
>
> Cluster list val : 1 4 24
> 1 24
>
> Cluster list : 1 24 24
> 1 24
>
> community-list handler : 1 120 136 1
> 136
>
> extcommunity str : 3 66 216 3
> 216
>
> extcommunity val : 3 8 72
> 4 96
>
> extcommunity : 3 40 120 4
> 176
>
> community str : 4 8192 32800 4
> 32800
>
> community val : 31 variable 744 32
> 768
>
> community : 31 40 1352 32
> 1408
>
> BGP multipath info : 1886967 48 110401272 1886967
> 110401272
>
> BGP adj out : 5485508 96 578004240 6599386
> 694406576
>
> BGP adj in : 4287743 48 276474536 4287743
> 276474536
>
> BGP synchronise : 8 48 496 10
> 608
>
> BGP adv : 0 64 0 5086993
> 366586872
>
> BGP adv attr : 0 24 0 888383
> 21587624
>
> BGP connected : 2 4 48
> 2 48
>
> BGP ancillary route info : 2143869 432 943368808 2143869
> 943368808
>
> BGP route : 2143879 136 303155384 2143879
> 303155384
>
> BGP node : 2056049 192 411343880 2056049
> 411343880
>
> BGP table : 87 56 4872 87
> 4872
>
> BGP aspath str : 263979 variable 17160632 263980
> 17161296
>
> BGP aspath segment data : 264217 variable 6923800 264219
> 6924032
>
> BGP aspath seg : 264217 24 6341224 264219
> 6341272
>
> BGP aspath : 263979 40 10559384 263979
> 10559384
>
> BGP attribute : 1965262 320 644785312 1965262
> 644785312
>
> BGP packet : 8 56 448 200
> 11488
>
> BGP update subgroup : 8 240 1984 10
> 2496
>
> BGP update group : 8 104 832 8
> 832
>
> BGP peer af : 24 80 2112 26
> 2288
>
> Peer description : 24 variable 576 24
> 576
>
> BGP peer hostname : 36 variable 1040 39
> 1128
>
> BGP peer : 33 20864 688776 35
> 730536
>
> BGP listen socket details : 2 144 304 2
> 304
>
> BGP instance : 27 variable 10808 27
> 10808
>
> Mac Hash Entry Intf String : 2 variable 48
> 3 72
>
> Mac Hash Entry : 2 16 48
> 2 48
>
> BGP Peer pthread Mutex : 1 40 40
> 1 40
>
> BGP Peer pthread Conditional : 1 48 56
> 1 56
>
> Peer KeepAlive Timer : 18 24 432 18
> 432
>
> BGP RPKI Revalidation : 0 variable 0 294
> 7056
>
> BGP RPKI RTRLib : 1253160 variable 50842592 1253222
> 60981208
>
> BGP RPKI Cache server group : 0 48 0
> 1 56
>
> BGP RPKI Cache server : 12 variable 640 12
> 640
>
> ZA-CTN-RS01A# sh memory bgpd | grep RPKI
>
> % Unknown action 'grep'
>
> ZA-CTN-RS01A# sh memory bgpd | include RPKI
>
> BGP RPKI Revalidation : 0 variable 0 294
> 7056
>
> BGP RPKI RTRLib : 1253160 variable 50842592 1253222
> 60981208
>
> BGP RPKI Cache server group : 0 48 0
> 1 56
>
> BGP RPKI Cache server : 12 variable 640 12
> 640
>
>
>
> Regards,
>
> Chris.
>
>
>
>
>
>
>
> *From:* Donald Sharp <donaldsharp72 at gmail.com>
> *Sent:* Thursday, 28 September 2023 18:37
> *To:* Chris Knipe <savage at savage.za.org>
> *Cc:* frog at lists.frrouting.org
> *Subject:* Re: [FROG] RPKI, this is getting a bit frustrating
>
>
>
> Can I see the output of `show memory` for bgpd? I'm not aware of any
> memory leaks in rpki( but that does not mean that they don't exist ). I'd
> also like to see the output of the rpki section of a show run.
>
>
>
> donald
>
>
>
> On Thu, Sep 28, 2023 at 12:33 PM Chris Knipe via frog <
> frog at lists.frrouting.org> wrote:
>
>
>
>
> ---------- Forwarded message ----------
> From: Chris Knipe <savage at savage.za.org>
> To: "frog at lists.frrouting.org" <frog at lists.frrouting.org>
> Cc:
> Bcc:
> Date: Thu, 28 Sep 2023 15:24:21 +0000
> Subject: RPKI, this is getting a bit frustrating
>
> Hi All,
>
>
>
> Been battling with this one now for a while, through a few different
> versions, and I am pulling my hair out here…. Is there any memory leaks in
> RPKI???
>
>
>
> My RPKI config:
>
> rpki
>
> rpki cache 172.65.0.2 8282 preference 1
>
> rpki cache 172.65.0.2 8283 preference 2
>
> exit
>
>
>
> Nothing more, nothing less. RPKI is (not yet) used in any route-maps,
> policies, etc.
>
>
>
> RPKI works absolutely fine. However, when I start bgpd, it has a resident
> memory size of ~3.6G (fine, considering it’s loading multiple full tables
> for IPv4 & ipv6). When RPKI isn’t configured, the router is stable, and
> memory usage is consistent and constant at ~3.6GB.
>
>
>
> However, as soon as RPKI is enabled, bgpd grows about 600MB in resident
> memory per day. Eventually, the system runs out of memory and bgpd is
> restarted (which, is obviously bad).
>
>
>
> root at MY-LOC-RS01A:~# ps auxxw|grep bgpd
>
> root 802 0.0 0.0 8784 2844 ? S<s Sep26 1:02
> /usr/lib/frr/watchfrr -d -F traditional zebra mgmtd bgpd ospfd ospf6d
> staticd
>
> frr 850 16.8 54.7 4656684 4429752 ? S<sl Sep26 547:23
> /usr/lib/frr/bgpd -d -F traditional -A 127.0.0.1 -M rpki -e 10
>
> root 12813 0.0 0.0 6608 2312 pts/2 S+ 17:21 0:00 grep
> --color=auto bgpd
>
>
>
> We’re already at 4.6GB up from 3.6GB when BGP has initially started (2d
> uptime for the longest BGP peer – see below).
>
>
>
>
>
> FRR version, and IPv4 BGP Peers (with prefixes):
>
> ZA-CTN-RS01A# sh ver
>
> FRRouting 9.0.1 (MY-LOC-RS01A) on Linux(5.15.0-84-generic).
>
> Copyright 1996-2005 Kunihiro Ishiguro, et al.
>
> configured with:
>
> '--build=x86_64-linux-gnu' '--prefix=/usr'
> '--includedir=${prefix}/include' '--mandir=${prefix}/share/man'
> '--infodir=${prefix}/share/info' '--sysconfdir=/etc' '--localstatedir=/var'
> '--disable-option-checking' '--disable-silent-rules'
> '--libdir=${prefix}/lib/x86_64-linux-gnu'
> '--libexecdir=${prefix}/lib/x86_64-linux-gnu' '--disable-maintainer-mode'
> '--localstatedir=/var/run/frr' '--sbindir=/usr/lib/frr'
> '--sysconfdir=/etc/frr' '--with-vtysh-pager=/usr/bin/pager'
> '--libdir=/usr/lib/x86_64-linux-gnu/frr'
> '--with-moduledir=/usr/lib/x86_64-linux-gnu/frr/modules'
> '--disable-dependency-tracking' '--enable-rpki' '--disable-scripting'
> '--enable-pim6d' '--with-libpam' '--enable-doc' '--enable-doc-html'
> '--enable-snmp' '--enable-fpm' '--disable-protobuf' '--disable-zeromq'
> '--enable-ospfapi' '--enable-bgp-vnc' '--enable-multipath=256'
> '--enable-user=frr' '--enable-group=frr' '--enable-vty-group=frrvty'
> '--enable-configfile-mask=0640' '--enable-logfile-mask=0640'
> 'build_alias=x86_64-linux-gnu' 'PYTHON=python3'
>
>
>
>
>
> IPv4 Unicast Summary (VRF default):
>
> BGP router identifier aaa.bbb.112.164, local AS number xxxxxx vrf-id 0
>
> BGP table version 8318219
>
> RIB entries 1693671, using 310 MiB of memory
>
> Peers 12, using 245 KiB of memory
>
> Dampening enabled.
>
>
>
> Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ
> Up/Down State/PfxRcd PfxSnt Desc
>
> aaa.bbb.112.165 4 xxxxxx 723094 1164605 8318219 0 0
> 2d05h58m 4 1665344 MY-LOC-RS01B
>
> aaa.bbb.112.240 4 xxxxxx 265406 743205 8318219 0 0
> 2d05h58m 835450 922653 MY-LOC-TR01A
>
> aaa.bbb.112.241 4 xxxxxx 156145 763504 8318219 0 0
> 2d05h58m 603608 922653 MY-LOC-TR01B
>
> aaa.bbb.112.242 4 xxxxxx 26776 761874 8318219 0 0
> 2d05h58m 113141 922653 MY-LOC-PR01A
>
> aaa.bbb.112.243 4 xxxxxx 37609 761874 8318219 0 0
> 2d05h58m 113141 922653 MY-LOC-PR01B
>
> aaa.bbb.112.244 4 xxxxxx 3241 745167 8318219 0 0
> 2d05h58m 0 922653 MY-LOC-CR01A
>
> aaa.bbb.112.245 4 xxxxxx 3241 743205 8318219 0 0
> 2d05h58m 0 922653 MY-LOC-CR01B
>
> aaa.bbb.112.248 4 xxxxxx 384 436702 8318219 0 0
> 06:16:02 0 922653 MY-LOC-IKE01A
>
> aaa.bbb.112.249 4 xxxxxx 381 306758 8318219 0 0
> 06:18:32 0 922653 MY-LOC-IKE01B
>
> aaa.bbb.113.151 4 xxxxxx 0 0 0 0 0
> never Active 0 MY-LOC-RS01A
>
> aaa.bbb.113.152 4 xxxxxx 0 0 0 0 0
> never Active 0 MY-LOC-RS01B
>
> Total number of neighbors 11
>
>
>
> Thanks,
>
> Chris.
>
>
>
>
>
>
>
>
>
>
> ---------- Forwarded message ----------
> From: Chris Knipe via frog <frog at lists.frrouting.org>
> To: "frog at lists.frrouting.org" <frog at lists.frrouting.org>
> Cc:
> Bcc:
> Date: Thu, 28 Sep 2023 15:24:21 +0000
> Subject: [FROG] RPKI, this is getting a bit frustrating
> _______________________________________________
> frog mailing list
> frog at lists.frrouting.org
> https://lists.frrouting.org/listinfo/frog
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.frrouting.org/pipermail/frog/attachments/20230928/9c8ea428/attachment-0001.htm>
More information about the frog
mailing list