ospf6d issues (crashes getting routes from my network)
I just upgraded from FreeBSD 11 to 12 and upgrade from quagga to FRR at the same time. bgpd and ospfd work fine, but ospf6 doesn't work. Not sure what I'm going wrong. Here is the output from zebra and ospf6. If I commend out all the interfaces EXCEPT lo0, ospf6d does not crash. router ospf6 ospf6 router-id 172.17.18.98 redistribute connected route-map MONKEY-ospf6 redistribute static route-map MONKEY-AND-MORE-ospf6 !interface cxl1 area 0.0.0.0 !interface cxl2 area 0.0.0.0 !interface cxl3 area 0.0.0.0 interface lo0 area 0.0.0.0 As soon as I get routes from my network, ospf6d tanks. Any thoughts? *** LOGS *** 2019/10/10 23:19:31 OSPF6: Scheduling cxl1 for sso 2019/10/10 23:19:31 OSPF6: SPF: Scheduled in 0 msec 2019/10/10 23:19:31 OSPF6: SPF processing: # Areas: 1, SPF runtime: 0 sec 5 usec, Reason: L+ 2019/10/10 23:19:31 OSPF6: Scheduling cxl2 for sso 2019/10/10 23:19:31 OSPF6: SPF: Scheduled in 50 msec 2019/10/10 23:19:31 OSPF6: Scheduling cxl3 for sso 2019/10/10 23:19:31 OSPF6: Neighbor state change 172.17.24.25%cxl3: [Down]->[Init] (HelloReceived) 2019/10/10 23:19:31 OSPF6: Neighbor state change 172.17.24.25%cxl3: [Init]->[ExStart] (2-WayReceived) 2019/10/10 23:19:31 OSPF6: Neighbor state change 172.17.24.25%cxl3: [ExStart]->[ExChange] (NegotiationDone) 2019/10/10 23:19:31 OSPF6: Neighbor state change 172.17.24.25%cxl3: [ExChange]->[Loading] (ExchangeDone) 2019/10/10 23:19:31 OSPF6: SPF processing: # Areas: 1, SPF runtime: 0 sec 109 usec, Reason: R+, R-, L+ 2019/10/10 23:19:31 OSPF6: SPF: Scheduled in 83 msec 2019/10/10 23:19:31 OSPF6: SPF processing: # Areas: 1, SPF runtime: 0 sec 116 usec, Reason: R+, N+, L+ 2019/10/10 23:19:33 OSPF6: Neighbor state change 10.8.254.122%cxl1: [Down]->[Init] (HelloReceived) 2019/10/10 23:19:33 OSPF6: Neighbor state change 10.8.254.122%cxl1: [Init]->[ExStart] (2-WayReceived) 2019/10/10 23:19:33 OSPF6: Neighbor state change 10.8.254.122%cxl1: [ExStart]->[ExChange] (NegotiationDone) 2019/10/10 23:19:33 OSPF6: Neighbor state change 10.8.254.122%cxl1: [ExChange]->[Loading] (ExchangeDone) 2019/10/10 23:19:33 OSPF6: SPF: Scheduled in 0 msec 2019/10/10 23:19:33 OSPF6: SPF processing: # Areas: 1, SPF runtime: 0 sec 99 usec, Reason: R+, R- 2019/10/10 23:19:36 OSPF6: Neighbor state change 172.17.24.25%cxl3: [Loading]->[Full] (LoadingDone) 2019/10/10 23:19:36 OSPF6: SPF: Scheduled in 0 msec 2019/10/10 23:19:31 ZEBRA: client 32 says hello and bids fair to announce only ospf6 routes vrf=0 2019/10/10 23:19:38 ZEBRA: [EC 4043309117] Client 'ospf6' encountered an error and is shutting down. 2019/10/10 23:19:38 ZEBRA: release_daemon_table_chunks: Released 0 table chunks 2019/10/10 23:19:38 ZEBRA: release_daemon_label_chunks: Released 0 label chunks 2019/10/10 23:19:39 ZEBRA: client 32 disconnected. 100 ospf6 routes removed from the rib *** CRASH *** If I run on the command line and don't background, it bombs after 7 seconds: # ospf6d Illegal instruction Here is the end of truss: # truss ospf6d ... mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34863378432 (0x81e04f000) mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34863382528 (0x81e050000) mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34863386624 (0x81e051000) mmap(0x0,4096,PROT_READ|PROT_WRITE,MAP_PRIVATE|MAP_ANON,-1,0x0) = 34863390720 (0x81e052000) SIGNAL 11 (SIGSEGV) code=SEGV_ACCERR trapno=12 addr=0x7fffdffffff8 process killed, signal = 4 I wasn't seeing any CORE files when the process dumped, so I turned this knob: sysctl kern.sugid_coredump=1 Now there is a 1 GByte core file! # /usr/obj/usr/src/amd64.amd64/gnu/usr.bin/gdb/gdb/gdb /usr/local/sbin/ospf6d ospf6d.core (gdb) bt #0 0x000000080030678c in hash_get () from /usr/local/lib/libfrr.so.0 #1 0x000000080033a9b2 in route_node_lookup () from /usr/local/lib/libfrr.so.0 #2 0x0000000000241345 in ospf6_route_lookup () #3 0x000000000022e718 in ospf6_abr_examin_summary () #4 0x000000000022f13b in ospf6_abr_examin_brouter () #5 0x00000000002473ad in ospf6_top_init () #6 0x000000000022efda in ospf6_abr_examin_summary () #7 0x000000000022f13b in ospf6_abr_examin_brouter () #8 0x00000000002473ad in ospf6_top_init () #9 0x000000000022efda in ospf6_abr_examin_summary () #10 0x000000000022f13b in ospf6_abr_examin_brouter () #11 0x00000000002473ad in ospf6_top_init () .... lines #3 - #5 repeat to over #8500 ... I stopped hitting ENTER around there and quit. Any help would be appreciated. Rudy
On 10/11/2019 2:37 AM, Rudy Rucker wrote:
I just upgraded from FreeBSD 11 to 12 and upgrade from quagga to FRR at the same time. bgpd and ospfd work fine, but ospf6 doesn't work. Not sure what I'm going wrong. Here is the output from zebra and ospf6.
If you do ldd ospf6d ospf6d: libfrr.so.0 => /usr/local/lib/libfrr.so.0 (0x80086d000) libyang.so.0.16 => /usr/local/lib/libyang.so.0.16 (0x800b3e000) libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x800e35000) libm.so.5 => /lib/libm.so.5 (0x801038000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x801268000) libjson-c.so.4 => /usr/local/lib/libjson-c.so.4 (0x801487000) librt.so.1 => /usr/lib/librt.so.1 (0x801697000) libthr.so.3 => /lib/libthr.so.3 (0x80189d000) libc.so.7 => /lib/libc.so.7 (0x801ac6000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x801e81000) libelf.so.2 => /lib/libelf.so.2 (0x802123000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x80233b000) is it linked to anything odd or old by chance ? (libmap.conf?) ---Mike
Same libs as yours... /usr/local/sbin/ospf6d: libfrr.so.0 => /usr/local/lib/libfrr.so.0 (0x800295000) libyang.so.0.16 => /usr/local/lib/libyang.so.0.16 (0x80036c000) libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x800466000) libm.so.5 => /lib/libm.so.5 (0x80046b000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x80049d000) libjson-c.so.4 => /usr/local/lib/libjson-c.so.4 (0x8004be000) librt.so.1 => /usr/lib/librt.so.1 (0x8004d1000) libthr.so.3 => /lib/libthr.so.3 (0x8004d9000) libc.so.7 => /lib/libc.so.7 (0x800506000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x8008fe000) libelf.so.2 => /lib/libelf.so.2 (0x8009a2000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x8009bd000) Note: I was having issues with 12.0-RELEASE and the binary distribution (pkg install frr7), so I recompiled to 12.1-FreeBSD and built frr7 from the ports. No optimization flags. Still crashes. :( What is interesting, if I just put lo0 in area 0, it doesn't crash. When I get routes (by adding real interfaces), it crashes. Backtrace is in 'hash_get'. Could I have a "bad" IPv6 route that when ospf6d tries to learn it, it crashes? #0 0x000000080030678c in hash_get () from /usr/local/lib/libfrr.so.0 I'm looking in hash.c, but my C skills are about 20 years rusty. Rudy On 10/11/19 8:52 AM, mike tancsa wrote:
On 10/11/2019 2:37 AM, Rudy Rucker wrote:
I just upgraded from FreeBSD 11 to 12 and upgrade from quagga to FRR at the same time. bgpd and ospfd work fine, but ospf6 doesn't work. Not sure what I'm going wrong. Here is the output from zebra and ospf6.
If you do
ldd ospf6d ospf6d: libfrr.so.0 => /usr/local/lib/libfrr.so.0 (0x80086d000) libyang.so.0.16 => /usr/local/lib/libyang.so.0.16 (0x800b3e000) libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x800e35000) libm.so.5 => /lib/libm.so.5 (0x801038000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x801268000) libjson-c.so.4 => /usr/local/lib/libjson-c.so.4 (0x801487000) librt.so.1 => /usr/lib/librt.so.1 (0x801697000) libthr.so.3 => /lib/libthr.so.3 (0x80189d000) libc.so.7 => /lib/libc.so.7 (0x801ac6000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x801e81000) libelf.so.2 => /lib/libelf.so.2 (0x802123000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x80233b000)
is it linked to anything odd or old by chance ? (libmap.conf?)
---Mike
Rudy - Would you mind subscribing to `frog`. It's a pain in the whumple-lump to approve your posts every time. What is the output of `show int brief`? donald On Fri, Oct 11, 2019 at 12:11 PM Rudy Rucker <rudy@monkeybrains.net> wrote:
Same libs as yours...
/usr/local/sbin/ospf6d: libfrr.so.0 => /usr/local/lib/libfrr.so.0 (0x800295000) libyang.so.0.16 => /usr/local/lib/libyang.so.0.16 (0x80036c000) libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x800466000) libm.so.5 => /lib/libm.so.5 (0x80046b000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x80049d000) libjson-c.so.4 => /usr/local/lib/libjson-c.so.4 (0x8004be000) librt.so.1 => /usr/lib/librt.so.1 (0x8004d1000) libthr.so.3 => /lib/libthr.so.3 (0x8004d9000) libc.so.7 => /lib/libc.so.7 (0x800506000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x8008fe000) libelf.so.2 => /lib/libelf.so.2 (0x8009a2000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x8009bd000)
Note: I was having issues with 12.0-RELEASE and the binary distribution (pkg install frr7), so I recompiled to 12.1-FreeBSD and built frr7 from the ports. No optimization flags.
Still crashes. :(
What is interesting, if I just put lo0 in area 0, it doesn't crash. When I get routes (by adding real interfaces), it crashes.
Backtrace is in 'hash_get'. Could I have a "bad" IPv6 route that when ospf6d tries to learn it, it crashes?
#0 0x000000080030678c in hash_get () from /usr/local/lib/libfrr.so.0
I'm looking in hash.c, but my C skills are about 20 years rusty.
Rudy
On 10/11/19 8:52 AM, mike tancsa wrote:
On 10/11/2019 2:37 AM, Rudy Rucker wrote:
I just upgraded from FreeBSD 11 to 12 and upgrade from quagga to FRR at the same time. bgpd and ospfd work fine, but ospf6 doesn't work. Not sure what I'm going wrong. Here is the output from zebra and ospf6.
If you do
ldd ospf6d ospf6d: libfrr.so.0 => /usr/local/lib/libfrr.so.0 (0x80086d000) libyang.so.0.16 => /usr/local/lib/libyang.so.0.16 (0x800b3e000) libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x800e35000) libm.so.5 => /lib/libm.so.5 (0x801038000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x801268000) libjson-c.so.4 => /usr/local/lib/libjson-c.so.4 (0x801487000) librt.so.1 => /usr/lib/librt.so.1 (0x801697000) libthr.so.3 => /lib/libthr.so.3 (0x80189d000) libc.so.7 => /lib/libc.so.7 (0x801ac6000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x801e81000) libelf.so.2 => /lib/libelf.so.2 (0x802123000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x80233b000)
is it linked to anything odd or old by chance ? (libmap.conf?)
---Mike
_______________________________________________ frog mailing list frog@lists.frrouting.org https://lists.frrouting.org/listinfo/frog
On 10/11/19 9:15 AM, Donald Sharp wrote:
Rudy -
Would you mind subscribing to `frog`. It's a pain in the whumple-lump to approve your posts every time.
No problem, I didn't see the subscribe location until the first email went through. I couldn't find a Mailman link here: http://docs.frrouting.org/en/latest/overview.html#mailing-lists
What is the output of `show int brief`?
cc0 down default cc1 down default cxl0 down default cxl1 up default 208.69.43.190/30 + 2607:f598:b010:f010::2/64 cxl2 up default 208.69.43.146/30 + 2607:f598:b000:f003::2/64 cxl3 up default 208.69.43.129/30 + 2607:f598:bc10:72::2/64 cxl4 down default cxl5 up default 206.197.187.13/24 + 2001:504:30::ba03:2329:1/64 cxl6 down default cxl7 down default ipfw0 down default ixl0 up default ixl1 down default lo0 up default 208.69.43.198/32 208.69.43.199/32 + 2607:f598:0:dadf::1/64 usbus0 down default usbus1 down default usbus2 down default usbus3 down default usbus4 down default usbus5 down default usbus6 down default usbus7 down default This box is a replacement for our old qugga box, and I'm excited about the 100Gbps ports from the chelsio card. As soon as this is stabilized, I'm going to start upgrading uplinks! Rudy
donald
On Fri, Oct 11, 2019 at 12:11 PM Rudy Rucker <rudy@monkeybrains.net> wrote:
Same libs as yours...
/usr/local/sbin/ospf6d: libfrr.so.0 => /usr/local/lib/libfrr.so.0 (0x800295000) libyang.so.0.16 => /usr/local/lib/libyang.so.0.16 (0x80036c000) libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x800466000) libm.so.5 => /lib/libm.so.5 (0x80046b000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x80049d000) libjson-c.so.4 => /usr/local/lib/libjson-c.so.4 (0x8004be000) librt.so.1 => /usr/lib/librt.so.1 (0x8004d1000) libthr.so.3 => /lib/libthr.so.3 (0x8004d9000) libc.so.7 => /lib/libc.so.7 (0x800506000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x8008fe000) libelf.so.2 => /lib/libelf.so.2 (0x8009a2000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x8009bd000)
Note: I was having issues with 12.0-RELEASE and the binary distribution (pkg install frr7), so I recompiled to 12.1-FreeBSD and built frr7 from the ports. No optimization flags.
Still crashes. :(
What is interesting, if I just put lo0 in area 0, it doesn't crash. When I get routes (by adding real interfaces), it crashes.
Backtrace is in 'hash_get'. Could I have a "bad" IPv6 route that when ospf6d tries to learn it, it crashes?
#0 0x000000080030678c in hash_get () from /usr/local/lib/libfrr.so.0
I'm looking in hash.c, but my C skills are about 20 years rusty.
Rudy
On 10/11/19 8:52 AM, mike tancsa wrote:
On 10/11/2019 2:37 AM, Rudy Rucker wrote:
I just upgraded from FreeBSD 11 to 12 and upgrade from quagga to FRR at the same time. bgpd and ospfd work fine, but ospf6 doesn't work. Not sure what I'm going wrong. Here is the output from zebra and ospf6.
If you do
ldd ospf6d ospf6d: libfrr.so.0 => /usr/local/lib/libfrr.so.0 (0x80086d000) libyang.so.0.16 => /usr/local/lib/libyang.so.0.16 (0x800b3e000) libexecinfo.so.1 => /usr/lib/libexecinfo.so.1 (0x800e35000) libm.so.5 => /lib/libm.so.5 (0x801038000) libcrypt.so.5 => /lib/libcrypt.so.5 (0x801268000) libjson-c.so.4 => /usr/local/lib/libjson-c.so.4 (0x801487000) librt.so.1 => /usr/lib/librt.so.1 (0x801697000) libthr.so.3 => /lib/libthr.so.3 (0x80189d000) libc.so.7 => /lib/libc.so.7 (0x801ac6000) libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x801e81000) libelf.so.2 => /lib/libelf.so.2 (0x802123000) libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x80233b000)
is it linked to anything odd or old by chance ? (libmap.conf?)
---Mike
_______________________________________________ frog mailing list frog@lists.frrouting.org https://lists.frrouting.org/listinfo/frog
[SOLVED] Got it running! I tested the theory that 'bad routes' were crashing ospf6d. The filter-list makes my ospf6d work! router ospf6 auto-cost reference-bandwidth 10000 area 0.0.0.0 filter-list prefix IN-OSPF6 in interface cxl1 area 0.0.0.0 interface cxl2 area 0.0.0.0 interface cxl3 area 0.0.0.0 ! ipv6 prefix-list IN-OSPF6 seq 3 permit 2607:f598::/32 le 64 ipv6 prefix-list IN-OSPF6 seq 99 deny ::/0 le 128 [BUG] I'm happy to turn on some debugging to show the routes crash ospf6d if anyone can tell me those commands. If I add a permit and force an session to restart, I can cause the running ospf6d to crash. repo: ipv6 prefix-list IN-OSPF6 seq 10 permit ::/0 le 128 router ospf6 no interface cxl3 area 0.0.0.0 interface cxl3 area 0.0.0.0 + Illegal instruction ospf6d Rudy
Can we turn on: debug ospf6 spf debug ospf6 route debug ospf6 flooding debug ospf6 lsa The original stack-trace was very odd imo. It didn't make sense to me. Maybe some debug will shed some light? In any event you can join us in the FRRouting slack for a quicker turnaround ( goto https://frrouting.org and click on the slack link ). thanks! donald On Sat, Oct 12, 2019 at 1:22 AM Rudy Rucker <rudy@monkeybrains.net> wrote:
[SOLVED]
Got it running! I tested the theory that 'bad routes' were crashing ospf6d. The filter-list makes my ospf6d work!
router ospf6 auto-cost reference-bandwidth 10000 area 0.0.0.0 filter-list prefix IN-OSPF6 in interface cxl1 area 0.0.0.0 interface cxl2 area 0.0.0.0 interface cxl3 area 0.0.0.0 ! ipv6 prefix-list IN-OSPF6 seq 3 permit 2607:f598::/32 le 64 ipv6 prefix-list IN-OSPF6 seq 99 deny ::/0 le 128
[BUG]
I'm happy to turn on some debugging to show the routes crash ospf6d if anyone can tell me those commands.
If I add a permit and force an session to restart, I can cause the running ospf6d to crash.
repo: ipv6 prefix-list IN-OSPF6 seq 10 permit ::/0 le 128 router ospf6 no interface cxl3 area 0.0.0.0 interface cxl3 area 0.0.0.0
+ Illegal instruction ospf6d
Rudy
_______________________________________________ frog mailing list frog@lists.frrouting.org https://lists.frrouting.org/listinfo/frog
participants (3)
-
Donald Sharp -
mike tancsa -
Rudy Rucker