<div dir="ltr"><div dir="ltr">Hi Donald,</div><div dir="ltr"><br></div><div dir="ltr">Did try your changes, <br><div>They work, Thanks a lot for the fix.</div><div>It did not crash zebra when disabled zebra.</div><div>Here the logs:</div><div><div>==> /zebra.log <==</div><div>2019/11/18 18:46:13.653386 ZEBRA: rib_delnode: 0:<a href="http://10.37.0.0/20">10.37.0.0/20</a>: rn 0x246f310, re 0x26910c0, removing</div><div>2019/11/18 18:46:13.653428 ZEBRA: rib_delnode: 0:<a href="http://10.50.1.0/24">10.50.1.0/24</a>: rn 0x246fef0, re 0x2693840, removing</div><div>2019/11/18 18:46:13.654427 ZEBRA: connection closed socket [45]</div><div>2019/11/18 18:46:13.654460 ZEBRA: FPM: <a href="http://10.37.0.0/20">10.37.0.0/20</a> triggering update to FPM - Reason: installing in kernel</div><div>2019/11/18 18:46:13.654530 ZEBRA: [EC 4043309116] Client 'ospf' encountered an error and is shutting down.</div><div>2019/11/18 18:46:13.654593 ZEBRA: rib_process: 0:<a href="http://10.37.0.0/20">10.37.0.0/20</a>: rn 0x246f310, removing re 0x26910c0</div><div>2019/11/18 18:46:13.654615 ZEBRA: rib_unlink: 0:<a href="http://10.37.0.0/20">10.37.0.0/20</a>: rn 0x246f310, re 0x26910c0</div><div>2019/11/18 18:46:13.654640 ZEBRA: FPM: <a href="http://10.50.1.0/24">10.50.1.0/24</a> triggering update to FPM - Reason: installing in kernel</div><div>2019/11/18 18:46:13.654684 ZEBRA: rib_process: 0:<a href="http://10.50.1.0/24">10.50.1.0/24</a>: rn 0x246fef0, removing re 0x2693840</div><div>2019/11/18 18:46:13.654700 ZEBRA: rib_unlink: 0:<a href="http://10.50.1.0/24">10.50.1.0/24</a>: rn 0x246fef0, re 0x2693840</div><div>2019/11/18 18:46:13.654738 ZEBRA: Closing client 'ospf'</div><div>2019/11/18 18:46:13.654810 ZEBRA: release_daemon_table_chunks: Released 0 table chunks</div><div>2019/11/18 18:46:13.654865 ZEBRA: client 45 disconnected. 0 ospf routes removed from the rib</div><div>2019/11/18 18:46:13.655098 ZEBRA: 0:<a href="http://10.37.0.0/20">10.37.0.0/20</a> update_from_ctx(): no fib nhg</div><div>2019/11/18 18:46:13.655129 ZEBRA: 0:<a href="http://10.37.0.0/20">10.37.0.0/20</a> update_from_ctx(): rib nhg matched, changed 'false'</div><div>2019/11/18 18:46:13.655159 ZEBRA: 0:<a href="http://10.37.0.0/20">10.37.0.0/20</a>: Redist update re 0x246fa80 (connected), old 0x0 (None)</div><div>2019/11/18 18:46:13.655197 ZEBRA: 0:<a href="http://10.50.1.0/24">10.50.1.0/24</a> update_from_ctx(): no fib nhg</div><div>2019/11/18 18:46:13.655227 ZEBRA: 0:<a href="http://10.50.1.0/24">10.50.1.0/24</a> update_from_ctx(): rib nhg matched, changed 'false'</div><div>2019/11/18 18:46:13.655256 ZEBRA: 0:<a href="http://10.50.1.0/24">10.50.1.0/24</a>: Redist update re 0x246f980 (connected), old 0x0 (None)</div><div>2019/11/18 18:46:13.655298 ZEBRA: FPM: zfpm_netlink_encode_route : RTM_NEWROUTE <a href="http://10.37.0.0/20">10.37.0.0/20</a>, Proto: kernel, Metric: 0</div><div>2019/11/18 18:46:13.655329 ZEBRA: FPM: Intf: 50, Gateway: <No address>, Recursive: no, Type: Directly connected, Encap type: none</div><div>2019/11/18 18:46:13.655360 ZEBRA: FPM: zfpm_netlink_encode_route : RTM_NEWROUTE <a href="http://10.50.1.0/24">10.50.1.0/24</a>, Proto: kernel, Metric: 0</div><div>2019/11/18 18:46:13.655389 ZEBRA: FPM: Intf: 54, Gateway: <No address>, Recursive: no, Type: Directly connected, Encap type: none</div><div>2019/11/18 18:46:14.017619 ZEBRA: Terminating on signal</div><div>2019/11/18 18:46:14.017684 ZEBRA: FPM: Zebra fpm-fini called<-------------------------------(Added log ontop of your PR).</div><div>2019/11/18 18:46:14.017721 ZEBRA: FPM: Stopping existing stats timer</div><div>2019/11/18 18:46:14.017760 ZEBRA: Zebra dataplane pre-fini called</div><div>2019/11/18 18:46:14.017868 ZEBRA: Closing client 'bfd'</div><div>2019/11/18 18:46:14.017941 ZEBRA: release_daemon_table_chunks: Released 0 table chunks</div><div>2019/11/18 18:46:14.017993 ZEBRA: client 29 disconnected. 0 bfd routes removed from the rib</div><div>2019/11/18 18:46:14.018173 ZEBRA: Closing client 'static'</div><div>2019/11/18 18:46:14.018236 ZEBRA: release_daemon_table_chunks: Released 0 table chunks</div><div>2019/11/18 18:46:14.018297 ZEBRA: client 34 disconnected. 0 static routes removed from the rib</div><div>2019/11/18 18:46:14.018437 ZEBRA: vrf_terminate: Shutting down vrf subsystem</div><div>2019/11/18 18:46:14.018468 ZEBRA: VRF 0 is to be deleted.</div><div>2019/11/18 18:46:14.018494 ZEBRA: VRF 0 is to be disabled.</div><div>2019/11/18 18:46:14.018524 ZEBRA: VRF default id 0 is now inactive</div><div>2019/11/18 18:46:14.018562 ZEBRA: MESSAGE: ZEBRA_VRF_DELETE default</div><div>2019/11/18 18:46:14.018635 ZEBRA: rib_unlink: 0:<a href="http://10.37.0.0/20">10.37.0.0/20</a>: rn 0x246f310, re 0x246fa80</div><div>2019/11/18 18:46:14.018656 ZEBRA: rib_unlink: 0:<a href="http://10.50.1.0/24">10.50.1.0/24</a>: rn 0x246fef0, re 0x246f980</div><div>2019/11/18 18:46:14.018671 ZEBRA: rib_unlink: 0:<a href="http://172.179.1.0/24">172.179.1.0/24</a>: rn 0x246f900, re 0x246f3c0</div><div>2019/11/18 18:46:14.018698 ZEBRA: rib_unlink: 0:<a href="http://10.37.0.0/20">10.37.0.0/20</a> (MRIB): rn 0x2690d60, re 0x246f500</div><div>2019/11/18 18:46:14.018717 ZEBRA: rib_unlink: 0:<a href="http://10.50.1.0/24">10.50.1.0/24</a> (MRIB): rn 0x246f820, re 0x246ef40</div><div>2019/11/18 18:46:14.018735 ZEBRA: rib_unlink: 0:<a href="http://172.179.1.0/24">172.179.1.0/24</a> (MRIB): rn 0x246fa00, re 0x246f690</div><div>2019/11/18 18:46:14.018756 ZEBRA: VRF default id 0 deleted</div><div>2019/11/18 18:46:14.018875 ZEBRA: Zebra dataplane fini called</div><div>2019/11/18 18:46:14.018928 ZEBRA: Zebra dataplane shutdown status check called</div><div>2019/11/18 18:46:14.018954 ZEBRA: Failed to find route for ctx: no table for afi 1, safi 1, vrf 0</div><div>2019/11/18 18:46:14.018985 ZEBRA: Failed to process dplane results: no route for 0:<a href="http://10.37.0.0/20">10.37.0.0/20</a></div><div>2019/11/18 18:46:14.019016 ZEBRA: Failed to find route for ctx: no table for afi 1, safi 1, vrf 0</div><div>2019/11/18 18:46:14.019045 ZEBRA: Failed to process dplane results: no route for 0:<a href="http://10.50.1.0/24">10.50.1.0/24</a></div><div>2019/11/18 18:46:14.019075 ZEBRA: Failed to find route for ctx: no table for afi 1, safi 1, vrf 0</div><div>2019/11/18 18:46:14.019122 ZEBRA: Failed to process dplane results: no route for 0:<a href="http://172.179.1.0/24">172.179.1.0/24</a></div><div>2019/11/18 18:46:14.019161 ZEBRA: Zebra final shutdown</div><div>2019/11/18 18:46:14.019189 ZEBRA: Zebra dataplane shutdown called</div><div>2019/11/18 18:46:14.338434 ZEBRA: [EC 100663304] ERROR: No such command on config line 6: ptm-enable</div><div>2019/11/18 18:46:14.338489 ZEBRA: [EC 100663304] ERROR: No such command on config line 22: ip route <a href="http://92.10.1.0/24">92.10.1.0/24</a> 10.37.0.18 1</div><div>2019/11/18 18:46:14.338515 ZEBRA: [EC 100663304] ERROR: No such command on config line 23: ip route <a href="http://93.10.1.0/24">93.10.1.0/24</a> 10.37.0.19 1</div><div>2019/11/18 18:46:14.338543 ZEBRA: [EC 100663304] ERROR: No such command on config line 24: ip route <a href="http://94.10.1.0/24">94.10.1.0/24</a> 10.37.0.20 1</div><div>2019/11/18 18:46:14.338569 ZEBRA: [EC 100663304] ERROR: No such command on config line 25: ip route <a href="http://100.10.1.1/32">100.10.1.1/32</a> 10.50.1.2 1</div><div>2019/11/18 18:46:14.338596 ZEBRA: [EC 100663304] ERROR: No such command on config line 26: ip route static bfd 10.50.1.1 10.50.1.2 single-hop</div><div>2019/11/18 18:46:14.366362 ZEBRA: client 29 says hello and bids fair to announce only bfd routes vrf=0</div><div>2019/11/18 18:46:14.395026 ZEBRA: client 34 says hello and bids fair to announce only static routes vrf=0</div><div>2019/11/18 18:46:14.491567 ZEBRA: connection to the FPM has gone down: closed socket in read</div></div><div><br></div><div>Thanks</div><div>Thippanna</div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Nov 18, 2019 at 11:42 AM Donald Sharp <<a href="mailto:sharpd@cumulusnetworks.com">sharpd@cumulusnetworks.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Thippanna --<br>
<br>
Yep looks like FPM doesn't like having it's plug pulled. I've<br>
submitted PR <a href="https://github.com/FRRouting/frr/pull/5361" rel="noreferrer" target="_blank">https://github.com/FRRouting/frr/pull/5361</a> which may fix<br>
your issue. Can you give it a try? I do not have a FPM setup at this<br>
moment to see if it properly stops the crash ( we may just be onto a<br>
different crash with this change ). In the meantime I'll try to<br>
figure out a FPM setup that I can reasonably test.<br>
<br>
donald<br>
<br>
On Sun, Nov 17, 2019 at 11:02 PM hongal <<a href="mailto:hongal@gmail.com" target="_blank">hongal@gmail.com</a>> wrote:<br>
><br>
> Hi FRRTeam,<br>
><br>
> I am seeing an assert as below, while I tried to play a bit with FRR 7.2/stable latest code.<br>
> code, with FPM enabled. Trigger for this is we tried to disable zebra .<br>
><br>
> This is highly reproducible if we enable FPM, with few 1-5routes/interface configured.<br>
> Here is the backtrace for the zebra core: but let me know if you need more info:<br>
><br>
> i1".<br>
> Core was generated by `/usr/lib/frr/zebra -d -A 127.0.0.1 -M snmp -M fpm'.<br>
> Program terminated with signal SIGABRT, Aborted.<br>
> #0 0x00007f1456463428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54<br>
> 54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.<br>
> [Current thread is 1 (Thread 0x7f1457641700 (LWP 403))]<br>
> (gdb) bt<br>
> #0 0x00007f1456463428 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:54<br>
> #1 0x00007f145646502a in __GI_abort () at abort.c:89<br>
> #2 0x00007f1457193188 in core_handler (signo=11, siginfo=0x7ffcb511eab0, context=0x7ffcb511e980) at lib/sigevent.c:228<br>
> #3 <signal handler called><br>
> #4 0x00007f1453e942a5 in rnode_is_srcnode (rn=0x2607400) at ./lib/srcdest_table.h:85<br>
> #5 0x00007f1453e942d3 in srcdest_rnode_table (rn=0x2607400) at ./lib/srcdest_table.h:90<br>
> #6 0x00007f1453e94359 in rib_dest_table (dest=0x2436050) at zebra/rib.h:471<br>
> #7 0x00007f1453e94373 in rib_dest_vrf (dest=0x2436050) at zebra/rib.h:479<br>
> #8 0x00007f1453e948d1 in netlink_route_info_fill (ri=0x7ffcb511f000, cmd=25, dest=0x2436050, re=0x0) at zebra/zebra_fpm_netlink.c:295<br>
> #9 0x00007f1453e953f5 in zfpm_netlink_encode_route (cmd=25, dest=0x2436050, re=0x0, in_buf=0x2528f34 "$", in_buf_len=8188) at zebra/zebra_fpm_netlink.c:575<br>
> #10 0x00007f1453e91c0e in zfpm_encode_route (dest=0x2436050, re=0x0, in_buf=0x2528f34 "$", in_buf_len=8188, msg_type=0x7ffcb511f2a4) at zebra/zebra_fpm.c:887<br>
> #11 0x00007f1453e91e10 in zfpm_build_route_updates () at zebra/zebra_fpm.c:990<br>
> #12 0x00007f1453e92290 in zfpm_build_updates () at zebra/zebra_fpm.c:1151<br>
> #13 0x00007f1453e9237f in zfpm_write_cb (thread=0x7ffcb511f5c0) at zebra/zebra_fpm.c:1189<br>
> #14 0x00007f14571a8240 in thread_call (thread=0x7ffcb511f5c0) at lib/thread.c:1757<br>
> #15 0x00007f145715fd71 in frr_run (master=0x2351ac0) at lib/libfrr.c:1054<br>
> #16 0x00000000004262bf in main (argc=8, argv=0x7ffcb511f878) at zebra/main.c:468<br>
> (gdb)<br>
><br>
><br>
><br>
> Here is the running config :<br>
> root@dcl4:~# vtysh-frr<br>
><br>
> Hello, this is FRRouting (version 7.2).<br>
> Copyright 1996-2005 Kunihiro Ishiguro, et al.<br>
><br>
> dcl4# show running-config<br>
> Building configuration...<br>
><br>
> Current configuration:<br>
> !<br>
> frr version 7.2<br>
> frr defaults traditional<br>
> hostname dcl4<br>
> log file zebra.log<br>
> log timestamp precision 6<br>
> agentx<br>
> log file bgpd.log<br>
> log file staticd.log<br>
> log file bfdd.log<br>
> !<br>
> debug zebra dplane<br>
> debug zebra events<br>
> debug zebra fpm<br>
> debug zebra kernel<br>
> debug zebra rib<br>
> !<br>
> debug vrf<br>
> !<br>
> enable password zebra<br>
> password zebra<br>
> !<br>
> router-id 1.1.1.1<br>
> !<br>
> router bgp 6500<br>
> bgp router-id 1.1.1.1<br>
> neighbor 10.37.0.17 remote-as 6500<br>
> !<br>
> address-family ipv4 unicast<br>
> maximum-paths 1<br>
> exit-address-family<br>
> !<br>
> address-family ipv6 unicast<br>
> maximum-paths 1<br>
> exit-address-family<br>
> !<br>
> line vty<br>
> !<br>
> end<br>
> dcl4#<br>
><br>
><br>
> My summary analysis after core analysis and code walk is:<br>
> While we do handle zebra:main.c:sigint():<br>
> After vrf_terminate() is called, we are still holding routes in FPM outbound queue.<br>
> And these routes need full context of route_node/route_info pointers, which are<br>
> freed in 'vrf_terminate()' call above:<br>
><br>
><br>
> ==========================================================<br>
> /* SIGINT handler. */<br>
> static void sigint(void)<br>
> {<br>
> struct vrf *vrf;<br>
> struct zebra_vrf *zvrf;<br>
> struct listnode *ln, *nn;<br>
> struct zserv *client;<br>
> static bool sigint_done;<br>
><br>
> if (sigint_done)<br>
> return;<br>
><br>
> sigint_done = true;<br>
><br>
> zlog_notice("Terminating on signal");<br>
><br>
> frr_early_fini();<br>
><br>
> zebra_dplane_pre_finish();<br>
><br>
> for (ALL_LIST_ELEMENTS(zrouter.client_list, ln, nn, client))<br>
> zserv_close_client(client);<br>
><br>
> zserv_close();<br>
> list_delete_all_node(zrouter.client_list);<br>
><br>
> zebra_ptm_finish();<br>
><br>
> if (retain_mode)<br>
> RB_FOREACH (vrf, vrf_name_head, &vrfs_by_name) {<br>
> zvrf = vrf->info;<br>
> if (zvrf)<br>
> SET_FLAG(zvrf->flags, ZEBRA_VRF_RETAIN);<br>
> }<br>
> if (zrouter.lsp_process_q)<br>
> work_queue_free_and_null(&zrouter.lsp_process_q);<br>
> vrf_terminate();<br>
> ns_walk_func(zebra_ns_early_shutdown);<br>
> zebra_ns_notify_close();<br>
><br>
> access_list_reset();<br>
> prefix_list_reset();<br>
> route_map_finish();<br>
> ......<br>
><br>
><br>
> ===================================<br>
><br>
> /*<br>
> * netlink_route_info_fill<br>
> *<br>
> * Fill out the route information object from the given route.<br>
> *<br>
> * Returns true on success and false on failure.<br>
> */<br>
> static int netlink_route_info_fill(netlink_route_info_t *ri, int cmd,<br>
> rib_dest_t *dest, struct route_entry *re)<br>
> {<br>
> struct nexthop *nexthop;<br>
> struct zebra_vrf *zvrf;<br>
><br>
> memset(ri, 0, sizeof(*ri));<br>
><br>
> ri->prefix = rib_dest_prefix(dest);<------Here also core seen sometimes<br>
> ri->af = rib_dest_af(dest);<--------------Core seen while looking up info,<br>
><br>
> ri->nlmsg_type = cmd;<br>
> zvrf = rib_dest_vrf(dest);<br>
> if (zvrf)<br>
><br>
> ==================Zebra Logs===================(may not match exact code, but<br>
> this is a typical debug log pattern), I clearly see this assert is after vrf's are freed.<br>
> =============================================<br>
> oot@dorado-colo-leaf4:~# thongal/tail-rs.sh dcl4<br>
> 2019/11/17 17:57:25.411522 ZEBRA: [EC 100663304] ERROR: No such command on config line 6: ptm-enable<br>
> 2019/11/17 17:57:25.440060 ZEBRA: client 29 says hello and bids fair to announce only bfd routes vrf=0<br>
> 2019/11/17 17:57:25.468711 ZEBRA: client 34 says hello and bids fair to announce only static routes vrf=0<br>
> 2019/11/17 17:57:28.142658 ZEBRA: client 40 says hello and bids fair to announce only bgp routes vrf=0<br>
> 2019/11/17 17:57:42.226148 ZEBRA: if_zebra_speed_update: eth0.500 old speed: 0 new speed: 10000<br>
> 2019/11/17 17:57:44.910151 ZEBRA: if_zebra_speed_update: eth1.4092 old speed: 0 new speed: 10000<br>
><br>
><br>
> 2019/11/17 18:02:05.107662 ZEBRA: connection closed socket [40]<br>
> 2019/11/17 18:02:05.107729 ZEBRA: [EC 4043309116] Client 'bgp' encountered an error and is shutting down.<br>
> 2019/11/17 18:02:05.107825 ZEBRA: Closing client 'bgp'<br>
> 2019/11/17 18:02:05.107926 ZEBRA: release_daemon_table_chunks: Released 0 table chunks<br>
> 2019/11/17 18:02:05.107966 ZEBRA: zebra/zebra_ptm.c:1441 failed to find process pid registration<br>
> 2019/11/17 18:02:05.108017 ZEBRA: client 40 disconnected. 0 bgp routes removed from the rib<br>
> 2019/11/17 18:02:05.504982 ZEBRA: Terminating on signal<br>
> 2019/11/17 18:02:05.505038 ZEBRA: Zebra dataplane pre-fini called<br>
> 2019/11/17 18:02:05.505170 ZEBRA: Closing client 'bfd'<br>
> 2019/11/17 18:02:05.505249 ZEBRA: release_daemon_table_chunks: Released 0 table chunks<br>
> 2019/11/17 18:02:05.505310 ZEBRA: client 29 disconnected. 0 bfd routes removed from the rib<br>
> 2019/11/17 18:02:05.505496 ZEBRA: Closing client 'static'<br>
> 2019/11/17 18:02:05.505566 ZEBRA: release_daemon_table_chunks: Released 0 table chunks<br>
> 2019/11/17 18:02:05.505657 ZEBRA: rib_delnode: 0:<a href="http://92.10.1.0/24" rel="noreferrer" target="_blank">92.10.1.0/24</a>: rn 0x1cbf3c0, re 0x1cbee40, removing<br>
> 2019/11/17 18:02:05.505691 ZEBRA: rib_delnode: 0:<a href="http://93.10.1.0/24" rel="noreferrer" target="_blank">93.10.1.0/24</a>: rn 0x1cbf620, re 0x1cbf4c0, removing<br>
> 2019/11/17 18:02:05.505711 ZEBRA: rib_delnode: 0:<a href="http://94.10.1.0/24" rel="noreferrer" target="_blank">94.10.1.0/24</a>: rn 0x1cbf880, re 0x1cbf720, removing<br>
> 2019/11/17 18:02:05.505731 ZEBRA: client 34 disconnected. 3 static routes removed from the rib<br>
> 2019/11/17 18:02:05.505870 ZEBRA: vrf_terminate: Shutting down vrf subsystem<br>
> 2019/11/17 18:02:05.505903 ZEBRA: VRF 0 is to be deleted.<br>
> 2019/11/17 18:02:05.505930 ZEBRA: VRF 0 is to be disabled.<br>
> 2019/11/17 18:02:05.505955 ZEBRA: VRF default id 0 is now inactive(deleting routes, but not freeing)<br>
> 2019/11/17 18:02:05.505988 ZEBRA: MESSAGE: ZEBRA_VRF_DELETE default<br>
> 2019/11/17 18:02:05.506052 ZEBRA: VRF default id 0 deleted<br>
> 2019/11/17 18:02:05.506095 ZEBRA: netlink_route_multipath(): RTM_DELROUTE <a href="http://92.10.1.0/24" rel="noreferrer" target="_blank">92.10.1.0/24</a> vrf 0(254)<br>
> 2019/11/17 18:02:05.506126 ZEBRA: rib_unlink: 0:<a href="http://10.37.0.0/20" rel="noreferrer" target="_blank">10.37.0.0/20</a>: rn 0x1c9a9c0, re 0x1ab02e0<br>
> 2019/11/17 18:02:05.506156 ZEBRA: netlink_talk: netlink-dp (NS 0) type RTM_DELROUTE(25), len=44 seq=8 flags 0x401<br>
> 2019/11/17 18:02:05.506185 ZEBRA: rib_unlink: 0:<a href="http://92.10.1.0/24" rel="noreferrer" target="_blank">92.10.1.0/24</a>: rn 0x1cbf3c0, re 0x1cbee40<br>
> 2019/11/17 18:02:05.506212 ZEBRA: rib_unlink: 0:<a href="http://93.10.1.0/24" rel="noreferrer" target="_blank">93.10.1.0/24</a>: rn 0x1cbf620, re 0x1cbf4c0<br>
> 2019/11/17 18:02:05.506238 ZEBRA: netlink_route_multipath(): RTM_DELROUTE <a href="http://93.10.1.0/24" rel="noreferrer" target="_blank">93.10.1.0/24</a> vrf 0(254)<br>
> 2019/11/17 18:02:05.506272 ZEBRA: rib_unlink: 0:<a href="http://94.10.1.0/24" rel="noreferrer" target="_blank">94.10.1.0/24</a>: rn 0x1cbf880, re 0x1cbf720<br>
> 2019/11/17 18:02:05.506304 ZEBRA: netlink_talk: netlink-dp (NS 0) type RTM_DELROUTE(25), len=44 seq=9 flags 0x401<br>
> 2019/11/17 18:02:05.506338 ZEBRA: rib_unlink: 0:<a href="http://172.179.1.0/24" rel="noreferrer" target="_blank">172.179.1.0/24</a>: rn 0x1cd8020, re 0x1cd7f40<br>
> 2019/11/17 18:02:05.506366 ZEBRA: netlink_route_multipath(): RTM_DELROUTE <a href="http://94.10.1.0/24" rel="noreferrer" target="_blank">94.10.1.0/24</a> vrf 0(254)<br>
> 2019/11/17 18:02:05.506407 ZEBRA: netlink_talk: netlink-dp (NS 0) type RTM_DELROUTE(25), len=44 seq=10 flags 0x401<br>
> 2019/11/17 18:02:05.506444 ZEBRA: rib_unlink: 0:<a href="http://10.37.0.0/20" rel="noreferrer" target="_blank">10.37.0.0/20</a> (MRIB): rn 0x1cbed60, re 0x1cbec80<br>
> 2019/11/17 18:02:05.506479 ZEBRA: rib_unlink: 0:<a href="http://172.179.1.0/24" rel="noreferrer" target="_blank">172.179.1.0/24</a> (MRIB): rn 0x1cd8220, re 0x1cd8140<br>
> 2019/11/17 18:02:05.506576 ZEBRA: Zebra dataplane fini called<br>
> 2019/11/17 18:02:05.506637 ZEBRA: Zebra dataplane shutdown status check called<br>
> 2019/11/17 18:02:05.506667 ZEBRA: Failed to find route for ctx: no table for afi 1, safi 1, vrf 0<br>
> 2019/11/17 18:02:05.506706 ZEBRA: Failed to process dplane results: no route for 0:<a href="http://10.37.0.0/20" rel="noreferrer" target="_blank">10.37.0.0/20</a><br>
> 2019/11/17 18:02:05.506744 ZEBRA: Failed to find route for ctx: no table for afi 1, safi 1, vrf 0<br>
> 2019/11/17 18:02:05.506791 ZEBRA: Failed to process dplane results: no route for 0:<a href="http://92.10.1.0/24" rel="noreferrer" target="_blank">92.10.1.0/24</a><br>
> 2019/11/17 18:02:05.506822 ZEBRA: Failed to find route for ctx: no table for afi 1, safi 1, vrf 0<br>
> 2019/11/17 18:02:05.506848 ZEBRA: Failed to process dplane results: no route for 0:<a href="http://93.10.1.0/24" rel="noreferrer" target="_blank">93.10.1.0/24</a><br>
> 2019/11/17 18:02:05.506877 ZEBRA: Failed to find route for ctx: no table for afi 1, safi 1, vrf 0<br>
> 2019/11/17 18:02:05.506905 ZEBRA: Failed to process dplane results: no route for 0:<a href="http://94.10.1.0/24" rel="noreferrer" target="_blank">94.10.1.0/24</a><br>
> 2019/11/17 18:02:05.506935 ZEBRA: Failed to find route for ctx: no table for afi 1, safi 1, vrf 0<br>
> 2019/11/17 18:02:05.506964 ZEBRA: Failed to process dplane results: no route for 0:<a href="http://172.179.1.0/24" rel="noreferrer" target="_blank">172.179.1.0/24</a><br>
> ZEBRA: Received signal 11 at 1574042525 (si_addr 0x2bca8, PC 0x7f69611dd2a7); aborting...<br>
> Program counter: /usr/lib/frr/modules/zebra_fpm.so(+0x62a7)[0x7f69611dd2a7]<br>
> Backtrace for 19 stack frames:<br>
> /usr/lib/libfrr.so.0(zlog_backtrace_sigsafe+0x6d)[0x7f69644abd10]<br>
> /usr/lib/libfrr.so.0(zlog_signal+0x1a1)[0x7f69644abbee]<br>
> /usr/lib/libfrr.so.0(+0x8e10c)[0x7f69644dc10c]<br>
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f6963b52390]<br>
> /usr/lib/frr/modules/zebra_fpm.so(+0x62a7)[0x7f69611dd2a7]<br>
> /usr/lib/frr/modules/zebra_fpm.so(+0x62d5)[0x7f69611dd2d5]<br>
> /usr/lib/frr/modules/zebra_fpm.so(+0x635b)[0x7f69611dd35b]<br>
> /usr/lib/frr/modules/zebra_fpm.so(+0x6375)[0x7f69611dd375]<br>
> /usr/lib/frr/modules/zebra_fpm.so(+0x68d3)[0x7f69611dd8d3]<br>
> /usr/lib/frr/modules/zebra_fpm.so(zfpm_netlink_encode_route+0x6d)[0x7f69611de3f7]<br>
> /usr/lib/frr/modules/zebra_fpm.so(+0x3c10)[0x7f69611dac10]<br>
> /usr/lib/frr/modules/zebra_fpm.so(+0x3e12)[0x7f69611dae12]<br>
> /usr/lib/frr/modules/zebra_fpm.so(+0x4292)[0x7f69611db292]<br>
> /usr/lib/frr/modules/zebra_fpm.so(+0x4381)[0x7f69611db381]<br>
> /usr/lib/libfrr.so.0(thread_call+0xb3)[0x7f69644f11e2]<br>
> /usr/lib/libfrr.so.0(frr_run+0x229)[0x7f69644a8d53]<br>
> /usr/lib/frr/zebra(main+0x37e)[0x4262b0]<br>
> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f6963797830]<br>
> /usr/lib/frr/zebra(_start+0x29)[0x4129a9]<br>
> in thread zfpm_write_cb scheduled from zebra/zebra_fpm.c:482<br>
> 2019/11/17 18:02:06.458539 ZEBRA: client 29 says hello and bids fair to announce only bfd routes vrf=0<br>
> 2019/11/17 18:02:06.488917 ZEBRA: client 34 says hello and bids fair to announce only static routes vrf=0<br>
> 2019/11/17 18:02:06.575787 ZEBRA: connection to the FPM has gone down: closed socket in read<br>
> =============================================================<br>
> Any help appreciated, Do not see any bugs/issue filed on this yet on github.<br>
><br>
> Thanks<br>
> Thippanna<br>
> _______________________________________________<br>
> frog mailing list<br>
> <a href="mailto:frog@lists.frrouting.org" target="_blank">frog@lists.frrouting.org</a><br>
> <a href="https://lists.frrouting.org/listinfo/frog" rel="noreferrer" target="_blank">https://lists.frrouting.org/listinfo/frog</a><br>
</blockquote></div>