[dev] CLI fuzzer

[Renato Westphal]

Hi all,

Just would like to share a small tool I wrote to test FRR. It's a CLI
fuzzer, a program that tests all existing CLI commands over and over
again and detects segfaults when they happen.

Link: https://github.com/rwestphal/frr-cli-fuzzer

The README.me contains installation and usage instructions. The CLI
fuzzer is only capable of finding simple bugs, like NULL pointer
dereferences, but it's useful as an additional testing tool in our
toolbox.

I left the CLI fuzzer running overnight and it found these segfaults
in FRR: https://hastebin.com/qiyunedeja.txt

Lots of the detected bugs refer to different variations of the same
command (e.g. bgpd's "no neighbor [...] maximum-prefix"). Some bugs,
like the ones found in nhrpd and isisd, only happen seconds after
sending the commands that triggered them. When this happens, the
command reported by the CLI fuzzer has no correspondence to the actual
command that triggered the segfault. In these cases, it's necessary to
look at the generated core dumps to understand why the daemon aborted.
In the case of the nhrpd segfaults, for example, they all refer to the
same problem (a dangling pointer in the nhrp_peer_check() function).

I think the CLI fuzzer should be especially useful for everyone
working on the northbound retrofitting process. Since all
configuration commands will be modified, there's always the
possibility of introducing bugs in the process. The CLI fuzzer should
help us to catch some of these bugs early during development.

Later on we should also be able to write a YANG-based configuration
fuzzer, which would generate random configurations and feed them to
FRR using configuration replace semantics. That would allow us to test
FRR more extensively and find other kinds of problems.

Regards,
-- 
Renato Westphal