[dev] Adding SQLite as a new optional dependency

[Renato Westphal]

Hi JR Rivers,

Thanks for your input. I tend to agree with you in the sense that we
could do the same thing using regular text files, thus a new
dependency isn't strictly necessary.

My main motivation to use SQLite was to solve the problem using higher
level programming constructs, allowing us to write more expressive
code which is both easier to understand and less error prone. Certain
operations, like implementing a ring buffer of transactions, can be
implemented using a simple SQL trigger when the database is
initialized [1]. Likewise, reducing the maximum number of stored
transactions ("configuration database max-transactions)" can be
implemented using a single SQL query [2]. Using regular text files,
we'd need to loop through a directory, identify the oldest
transactions and remove them one by one. Using regular text files we'd
also need to define where and how to store the metadata associated to
each configuration transaction, kind of reinventing the wheel by
implementing a primitive database.

Since SQLite is a small library, I didn't consider adding it as an
optional dependency to be a big issue (we already have python as a
build dependency, and python depends on SQLite). But I'll take your
opinion into consideration and reevaluate the idea of using text files
to store configuration transactions. I'll ping other developers
looking for suggestions.

[1] https://gist.github.com/rwestphal/457e9a763b8b24dfa367e239b7fc2979
[2] https://gist.github.com/rwestphal/a3afbbe8432bceb4edfd4e99da049227

Regards,
Renato.

On Mon, Jun 18, 2018 at 1:59 PM, JR Rivers <jrrivers at cumulusnetworks.com> wrote:
>
> It seems like an SQL database is a bit overkill for configuration file
> rollback....
>
> On Mon, Jun 18, 2018 at 10:41 AM, Renato Westphal
> <renato at opensourcerouting.org> wrote:
>>
>> Hi all,
>>
>> As part of the work in the new northbound architecture, I implemented
>> support for configuration rollbacks in FRR. Here's a short demo:
>> https://gist.github.com/rwestphal/caedd9f3e4888cab2d631cc21967e93d
>>
>> To make this work, FRR needs to keep track of all committed
>> configuration transactions in non-volatile memory (so they persist
>> across reboots). One possibility to implement this would be to store
>> the transactions using text files in the filesystem. The problem with
>> this approach is that it would be necessary to write a lot of code to
>> do things correctly, handling all possible errors (lots of things can
>> go wrong). Using SQLite we have database transactions, which make
>> things much easier (either all operations succeed or none do). SQLite
>> in general provides a much better interface to store and retrieve
>> transactions, not to mention its high-performance properties. I
>> consider it to be the right tool for the job. But if anyone has an
>> objection against using SQLite in FRR please speak up.
>>
>> Initially SQLite should be added as an optional dependency tied to the
>> --enable-config-rollbacks configuration option. This way users not
>> interested on configuration rollbacks won't need to install the new
>> dependency. We might revisit this in the future if we find other use
>> cases for SQLite in FRR.
>>
>> When FRR is built with --enable-config-rollbacks, all daemons will
>> have a new option to specify the path of the daemon-specific database
>> file: --db-file (the default is ${localstatedir}/<daemon-name>.db).
>> The special :memory: filename can be used to open a new in-memory
>> database (which doesn't persist across reboots or daemon restarts).
>>
>> I should say that configuration rollbacks will only work for daemons
>> converted to the new northbound model. Speaking of which, the
>> Northbound API is almost ready to be submitted as a PR. I'm finishing
>> to polish the code and writing documentation, my estimation is that
>> everything should be ready within the following two weeks.
>>
>> Regards,
>> --
>> Renato Westphal
>>
>> _______________________________________________
>> dev mailing list
>> dev at lists.frrouting.org
>> https://lists.frrouting.org/listinfo/dev
>
>



-- 
Renato Westphal