Adding SQLite as a new optional dependency
Hi all, As part of the work in the new northbound architecture, I implemented support for configuration rollbacks in FRR. Here's a short demo: https://gist.github.com/rwestphal/caedd9f3e4888cab2d631cc21967e93d To make this work, FRR needs to keep track of all committed configuration transactions in non-volatile memory (so they persist across reboots). One possibility to implement this would be to store the transactions using text files in the filesystem. The problem with this approach is that it would be necessary to write a lot of code to do things correctly, handling all possible errors (lots of things can go wrong). Using SQLite we have database transactions, which make things much easier (either all operations succeed or none do). SQLite in general provides a much better interface to store and retrieve transactions, not to mention its high-performance properties. I consider it to be the right tool for the job. But if anyone has an objection against using SQLite in FRR please speak up. Initially SQLite should be added as an optional dependency tied to the --enable-config-rollbacks configuration option. This way users not interested on configuration rollbacks won't need to install the new dependency. We might revisit this in the future if we find other use cases for SQLite in FRR. When FRR is built with --enable-config-rollbacks, all daemons will have a new option to specify the path of the daemon-specific database file: --db-file (the default is ${localstatedir}/<daemon-name>.db). The special :memory: filename can be used to open a new in-memory database (which doesn't persist across reboots or daemon restarts). I should say that configuration rollbacks will only work for daemons converted to the new northbound model. Speaking of which, the Northbound API is almost ready to be submitted as a PR. I'm finishing to polish the code and writing documentation, my estimation is that everything should be ready within the following two weeks. Regards, -- Renato Westphal
It seems like an SQL database is a bit overkill for configuration file rollback.... On Mon, Jun 18, 2018 at 10:41 AM, Renato Westphal < renato@opensourcerouting.org> wrote:
Hi all,
As part of the work in the new northbound architecture, I implemented support for configuration rollbacks in FRR. Here's a short demo: https://gist.github.com/rwestphal/caedd9f3e4888cab2d631cc21967e93d
To make this work, FRR needs to keep track of all committed configuration transactions in non-volatile memory (so they persist across reboots). One possibility to implement this would be to store the transactions using text files in the filesystem. The problem with this approach is that it would be necessary to write a lot of code to do things correctly, handling all possible errors (lots of things can go wrong). Using SQLite we have database transactions, which make things much easier (either all operations succeed or none do). SQLite in general provides a much better interface to store and retrieve transactions, not to mention its high-performance properties. I consider it to be the right tool for the job. But if anyone has an objection against using SQLite in FRR please speak up.
Initially SQLite should be added as an optional dependency tied to the --enable-config-rollbacks configuration option. This way users not interested on configuration rollbacks won't need to install the new dependency. We might revisit this in the future if we find other use cases for SQLite in FRR.
When FRR is built with --enable-config-rollbacks, all daemons will have a new option to specify the path of the daemon-specific database file: --db-file (the default is ${localstatedir}/<daemon-name>.db). The special :memory: filename can be used to open a new in-memory database (which doesn't persist across reboots or daemon restarts).
I should say that configuration rollbacks will only work for daemons converted to the new northbound model. Speaking of which, the Northbound API is almost ready to be submitted as a PR. I'm finishing to polish the code and writing documentation, my estimation is that everything should be ready within the following two weeks.
Regards, -- Renato Westphal
_______________________________________________ dev mailing list dev@lists.frrouting.org https://lists.frrouting.org/listinfo/dev
Hi JR Rivers, Thanks for your input. I tend to agree with you in the sense that we could do the same thing using regular text files, thus a new dependency isn't strictly necessary. My main motivation to use SQLite was to solve the problem using higher level programming constructs, allowing us to write more expressive code which is both easier to understand and less error prone. Certain operations, like implementing a ring buffer of transactions, can be implemented using a simple SQL trigger when the database is initialized [1]. Likewise, reducing the maximum number of stored transactions ("configuration database max-transactions)" can be implemented using a single SQL query [2]. Using regular text files, we'd need to loop through a directory, identify the oldest transactions and remove them one by one. Using regular text files we'd also need to define where and how to store the metadata associated to each configuration transaction, kind of reinventing the wheel by implementing a primitive database. Since SQLite is a small library, I didn't consider adding it as an optional dependency to be a big issue (we already have python as a build dependency, and python depends on SQLite). But I'll take your opinion into consideration and reevaluate the idea of using text files to store configuration transactions. I'll ping other developers looking for suggestions. [1] https://gist.github.com/rwestphal/457e9a763b8b24dfa367e239b7fc2979 [2] https://gist.github.com/rwestphal/a3afbbe8432bceb4edfd4e99da049227 Regards, Renato. On Mon, Jun 18, 2018 at 1:59 PM, JR Rivers <jrrivers@cumulusnetworks.com> wrote:
It seems like an SQL database is a bit overkill for configuration file rollback....
On Mon, Jun 18, 2018 at 10:41 AM, Renato Westphal <renato@opensourcerouting.org> wrote:
Hi all,
As part of the work in the new northbound architecture, I implemented support for configuration rollbacks in FRR. Here's a short demo: https://gist.github.com/rwestphal/caedd9f3e4888cab2d631cc21967e93d
To make this work, FRR needs to keep track of all committed configuration transactions in non-volatile memory (so they persist across reboots). One possibility to implement this would be to store the transactions using text files in the filesystem. The problem with this approach is that it would be necessary to write a lot of code to do things correctly, handling all possible errors (lots of things can go wrong). Using SQLite we have database transactions, which make things much easier (either all operations succeed or none do). SQLite in general provides a much better interface to store and retrieve transactions, not to mention its high-performance properties. I consider it to be the right tool for the job. But if anyone has an objection against using SQLite in FRR please speak up.
Initially SQLite should be added as an optional dependency tied to the --enable-config-rollbacks configuration option. This way users not interested on configuration rollbacks won't need to install the new dependency. We might revisit this in the future if we find other use cases for SQLite in FRR.
When FRR is built with --enable-config-rollbacks, all daemons will have a new option to specify the path of the daemon-specific database file: --db-file (the default is ${localstatedir}/<daemon-name>.db). The special :memory: filename can be used to open a new in-memory database (which doesn't persist across reboots or daemon restarts).
I should say that configuration rollbacks will only work for daemons converted to the new northbound model. Speaking of which, the Northbound API is almost ready to be submitted as a PR. I'm finishing to polish the code and writing documentation, my estimation is that everything should be ready within the following two weeks.
Regards, -- Renato Westphal
_______________________________________________ dev mailing list dev@lists.frrouting.org https://lists.frrouting.org/listinfo/dev
-- Renato Westphal
What's about using libgit2 ? https://libgit2.github.com/docs/guides/101-samples/
Given the state of hard drives on whitebox switches, I think we need to carefully evaluate doing anything that might further involve a file system. donald On Mon, Jun 18, 2018 at 8:03 PM, Vincent Jardin <vincent.jardin@6wind.com> wrote:
What's about using libgit2 ? https://libgit2.github.com/docs/guides/101-samples/
_______________________________________________ dev mailing list dev@lists.frrouting.org https://lists.frrouting.org/listinfo/dev
@Vincent Thanks for your suggestion. I think libgit2 could be a good fit in the sense that we would store only the deltas instead of full configurations. But other than that, I think libgit2 would introduce more complexity and would bring other challenges to solve. I'll analyze this more carefully tomorrow. @Donald Yes, Martin raised the same concern last time I talked to him. Using the SQLite solution this wouldn't be a problem since SQLite supports in-memory databases. As a matter of fact, this shouldn't be a problem for any solution we choose as long as we store the transactions on a tmpfs directory. On Mon, Jun 18, 2018 at 9:10 PM, Donald Sharp <sharpd@cumulusnetworks.com> wrote:
Given the state of hard drives on whitebox switches, I think we need to carefully evaluate doing anything that might further involve a file system.
donald
On Mon, Jun 18, 2018 at 8:03 PM, Vincent Jardin <vincent.jardin@6wind.com> wrote:
What's about using libgit2 ? https://libgit2.github.com/docs/guides/101-samples/
_______________________________________________ dev mailing list dev@lists.frrouting.org https://lists.frrouting.org/listinfo/dev
-- Renato Westphal
Hello Renato, Donald, As we need a Database which is suitable for the embedded constraints, why not also looking to Berkley DB or UnQlite. Both of them are written in C, simple to link and store all the database in a single file and of course are design for the embedded devices. my 2 cts, Olivier Le 19/06/2018 à 03:27, Renato Westphal a écrit :
@Vincent Thanks for your suggestion. I think libgit2 could be a good fit in the sense that we would store only the deltas instead of full configurations. But other than that, I think libgit2 would introduce more complexity and would bring other challenges to solve. I'll analyze this more carefully tomorrow.
@Donald Yes, Martin raised the same concern last time I talked to him. Using the SQLite solution this wouldn't be a problem since SQLite supports in-memory databases. As a matter of fact, this shouldn't be a problem for any solution we choose as long as we store the transactions on a tmpfs directory.
On Mon, Jun 18, 2018 at 9:10 PM, Donald Sharp <sharpd@cumulusnetworks.com> wrote:
Given the state of hard drives on whitebox switches, I think we need to carefully evaluate doing anything that might further involve a file system.
donald
On Mon, Jun 18, 2018 at 8:03 PM, Vincent Jardin <vincent.jardin@6wind.com> wrote:
What's about using libgit2 ? https://libgit2.github.com/docs/guides/101-samples/
_______________________________________________ dev mailing list dev@lists.frrouting.org https://lists.frrouting.org/listinfo/dev
_________________________________________________________________________________________________________________________ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci. This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you.
IMO sqlite is a good fit here. - The codebase overhead is extremely low (sqlite amalgamation is 1 source file, so we don’t even need a new external dependency) - As Renato mentioned, sqlite supports in-memory databases and I’ve had positive experiences using them myself - sqlite is rock solid, well understood, easy to debug, and fast enough for our purposes My thoughts on the other suggestions in this thread: - libgit2 - agree with Renato, this library has a massive amount of complexity compared to the sliver of functionality we’d be using - BerkelyDB - overkill for this task and a big (as in size) dependency - unqlite - looks neat but I see no reason to use this over sqlite, which has much larger traction My 2c.
On Jun 19, 2018, at 4:17 AM, <olivier.dugeon@orange.com> <olivier.dugeon@orange.com> wrote:
Hello Renato, Donald,
As we need a Database which is suitable for the embedded constraints, why not also looking to Berkley DB or UnQlite.
Both of them are written in C, simple to link and store all the database in a single file and of course are design for the embedded devices. my 2 cts,
Olivier
Le 19/06/2018 à 03:27, Renato Westphal a écrit :
@Vincent Thanks for your suggestion. I think libgit2 could be a good fit in the sense that we would store only the deltas instead of full configurations. But other than that, I think libgit2 would introduce more complexity and would bring other challenges to solve. I'll analyze this more carefully tomorrow.
@Donald Yes, Martin raised the same concern last time I talked to him. Using the SQLite solution this wouldn't be a problem since SQLite supports in-memory databases. As a matter of fact, this shouldn't be a problem for any solution we choose as long as we store the transactions on a tmpfs directory.
On Mon, Jun 18, 2018 at 9:10 PM, Donald Sharp <sharpd@cumulusnetworks.com> <mailto:sharpd@cumulusnetworks.com> wrote:
Given the state of hard drives on whitebox switches, I think we need to carefully evaluate doing anything that might further involve a file system.
donald
On Mon, Jun 18, 2018 at 8:03 PM, Vincent Jardin <vincent.jardin@6wind.com> <mailto:vincent.jardin@6wind.com> wrote:
What's about using libgit2 ? https://libgit2.github.com/docs/guides/101-samples/ <https://libgit2.github.com/docs/guides/101-samples/>
_______________________________________________ dev mailing list dev@lists.frrouting.org <mailto:dev@lists.frrouting.org> https://lists.frrouting.org/listinfo/dev <https://lists.frrouting.org/listinfo/dev>
_________________________________________________________________________________________________________________________
Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration, Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
This message and its attachments may contain confidential or privileged information that may be protected by law; they should not be distributed, used or copied without authorisation. If you have received this email in error, please notify the sender and delete this message and its attachments. As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified. Thank you. _______________________________________________ dev mailing list dev@lists.frrouting.org https://lists.frrouting.org/listinfo/dev
Le 19/06/2018 à 20:41, Quentin Young a écrit :
My thoughts on the other suggestions in this thread:
- libgit2 - agree with Renato, this library has a massive amount of complexity compared to the sliver of functionality we’d be using
OK
- BerkelyDB - overkill for this task and a big (as in size) dependency
+1
- unqlite - looks neat but I see no reason to use this over sqlite, which has much larger traction
No comment on it.
participants (7)
-
Donald Sharp -
JR Rivers -
olivier.dugeon＠orange.com -
Quentin Young -
Renato Westphal -
Vincent Jardin -
Vincent JARDIN