Subject: Re: Quagga Segment Routing implementation progress
From: Olivier Dugeon <[email-masked]>
To: Roopa Prabhu <[email-masked]>
Cc: Vivek Venkatraman <[email-masked]>, Renato Westphal <[email-masked]>, Donald Sharp <[email-masked]>, Lou Berger <[email-masked]>, Dinesh Dutt <[email-masked]>, Daniel Walton <[email-masked]>, Alistair Woodman <[email-masked]>, Martin Winter <[email-masked]>, "David Ahern" <[email-masked]>
Date: Fri, 29 Jul 2016 19:41:53 +0200

Hello Roopa,

Finally, we run into a strange issue. We don't know where the problem is but, it seems that the 3rd label on the stack is corrupted. In fact, the 3rd label, and only this one (we try all sort of combination from 1 to 8 labels stack) is mask with 0x03. So, once configured with iproute2 or Quagga, what you read with iproute2 (ip route list command) is a third label with the lower 8bits mask with 0x03. If we try a ping, a tcpdump show that the label is also corrupted. So, it is store in the linux kernel as corrupted, and thus, it is not a problem of reading it with ip route command.

Now, we don't know if the problem is located at the ioctl when you attempt to push a label (as the problem occurs with both iproute and Quagga, it seems not a problem a the user layer) or in the kernel.

For the moment, we circumvent the issue by using label that end in 0x03 ;-)

Can you have a look at that bug ?

Regards,

Olivier

Le 18/07/2016 20:08, Roopa Prabhu a écrit :
>
>
> On Mon, Jul 18, 2016 at 2:19 AM, <[email-masked] <mailto:[email-masked]>> wrote:
>
>     Hello Roopa, Vivek,
>
>     Thanks for the pointer and information.
>
>     Up to know we setup our SR implementation in PHP mode by default and waiting for new MPLS version that support LER egress.
>     Feel free to contact us if you need help, such as performing tests.
>
>     Concerning the label stack depth, can you confirm that it is sufficient to change the value of MAX_NEW_LABELS
>     in net/mpls/internal.h file and recompile the kernel to increase the number of labels in the stack ?
>
>
>
> That should work. Let us know if you run into any issues.
>
>
>  
>
>
>
>     Le 14/07/2016 05:29, Roopa Prabhu a écrit :
>>
>>
>>     On Wed, Jul 13, 2016 at 10:45 AM, Vivek Venkatraman <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>         Hi Olivier,
>>
>>         LER support is there in the kernel, the only limitation that I'm aware of is that after terminating the label, the packet would get forwarded to a nexthop/device and there isn't a way yet to do another lookup (e.g., on IP destination) or locally consume it.
>>
>>
>>     yes, vivek is right. And we are planning to fix that limitation soon. 
>>
>>     if you need examples of configuring LER routes you can find them in the slides here: http://www.netdevconf.org/1.1/proceedings/slides/prabhu-mpls-tutorial.pdf
>>      
>>
>>
>>         The above info may be dated as it has been a little while since I worked on MPLS. I'll discuss further with Roopa and either one of us will respond with additional info on this as well as your other question.
>>
>>         Vivek
>>
>>         On Wed, Jul 13, 2016 at 8:15 AM, <[email-masked] <mailto:[email-masked]>> wrote:
>>
>>             Hello Vivek,
>>
>>             Thanks for your answer. Do you know if there is plane to add LER support for the linux kernel ? If not, how we could help ?
>>
>>             Concerning the stack depth limitation, is it sufficient to change the MAX_NEW_LABELS define to let say 8 instead of 2 in net/mpls/internal.h file ? or do we need more modification ?
>>
>>             Regards
>>
>>             Olivier
>>
>>             Le 13/07/2016 07:14, Vivek Venkatraman a écrit :
>>>             Hi Olivier,
>>>
>>>             First of all, delighted to hear the progress of OSPF-SR.
>>>
>>>             I am copying my colleague Roopa who did the kernel MPLS implementation and will be able to provide more info. See inline for my take.
>>>
>>>             On Tue, Jul 12, 2016 at 6:35 AM, <[email-masked] <mailto:[email-masked]>> wrote:
>>>
>>>                 Dear all,
>>>
>>>                 I change the subject to continue exchanging on MPLS support and our implementation of Segment Routing.
>>>
>>>                 First, we progress well. Up to now we have a full implementation on top of OSPF. We could exchange all Segment Routing information with Cisco IosXR: Segment Routing Global Block, Node SID, Adjacency SID, Router Capabilities ... through Router Info, Extended Link and Extended Prefix new Opaque LSA.
>>>
>>>                 Based on latest Renato LDP code, we also implement the MPLS configuration in the Linux Kernel (4.5.4). MPLS FIB is correctly instantiated and packet are correctly forwarded.
>>>
>>>                 However, we ran into 2 problems which are more related to how MPLS has been implemented in the Linux Kernel rather than a Quagga problem. But, perhaps you could help us.
>>>
>>>                 First, our setup is as follow:
>>>
>>>                  ---------          --------         ---------
>>>                  [IOS-XR1] <------> [Quagga] <-----> [IOS-XR2]
>>>                  ---------          --------         ---------
>>>                  Loopback0           dummy0          Loopback0
>>>                 192.168.1.2       192.168.1.1       192.168.1.3
>>>                  SID: 200           SID: 100          SID: 300
>>>
>>>                 With a common SRGB fix at 20000. Both IOS-XR announced their respective SID with the PHP (Penultimate Host Poping) attribute. Quagga router uses 2 physical Ethernet interfaces and one dummy interface for the loopback.
>>>
>>>                 When IOS-XR1 (or IOS-XR2) ping IOS-XR2 (respectively IOS-XR1) Loopback, the packets are transmit with an MPLS label 20300 (respectively 20200) to the Quagga which POP the Label before forwarding the IP packets to the IOS-XR2 (respectively IOS-XR1). Then IOS-XR2 (or IOS-XR1) sends ICMP reply packets with an MPLS label 20200 (or 20300) to the Quagga which again POP the label before delivery them to the IOS-XR1 (or IOS-XR2). All works well in these cases.
>>>
>>>                 Now, if we try to ping the Quagga Loopback from one of the IOS-XR, it doesn't work. Cisco router sends ICMP request with an MPLS label 20100. Quagga router POP the label before delivery them to the dummy interface, which don't react and never sends back ICMP reply message. If we try to send ping from the Quagga to the IOS-XR, ICMP request reach the Cisco which sends back ICMP reply with the 20100 label, pop by the interface before delivery them to the dummy interface which doesn't react again. If we configure the Quagga to announce its SID with PHP enable, this time the Cisco routers send their ICMP request message without MPLS label and it is working well letting us thinking that there is a problem with dummy interface. We try a tun/tap and physical dedicated Ethernet interface instead of dummy interface for the Loopback, but it the same.
>>>
>>>                 So the problem seems to be related to the Linux Kernel MPLS implementation, but we are don't sure if it is:
>>>
>>>                 1/ an MPLS FIB configuration problem: We try several configuration (through iproute2)
>>>
>>>                  - 15100 dev lo ==> packets are POP then delivered to the lo interface
>>>                  - 15100 dev xxx ==> packet are POP then delivered to the xxx (dummy0,  ehternet or tun/tap interface) interface
>>>                  - 15100 via inet 192.168.1.1 dev xxx ==> same above with next-hop IP address i.e. the Loopback address
>>>
>>>                 2/ an unsupported MPLS configuration in the Linux kernel as it is working with PHP enable. It is hard to find mpls example on the web, but perhaps you have some working example with and without PHP.
>>>
>>>
>>>             The kernel implementation started out with just LSR support - so swap or pop-and-forward. For LER, I believe the egress (terminating side) behavior still relies on pop-and-forward and cannot yet fully do a pop-and-lookup/consume. This is why your test works fine with PHP but does not work when PHP is not enabled.
>>>
>>>             One question though - isn't PHP the most common scenario?
>>>
>>>              
>>>
>>>
>>>                 3/ a problem related to the Loopback. Do you have some advise about how to configure a Loopback Interface in Quagga ? Is it preferable to use a dummy device ? a tun/tap ? a physical Ethernet or a virtual Ethernet one ?
>>>
>>>                 In addition, we don't find clearly in the Linux Kernel nor in iproute2 program which is the maximum label stack. When trying some manual configuration with iproute2, it is not possible to stack more than 2 labels. Do you know if it is possible to increase this limitation ? if yes where ? in Linux Kernel ? in iproute2 ? both ?
>>>
>>>
>>>             It is the kernel that is limiting it to 2. I don't think there is any penalty incurred to increasing it. Keep in mind that the labels are "flat" i.e., there isn't any explicit hierarchy support.
>>>              
>>>
>>>
>>>                 Thanks for your help
>>>
>>>                 Olivier
>>>
>>>
>>>                 -- 
>>>                 logo Orange <http://www.orange.com>
>>>
>>>                  
>>>
>>>                 Olivier Dugeon
>>>                 Orange Expert, Future Networks
>>>                 Open Source Referent
>>>                 Orange/IMT/OLN/WTC/IEE/OPEN
>>>
>>>                  
>>>
>>>                 fixe : +33 2 96 05 28 80 <tel:%2B33%202%2096%2005%2028%2080>
>>>                 mobile : +33 6 82 90 37 85 <tel:%2B33%206%2082%2090%2037%2085>
>>>                 [email-masked] <mailto:[email-masked]>
>>>
>>>                  
>>>
>>>
>>
>>             _________________________________________________________________________________________________________________________
>>
>>             Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
>>             pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
>>             a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
>>             Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>>
>>             This message and its attachments may contain confidential or privileged information that may be protected by law;
>>             they should not be distributed, used or copied without authorisation.
>>             If you have received this email in error, please notify the sender and delete this message and its attachments.
>>             As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>>             Thank you.
>>
>>
>>
>
>     _________________________________________________________________________________________________________________________
>
>     Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
>     pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
>     a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
>     Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.
>
>     This message and its attachments may contain confidential or privileged information that may be protected by law;
>     they should not be distributed, used or copied without authorisation.
>     If you have received this email in error, please notify the sender and delete this message and its attachments.
>     As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
>     Thank you.
>
>