<div dir="ltr">Hi Lou, Philippe, All,<div><br></div><div>The PR that I submitted already addresses for the most part inter-subnet routing (i.e., bridge+router scenario) if employing asymmetric routing (<a href="https://tools.ietf.org/html/draft-ietf-bess-evpn-inter-subnet-forwarding">https://tools.ietf.org/html/draft-ietf-bess-evpn-inter-subnet-forwarding</a> section 4). [I say "for the most part" because some additional changes are needed for advertisement of gateway MACIP in the case of centralized gateway and a few other things.] Changes are of course needed for symmetric routing (section 5 of aforementioned draft). I'll describe both of these below. <br></div><div><br></div><div>At the end, I'll propose some thoughts on extending this for other EVPN encapsulation - specifically MPLS - to support traditional VPLS.</div><div><br></div><div>Asymmetric routing:</div><div><br></div><div>Here, we're dealing only with host routes and the ingress VTEP/NVE will route to the virtual subnet where the destination is, so that the egress VTEP/NVE only does bridging. In VxLAN terms, we're only dealing with L2 VNIs which need to be provisioned on all VTEPs. MACs are learnt against a VLAN through kernel notifications, mapped to a VxLAN/VNI and advertised. Likewise, neighbor entries (ARP/ND) are learnt on an SVI by listening to kernel notifications; the mapping to the VxLAN/VNI is straightforward and MACIP routes are originated using this L2 VNI. The logic on the receive side is straightforward too. The received RTs map to the VNI which maps to the VLAN. MAC routes would be installed into the FDB while MACIP routes would result in neighbor entries being created on the SVI (corresponding to the VLAN).</div><div><br></div><div>The above functionality is all present in the PR submitted. While the target of the PR was just EVPN for L2 with ARP suppression, it can accomplish routing too. Note that ARP suppression requires some additional functionality in the Linux kernel which Cumulus Networks is working to get into the upstream kernel.</div><div><br></div><div>The only additional provisioning we had planned to introduce was whether to advertise our SVI MAC or not - needed only on gateway devices. This was to be under "address-family l2vpn evpn":</div><div><br></div><div><font face="monospace, monospace" size="1">router bgp <as></font></div><div><font face="monospace, monospace" size="1"> address-family l2vpn evpn</font></div><div><font face="monospace, monospace" size="1"> advertise-default-gateway</font></div><div><br></div><div>However, I'll propose some changes to the provisioning at the end of this note.</div><div><br></div><div>Symmetric routing:</div><div><br></div><div>Clearly, this is more scalable and brings in the "inter-connect subnet" (L3 VNI). It also introduces the ability to do prefix routing with EVPN type-5 routes.</div><div><br></div><div>The L3 VNI is a parameter per tenant - i.e., per L3 VRF. This is planned to be the only required/mandatory configuration on top of what my PR introduces. The tenant (L3 VRF) configuration already exists and the L3 VNI was going to be added to it. The RD and RTs (for the tenant) could be auto-derived from this L3 VNI, but could optionally be configured. The planned configuration is/was:</div><div><br></div><div><font face="monospace, monospace" size="1">router bgp <as> vrf <tenant VRF></font></div><div><font face="monospace, monospace" size="1"> <any existing configuration such as "redistribute connected" or "network"></font></div><div><font face="monospace, monospace" size="1"> l2vpn evpn l3vni <vni></font></div><div><font face="monospace, monospace" size="1"> rd <RD></font></div><div><font face="monospace, monospace" size="1"> route-target <import | export | both> <RT></font></div><div><br></div><div>If the community decision is to configure the RD and RT configs as "vrf-policy" against the default VRF in BGP, the above will of course change.</div><div><br></div><div>The way symmetric routing operates is as follows. There is no change to advertisement or reception of MAC-only type-2 routes, these will only contain the L2 VNI. For MACIP type-2 routes, when the neighbor (ARP/ND) is learnt by listening to a kernel notification, the SVI that the entry is learnt on will be part of the tenant's VRF and that will provide the L3 VNI and L3 RTs. The RouterMAC extended community has to be added and the MAC will be derived from the interface corresponding to the L3 VNI (the "inter-connect subnet" interface). On the receive side, if the route has 2 VNIs, the MAC and Neighbor entry will be installed against the L2 VNI (if present locally) as before while the IP host route will be processed and imported into any L3 VRFs (BGP's RIB) that match its RTs. There is some special handling required because the next hop is the remote VTEP/NVE whose MAC should be set up as the received Router MAC.</div><div><br></div><div>What the above shows is that there isn't an explicit hierarchy of L2 VNIs (subnets) of a tenant to the tenant's L3 VRF...but it is present implicitly (the SVIs corresponding to those subnets will be assigned to the tenant's VRF).</div><div><br></div><div>For external routing, the plan is that by default, any routes in the L3 VRF (in BGP's RIB) will be advertised to EVPN peers as type-5 routes. The current thought is that this can be controlled using existing route-map constructs (TBD). Internal (i.e., EVPN) routes are already present in the L3 VRF (BGP's RIB) as mentioned above. Existing route-maps can be used to control how these are advertised externally - currently using VRF-lite BGP peerings, in future using L3VPN.</div><div><br></div><div>For inter-DC connectivity, EVPN single-hop or multi-hop peerings can be setup between the border EVPN routers in each DC. If some/all tenants do not need their L2 domain stretched across the DCs but only need L3 connectivity (i.e., subnets contained to one DC), only EVPN type-5 routes need to be exchanged on the inter-DC peering. The current plan is to implement an addition to route-map matching for that - "match evpn route-type <type>".</div><div><br></div><div><br></div><div>Extending/generalizing the provisioning for the non-VxLAN use case:</div><div><br></div><div>In the case of EVPN for VxLAN, a VLAN is mapped to a VxLAN (VNI) by the operator and whether it is a single broadcast domain per EVI or multiple broadcast domains per VNI, the VNI is sufficient to identify the bridge table as per section 5.1.2 of <a href="https://tools.ietf.org/html/draft-ietf-bess-evpn-overlay">https://tools.ietf.org/html/draft-ietf-bess-evpn-overlay</a>. This does lend itself to a rather simplified configuration for VxLAN that would be a big advantage to retain.</div><div><br></div><div>Whether EVPN should be used for VNIs or not (i.e., "advertise-all-vni" under BGP L2VPN/EVPN address-family configuration in my PR) should move to the entity (i.e., zebra) which creates/handles EVIs.</div><div><br></div><div>The term "vni" is specific to VxLAN and cannot be used for other EVPN. Our preference is for "evi" but it is up to the community to decide whether "evi", "vsi" or something else is the most appropriate.</div><div><br></div><div>For VxLAN, it is convenient to refer to the EVI (Ethernet Virtual Instance) by its VNI for the common case; for other cases, there is no such well-known identifier and the EVI is likely to be identified by name (just like a L3 VRF).</div><div><br></div><div>The proposed commands are as follows. These are initial thoughts subject to more refinement - partly because the Linux kernel does not currently have a forwarding model for L2oMPLS.</div><div><br></div><div><font face="monospace, monospace" size="1">l2vpn evpn advertise-vni <all | list of VNIs></font></div><div>-- The handler of this command will be "zebra" and it is in lieu of the "advertise-all-vni" command as stated above.</div><div>-- This only applies if using EVPN for VxLAN</div><div><br></div><div><font face="monospace, monospace" size="1">l2vpn evpn evi <name></font></div><div><font face="monospace, monospace" size="1"> encapsulation <vxlan | mpls></font></div><div><font face="monospace, monospace" size="1"> bridge-table <table | bridge-name></font></div><div><font face="monospace, monospace" size="1"> <any MPLS/label allocation parameters - if encap is mpls></font></div><div><font face="monospace, monospace" size="1"> <any VxLAN parameters - if encap is vxlan></font></div><div>-- The above syntax/commands will be used to create EVIs for MPLS, and if needed, for VxLAN.</div><div>-- The handler of these commands will be "zebra"</div><div><br></div><div><font face="monospace, monospace" size="1">router bgp <as></font></div><div><font face="monospace, monospace" size="1"> l2vpn evpn { vni <vni> | evi <name> }</font></div><div><font face="monospace, monospace" size="1"> rd <rd></font></div><div><font face="monospace, monospace" size="1"> route-target <import | export | both> <rt></font></div><div>-- The above syntax/commands will be used to define the RD/RT parameters for a VNI/EVI if the auto-derivation is not desired.</div><div>-- The handler of the above will clearly be "bgpd"</div><div><br></div><div>The L3 VNI configuration - which is against a L3 VRF - is as proposed earlier.</div><div><br></div><div>The "advertise-default-gateway" configuration for asymmetric routing can be modified based on the final consensus on the above.</div><div><br></div><div><br></div><div>Vivek</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jun 20, 2017 at 9:44 AM, Lou Berger <span dir="ltr"><<a href="mailto:lberger@labn.net" target="_blank">lberger@labn.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">see below.<br>
<br>
<br>
On 6/20/2017 11:42 AM, Philippe Guibert wrote:<br>
> Hi Lou, Vivek,<br>
><br>
> inline my comments,<br>
><br>
><br>
> On Mon, Jun 19, 2017 at 4:51 PM, Lou Berger <<a href="mailto:lberger@labn.net">lberger@labn.net</a>> wrote:<br>
>><br>
>> On 6/17/2017 1:25 AM, Vivek Venkatraman wrote:<br>
>>> Hi Lou and Philippe,<br>
>>><br>
>>> Besides the provisioning angle, how these entities map to current data<br>
>>> structures and code flow also is a consideration, IMO. Everything<br>
>>> about the current VRF ('struct bgp' or 'struct zebra_vrf' or soon to<br>
>>> be introduced support for OSPF) pertains to a Layer-3 routing<br>
>>> instance. I'm not sure it is a good idea to either morph that<br>
>>> construct into a Layer-2 (bridging) instance too or try to envelope a<br>
>>> Layer-2 instance and a Layer-3 instance under something else (like<br>
>>> "network-instance").<br>
>> What are your thoughts on when we get to bridge+router capability?<br>
>> Allowing for this is part of what drove my comments.<br>
>><br>
>> I have a fair bit of this in a prototype using rfapi and openflow, but<br>
>> not quete working or in sharable form. (Just need to find a few days to<br>
>> code!)<br>
>><br>
>>> My line of thinking after discussion with a few of my colleagues is as<br>
>>> follows:<br>
>>><br>
>>> 1. We should keep Layer-3 instance and Layer-2 instance configuration<br>
>>> distinct, though there may be some common parameters that both have.<br>
>> Again, we can't forget about the real case of bridge routers and cause<br>
>> us problems down the road.<br>
> I am not sure about that.<br>
> Right now, I don't have the exact use case that shows that. I will<br>
> give more information later.<br>
> What I mean is that there may be some instances that can get both L3<br>
> and L2 information.<br>
> As far as I know, this is the case for RT2 entries with two labels inside.<br>
from the bgp distribution side, there is both a (L2VPN) mac and (L3VPN)<br>
CE-advertised/PE-learned route to distribute in this case.<br>
<br>
><br>
>>> 2. While internal semantics indicate some parameters are "CE related"<br>
>>> and other parameters are "PE related", there are instances where it<br>
>>> can be argued either way. In any case, it would be easier for the<br>
>>> user/operator to have the configuration for one entity in one place,<br>
>>> as much as possible.<br>
>> I think your distinction is customer-facing (within VRF/VSI) and<br>
>> provider-facing (within core/provider transport), right?<br>
>><br>
>>> 3. The reference to "VNI" directly and some aspects of the<br>
>>> configuration syntax in my PR make it too specific to EVPN for VxLAN.<br>
>>> This should be made more generic.<br>
>> Does VSI work for you?<br>
> Does VSI covers MPLs labels ?<br>
I think of VSI = VRF for L2. So labels may or may be used based on config.<br>
<br>
Lou<br>
<br>
><br>
>>> 4. There are certain aspects though about EVPN for VxLAN that allow<br>
>>> for auto-creation and auto-derivation for which allowance has to be<br>
>>> made. For e.g., I'm sure an operator wouldn't want to configure a<br>
>>> named-instance for each VNI on the system when the VNI itself can be<br>
>>> the key and can be learnt by the routing control plane from the kernel.<br>
>>><br>
>>> With these in mind, I'll discuss a proposal with my colleagues before<br>
>>> bringing it here for further discussion next week.<br>
>>><br>
>> Excellent!<br>
> ++1<br>
><br>
> Philippe<br>
><br>
><br>
><br>
>>><br>
>>> On Thu, Jun 15, 2017 at 6:53 AM, Lou Berger <<a href="mailto:lberger@labn.net">lberger@labn.net</a><br>
>>> <mailto:<a href="mailto:lberger@labn.net">lberger@labn.net</a>>> wrote:<br>
>>><br>
>>> Philippe,<br>
>>><br>
>>> I agree with your analysis. One additional point bellow.<br>
>>><br>
>>><br>
>>> On 6/15/2017 9:18 AM, Philippe Guibert wrote:<br>
>>> > Hi Vivek,<br>
>>> ><br>
>>> > I just saw Lou's reply message.<br>
>>> > Initial agreement was using vrf-policy under bgp node.<br>
>>> > If I understand correctly, there is kind of redundancy with EVPN<br>
>>> > address-family vni configuration command used on CE side.<br>
>>> ><br>
>>> > Your remark is very interesting. I think it is worth looking at<br>
>>> how to<br>
>>> > fuse both configuration ways.<br>
>>> > Please find below some remarks on what could be done.<br>
>>> ><br>
>>> >> In addition, there are commands to configure the RD and RTs if<br>
>>> >> auto-derivation is not desired - for e.g., peering with<br>
>>> third-party BGP<br>
>>> >> system. The syntax for this is shown through an example<br>
>>> configuration below<br>
>>> >> (which also shows steps #1 and #2).<br>
>>> >><br>
>>> >> router bgp 65001<br>
>>> >> address-family l2vpn evpn<br>
>>> >> neighbor SPINE activate<br>
>>> >> vni 10100<br>
>>> >> rd 1:10100<br>
>>> >> route-target import 1:10100<br>
>>> >> exit-vni<br>
>>> >> advertise-all-vni<br>
>>> >> exit-address-family<br>
>>> >><br>
>>> >> I see it as being very useful to have all the configuration<br>
>>> relevant to a<br>
>>> >> VRF (or VNI) in one place.<br>
>>> > Thanks for point out that. I agree with you<br>
>>> ><br>
>>> >> One topic of discussion is regarding this optional VNI<br>
>>> configuration.<br>
>>> >> Instead of the keyword being "vni", should it be "vni-policy"<br>
>>> to match with<br>
>>> >> "vrf-policy"?<br>
>>> > If we want to have a common vty node to enter the information, I<br>
>>> would<br>
>>> > like to draw your attention to the following:<br>
>>> > - This vty node should be used for not only EVPN,but also VPNVx<br>
>>> > address families.<br>
>>> > The vni value used should be an attribute of that VPN object ( since<br>
>>> > vxlan does not apply to VPNVx afi/safi).<br>
>>> > This vty node should be moved from evpn address-family to bgp node.<br>
>>> ><br>
>>> > - This vty node stands for a VRF that should be used.<br>
>>> > It is true that in comparison to route-map concept, the "vrf-policy"<br>
>>> > wording could be improved.<br>
>>> > But we have to distinguish the vtynode from VRF node that is used<br>
>>> > outside of BGP node.<br>
>>> ><br>
>>> > Based on vrf-policy wording, the subnode vty commands should be<br>
>>> added<br>
>>> > vni keyword.<br>
>>> This would also work nicely for the future case where the VNI is a<br>
>>> bridge/router. it would just fall out by specifying a valid vrf<br>
>>> name in<br>
>>> the vrf-policy. Although this does lead to the slightly ugly case of<br>
>>> needing to rename the 'policy' when associating a running VNI/VSI<br>
>>> (bridge) with a VRF. perhaps it makes sense to uncouple of the<br>
>>> binding<br>
>>> of the "policy" with the VRF name and VSI/VNI ID. e.g.,<br>
>>><br>
>>><br>
>>> network-instance <node-name><br>
>>> !when associated with a named BGP VRF<br>
>>> vrf <vrf-name><br>
>>> !when associated with a vni<br>
>>> vni <vni-id><br>
>>> rd <value><br>
>>> rt (import|export|both) <value> [<value-list>]<br>
>>> label <value><br>
>>> route-map <mapname><br>
>>> tunnel advertisement-method <encap-attribute|evpn><br>
>>> tunnel type (none|l2tpv3overip|gre|ipinip|<wbr>vxlan|mpls|mplsovergre)<br>
>>><br>
>>><br>
>>> The name network-instance comes from<br>
>>> <a href="https://tools.ietf.org/html/draft-ietf-rtgwg-ni-model" rel="noreferrer" target="_blank">https://tools.ietf.org/html/<wbr>draft-ietf-rtgwg-ni-model</a><br>
>>> <<a href="https://tools.ietf.org/html/draft-ietf-rtgwg-ni-model" rel="noreferrer" target="_blank">https://tools.ietf.org/html/<wbr>draft-ietf-rtgwg-ni-model</a>><br>
>>><br>
>>> An alternative is to not add vrf/vni above and do something like<br>
>>> router bgp XXX vrf <vrf-name>*<br>
>>> network-instance <node-name><br>
>>><br>
>>> and under a new bgp vsi (I like the more generic name VSI over<br>
>>> VNI) node<br>
>>> vsi <vni-id><br>
>>> network-instance <node-name><br>
>>><br>
>>> but then it's up to the config reader to notice when something is<br>
>>> a bridge and/or router instance. (so I prefer the first<br>
>>><br>
>>><br>
>>> > Vivek, those changes have also wider impact, I mean, at least<br>
>>> > internally in the BGP daemon.<br>
>>> > I list some of the changes that may be done, if we merge [both vty<br>
>>> > nodes in a single one.<br>
>>> ><br>
>>> > - The VRF configuration calls VNC code, while VNI configuration<br>
>>> calls<br>
>>> > bgp_evpn_vty.c code.<br>
>>> > This should be put in a separate file. A registration mechanism with<br>
>>> > EVPN and VNC could apply.<br>
>>> Yes this would need to change.<br>
>>><br>
>>> > - Some other attributes of EVPN ( RD and RT auto derivation)<br>
>>> could be<br>
>>> > configurable within that new VRF instance.<br>
>>> ><br>
>>> > - Also, regarding the advertise-all-vni command, does that mean that<br>
>>> > such VNI ( VRFs objects ) should be instantiated too ?<br>
>>> > I mean, I am sorry, I did not attend the specific EVPN meeting you<br>
>>> > lead a few weeks ago. I know Lou was there.<br>
>>><br>
>>> > Perhaps you talked about the way to exchange VNI information between<br>
>>> > EVPN and VNC ?<br>
>>> We did, but all future (non blocking) stuff. The sole blocking issue<br>
>>> from my perspective is resolving this discussion.<br>
>>><br>
>>> Lou<br>
>>><br>
>>> > Regards,<br>
>>> ><br>
>>> > Philippe<br>
>>> ><br>
>>> ><br>
>>> > [0]<br>
>>> <a href="https://docs.google.com/document/d/1w_ie2tNXCgn0N3ZNFGYTK6lJkwMmk_XN5yz33MMNNqM/edit#" rel="noreferrer" target="_blank">https://docs.google.com/<wbr>document/d/1w_<wbr>ie2tNXCgn0N3ZNFGYTK6lJkwMmk_<wbr>XN5yz33MMNNqM/edit#</a><br>
>>> <<a href="https://docs.google.com/document/d/1w_ie2tNXCgn0N3ZNFGYTK6lJkwMmk_XN5yz33MMNNqM/edit#" rel="noreferrer" target="_blank">https://docs.google.com/<wbr>document/d/1w_<wbr>ie2tNXCgn0N3ZNFGYTK6lJkwMmk_<wbr>XN5yz33MMNNqM/edit#</a>><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> ><br>
>>> >> I don't think "vni-policy", or for that matter, "vrf-policy" is<br>
>>> the best<br>
>>> >> choice due to the following two main reasons:<br>
>>> >><br>
>>> >> 1. A "policy" is a fairly familiar construct in routing<br>
>>> parlance. It<br>
>>> >> commonly refers to a set of rules or definitions that are<br>
>>> generically<br>
>>> >> specified and can then be applied to different "attach points".<br>
>>> In FRR, a<br>
>>> >> "route-map" would be a good example of such a policy. It may be<br>
>>> misleading<br>
>>> >> to call the specific configuration for a VNI or VRF as "policy",<br>
>>> >> particularly when the VNI/VRF may later support import/export<br>
>>> policies<br>
>>> >> (route-maps).<br>
>>> >><br>
>>> >> 2. The configuration syntax that emerged for VRFs after the<br>
>>> last round of<br>
>>> >> discussions separates out the "CE side" configuration (e.g., CE<br>
>>> neighbors,<br>
>>> >> redistribution etc.) from the "PE side" configuration (e.g., RD<br>
>>> and RT<br>
>>> >> configuration). From the user/operator's perspective, I do not<br>
>>> see any value<br>
>>> >> add in this separation, only a potential source of confusion<br>
>>> since the same<br>
>>> >> entity (VRF) needs to be configured in multiple places. It can<br>
>>> also be<br>
>>> >> debatable where a configuration lies. For e.g., should<br>
>>> "vrf-import-policy"<br>
>>> >> reside on the "PE side" as it deals with received L3VPN routes<br>
>>> or on the "CE<br>
>>> >> side" as it decides which routes to import into which VRF table?<br>
>>> >><br>
>>> >> I see it as being very useful to have all the configuration<br>
>>> relevant to a<br>
>>> >> VRF (or VNI) in one place.<br>
>>> >><br>
>>> >> The purpose of this email is to solicit wider feedback since<br>
>>> only a few<br>
>>> >> people participated in the earlier discussion and their<br>
>>> positions are very<br>
>>> >> likely unchanged. My suggestion would be to have the initial<br>
>>> deliberations<br>
>>> >> on the list and if that does not converge or indicates the need<br>
>>> for a<br>
>>> >> meeting, the maintainers will call one.<br>
>>> >><br>
>>> >> Based on the consensus that emerges, I shall update my PR<br>
>>> and/or introduce<br>
>>> >> modifications in a subsequent PR - if needed.<br>
>>> >><br>
>>> >> Vivek<br>
>>> >><br>
>>> >><br>
>>> >> ______________________________<wbr>_________________<br>
>>> >> dev mailing list<br>
>>> >> <a href="mailto:dev@lists.frrouting.org">dev@lists.frrouting.org</a> <mailto:<a href="mailto:dev@lists.frrouting.org">dev@lists.frrouting.<wbr>org</a>><br>
>>> >> <a href="https://lists.frrouting.org/listinfo/dev" rel="noreferrer" target="_blank">https://lists.frrouting.org/<wbr>listinfo/dev</a><br>
>>> <<a href="https://lists.frrouting.org/listinfo/dev" rel="noreferrer" target="_blank">https://lists.frrouting.org/<wbr>listinfo/dev</a>><br>
>>> >><br>
>>><br>
>>><br>
<br>
</blockquote></div><br></div>