To add some resolution to this question, I was able to get this working with veth interfaces. Same FRR configuration as before, but with the following interface configuration on the VTEPs: /usr/sbin/ip link add dev br0 type bridge /usr/sbin/ip link add dev vxlan0 type vxlan id 1 local ${LOCAL_IP} dstport 4789 nolearning /usr/sbin/ip link add name vif0 type veth peer name vif1 # slave vxlan0 & vif0 to br0 /usr/sbin/ip link set dev vxlan0 master br0 /usr/sbin/ip link set dev vif0 master br0 # mark interfaces up /usr/sbin/ip link set up dev br0 /usr/sbin/ip link set up dev vxlan0 /usr/sbin/ip link set up dev vif0 /usr/sbin/ip link set up dev vif1 #add IP to vif1 /usr/sbin/ip addr add ${VXLAN_IP} dev vif1 On Sat, Jan 13, 2018 at 10:44 AM Henry Snow <henry.snow@nielsen.com> wrote:
I have a bare metal hosting provider that provides me with layer-3 networking to every server. For some applications, I require a layer-2 network spanning the servers. This network is for the applications running on the server OS, not via VM or container tennants.
I've been following this blog post: https://vincent.bernat.im/en/blog/2017-vxlan-bgp-evpn, using frr 3.1dev from git master from several nights ago. I believe I have the route reflector + vteps configured with respect to bgpd as described, but no routes are advertised. I do see bgp neighbors as expected.
When I run 'show evpn mac vni 100', I do not see any mac addresses. My suspicion is that it is because I'm not using a bridge, but assigning an IP address directly to the vxlan interface. Looking at the source code (my C is very rusty), it looks like it enumerates all the VNIs, finds the bridges they're slaved to, then enumerates the interfaces on the bridge. I'm looking at this: https://github.com/FRRouting/frr/blob/8eac4812601f069f14fc7950e10904c2ae0ae7...
Is there a way to configure my interfaces without a bridge (or is this a bad approach)? If a bridge and additional slaved interface is required, any suggestions on how to configure the bridge with no additional tenants on the servers?
Additional Thoughts: - using static unicast discovery, vxlan membership across servers works - adding a bridge interface and slaving the vxlan interface + a dummy interface, assigning an ip to the dummy did not seem to work - tried kernel 4.4 from elrepo, (tends to work better from previous docker overlay network experiences)
Config: compiled from commit c1240044fbf081bb7407b1449d3954e4b63fec9f on CentOS 7.4, with --enable-cumulus
Route Reflector: 10.40.1.131/31
bgpd.conf: router bgp 65000 bgp router-id 10.40.1.131 bgp cluster-id 10.40.1.131 bgp log-neighbor-changes no bgp default ipv4-unicast coalesce-time 1000 neighbor fabric peer-group neighbor fabric remote-as 65000 neighbor fabric update-source 10.40.1.131 neighbor fabric capability extended-nexthop bgp listen range 10.40.1.128/25 peer-group fabric ! address-family l2vpn evpn neighbor fabric activate neighbor fabric route-reflector-client exit-address-family !
layer-2 member server example: 10.40.1.129/31
bgpd.conf: router bgp 65000 bgp router-id 10.40.1.129 no bgp default ipv4-unicast coalesce-time 1000 neighbor fabric peer-group neighbor fabric remote-as 65000 neighbor fabric capability extended-nexthop neighbor 10.40.1.131 peer-group fabric ! address-family l2vpn evpn neighbor fabric activate advertise-all-vni exit-address-family !
vxlan configured with: # ip link vxlan100 type vxlan id 100 local 10.40.1.129 port 4789 nolearning # ip link set up dev vxlan100 # ip addr add 172.31.254.2/24 dev vxlan100
Thanks for any suggestions (or course corrections, as this is new to me)!
-- Henry Snow | Director, Site Reliability Engineering eXelate, a Nielsen company <http://exelate.com/> E: henry.snow@nielsen.com | M: 347.366.2700 <(347)%20366-2700> 675 6th Ave, 3rd Floor, New York, NY 10010
-- Henry Snow | Director, Site Reliability Engineering eXelate, a Nielsen company <http://exelate.com/> E: henry.snow@nielsen.com | M: 347.366.2700 675 6th Ave, 3rd Floor, New York, NY 10010