Setting up Enhanced vPC on Nexus 5600

So I’ve played around with two Cisco Nexus 5672UP switches which will be in production soon but before that I wanted to see what it took to get enhanced vPC online. Along the way I was into some new territory as I never stood up vPC before, so in this post I have posted some things to keep in mind and running-config examples, all of which I hope is helpful for a reference.

Before we go into the configuration of vPC I wanted to mentioned that the Nexus 5600 series switch is a newer product in the Cisco Data Center portfolio. I would guess that it would eventually replace the 5500 Nexus switch series but as of this writing Cisco has not put any end of life notice on the 5500 hardware. One of the main differences on the 5600 is the built-in layer 3 capabilities. The 5500 required a layer 3 card that used an expansion slot and only the higher-end Nexus products offered layer 3. To find out more information about the 5600 on Cisco’s website: Cisco Nexus 5600 Platform Switches Data Sheet

So when I was looking at configuration examples I noticed some confusion between a dual-homed FEX setup and an enhanced vPC setup, so I’ll try to clear the air. The “Enhanced” vPC is able to set up an etherchannel link from two different Nexus switches. The host/server that connects to these two different Nexus switches must support Active/Active NICs. This configuration is almost the same if you were setting up a dual-homed FEX but with just Active/Passive NICs setup on the host/server. So here is a question if the configuration is almost the same what needs to be added to support enhanced vPC?

The software I am working with is version 7.0(1)N1(1) and we need to enable the features of vpc, fex and lacp.

feature vpc
feature fex
feature lacp

We also need to have some VLANs created on both 5ks. In this example I created 25-35

vlan 25-35

We need to start a VPC domain, this must be the same on both switches.

SWITCH 1:

vpc domain 1
peer-keepalive destination 169.254.1.2 source 169.254.1.1

SWITCH 2:

vpc domain 1
peer-keepalive destination 169.254.1.1 source 169.254.1.2

A note about the peer-keepalive, the 5500/5600 Nexus has a dedicated management NIC card on the front of the unit. (Where the power is) The reason I did this was I did not want to waste a 10Gb connection for just a keep-alive link. (Although it can be done :/ )  In this example I don’t have a “dedicated” management network so I decided to use the management NIC of both 5ks. By default you have a another vrf context called management on the 5ks running-config. So on both 5ks I give that vrf context an IP address and directly connected both 5ks together.

SWITCH 1:

interface mgmt0
vrf member management
ip address 169.254.1.1/24

SWITCH 2:

interface mgmt0
vrf member management
ip address 169.254.1.2/24

Another helpful tip that I could not find out was if the management NICs would auto MDI/MDIX. So I just plugged in a straight-though and it works, so you don’t need a crossover cable if you don’t have one nearby. 🙂

We now need to create a vPC peer-link between the two Nexus switches. In this example I am using two 10Gb links, which are the same interfaces on both Nexus switches. (Consistency is your friend)

interface Ethernet1/31
  channel-group 1 mode active

interface Ethernet1/32
  channel-group 1 mode active

Both switches are referring to port-channel one for interfaces 31 and 32. In the port-channel we will trunk the vlans we have created earlier along with setting this port-channel into a vpc peer-link.

interface port-channel1
  switchport mode trunk
  switchport trunk allowed vlan 25-35
  vpc peer-link

To keep the config short I am only referring to two FEXes in this example. (Remember so far in this post you would do exactly the same configuration if you just wanted to set up dual-homed FEXes)

Starting with FEX 101 on SWITCH 1:

interface Ethernet1/1
  description FEX-101-PORT-1
  switchport mode fex-fabric
  fex associate 101
  channel-group 101

interface Ethernet1/2
  description FEX-101-PORT-3
  switchport mode fex-fabric
  fex associate 101
  channel-group 101

interface port-channel101
  switchport mode fex-fabric
  fex associate 101
  vpc 101

Continue with FEX 101 on SWITCH 2:

interface Ethernet1/1
  description FEX-101-PORT-2
  switchport mode fex-fabric
  fex associate 101
  channel-group 101

interface Ethernet1/2
  description FEX-101-PORT-4
  switchport mode fex-fabric
  fex associate 101
  channel-group 101

interface port-channel101
  switchport mode fex-fabric
  fex associate 101
  vpc 101

Start with FEX 102 on SWITCH 1:

interface Ethernet1/3
  description FEX-102-PORT-1
  switchport mode fex-fabric
  fex associate 102
  channel-group 102

interface Ethernet1/4
  description FEX-102-PORT-3
  switchport mode fex-fabric
  fex associate 102
  channel-group 102

interface port-channel102
  switchport mode fex-fabric
  fex associate 102
  vpc 102

Continue with FEX 102 on SWITCH 2:

interface Ethernet1/3
  description FEX-102-PORT-2
  switchport mode fex-fabric
  fex associate 102
  channel-group 102

interface Ethernet1/4
  description FEX-102-PORT-4
  switchport mode fex-fabric
  fex associate 102
  channel-group 102

interface port-channel102
  switchport mode fex-fabric
  fex associate 102
  vpc 102

So we have just setup two dual-homed FEXes on the 5672UP switches, here is where we can kick it up a notch to set up enhanced vPC. If I had a host that plugged into FEX 101 and FEX 102 and I wanted to set up etherchannel so that host could send traffic on out both links I would put the following configuration on BOTH switches:

interface eth101/1/1, eth101/1/2
channel-group 2 mode active
interface eth102/1/1, eth102/1/2
channel-group 2 mode active
interface po2
switchport access vlan 25

Here is the result, this is now an enhanced vPC. Some key points of this setup is be consistent with your configuration, it all has to match on both switches otherwise expect problems to arise. Enhanced vPC offers some high level redundancy, by just looking at the picture below you could have multiple failures and servers and applications would still be accessible. This makes this type of setup a common practice if business needs demand it. So I’ll stop it there like before I hope this information is helpful as a reference. Feel free to comment below if you have any questions.

vpc-nexus

Advertisements

14 thoughts on “Setting up Enhanced vPC on Nexus 5600

    • Ryan December 16, 2014 / 8:25 pm

      Hey Joel,
      Thanks for pointing that out I fixed the image and thanks for the reference.

  1. Zak Post January 14, 2015 / 7:04 am

    Thanks for this, it was easy to follow up to get a working configuration.

    • Ryan January 14, 2015 / 7:45 pm

      Glad you were able to use it. 🙂

  2. Kerry January 26, 2015 / 2:46 pm

    Thank you for this write up, great information.

    In my setup the host has 2 10g NICS for a total of 4 uplinks. Would I be able to follow your example here for PO2 but add in all 4 ports? (Ex 101/1/1-4 and 102/1/1-4

    • Ryan January 26, 2015 / 10:17 pm

      Hey Kerry,
      Yes you would be able to add 4 ports in a port-channel to the FEXes. You could almost do the same thing with two 5600s without the FEXes if you wanted. Just setup a VPC and port-channel on both 5600s instead of using the FEXes.

  3. John Naggets April 10, 2015 / 9:49 am

    What about using the QSFP ports instead of the Ethernet1/31 and Ethernet1/32 to connect both switches together using vPC, would that also work?

    • Ryan April 10, 2015 / 10:06 pm

      Hey John,
      Yes I would think that would work. However I don’t know if I would use the QSFP ports for a vPC link. Those type of ports would be better for uplinks. I don’t think you would be running that much traffic thorough a vPC link unless something is wrong. From what I looked at it seemed like you would want to balance it out. So for example if I had an uplink on 5k-1 to Core-1, I would also want an uplink on 5k-2 going to Core-1. That way the traffic does not need to go through the vPC unless there is a problem.

      Hope that helps.

  4. Nishant September 4, 2015 / 6:24 am

    Thanks, I found your post really helpful. I am in the process of implementing eVPC on 5672 UP.

    I have few questions with respect to the same,

    1.You had mentioned that the host/server that connects to these different Fexes must support Active/Active NICs, does it mean that the host or server should support LACP? What if my host doesn’t support LACP?
    2.Can I dual home a VMware ESXi (active-active NICs on route based originating virtual port ID load balancing policy) to the fexes? …
    3.Can I configure the fex port channel to static ( mode on) instead of active or lacp ?

    interface eth101/1/1, eth101/1/2
    channel-group 2 mode on
    interface eth102/1/1, eth102/1/2
    channel-group 2 mode on
    interface po2
    switchport access vlan 25

    Thanks in advance,
    Nishant

  5. John January 30, 2016 / 5:42 am

    Does this setup really need 8 network connections between the two nexus 5k and the two nexus 2k? Would it also work with only 4 network connections instead? and in that case would it still be called “dual homed”? I am not sure to understand the advantage of 8 network connections.

    • Ryan January 30, 2016 / 8:54 am

      Hey John,
      You could use 4 connections instead of 8 you just loose the amount of bandwidth available instead of 40Gbps you would have 20Gbps that is available to the 2ks. It would also still be called a dual homed setup because that 2k switch is connected two different 5k switches.

      Ryan.

  6. hostingnuggets April 24, 2016 / 7:25 am

    Ryan, I was wondering when connecting your server redundatly on Po2 shouldn’t you specify a VPC? For instance you have:

    interface po2
    switchport access vlan 25

    but as far as I understand shouldn’t it be assigned a VPC, for example:

    interface po2
    switchport access vlan 25
    vpc 2

    ?

    • Ryan April 25, 2016 / 5:45 pm

      You are correct if you would have connected the server directly to the 5ks you have to specify a VPC or if you had a single FEX connected to the 5k. In this example however I’m using FEXes which already have a VPC configured from both 5ks and I’m using port-channels off of those FEXes this is called “Enhanced vPC”. There is a difference between them, when you create an Enhanced vPC you don’t specify a VPC on the FEXes because there is already a vPC created upstream.

      Regular vPC with FEXes: LINK

      Enhanced vPC with FEXes: LINK

      Hope that Helps,
      Ryan

Comments are closed.