Browsed by
Tag: 5000

Data Center: Nexus vPC Technology

Data Center: Nexus vPC Technology

Hi Cisco friends! I had a great question from a customer today regarding failure scenarios and vPC. On the surface, I thought this is an easy one. However, when I really gave it deep thought it really depends on the type of failure. Was the failure on the peer-link, peer keepalive, vPC member port, or the worst case dual active/double failure?

Let’s go through some of the failure examples.

vPC Member Port Failure
If one vPC member port goes down – for instance, if a link from a NIC goes down – the member is removed from the PortChannel without bringing down the vPC entirely. Conversely, the switch on which the remaining port is located will allow frames to be sent from the peer link to the vPC orphan port. The Layer 2 forwarding table for the switch that detected the failure is also updated to point the MAC addresses that were associated with the vPC port to the peer link.

vPC Complete Dual-Active Failure (Double Failure)
If both the peer link and the peer-keepalive link are disconnected, the Cisco Nexus switch does not bring down the vPC, because each Cisco Nexus switch cannot discriminate between a vPC device reload and a combined peer-link and peer-keepalive-link failure.

The main problem with a dual-active scenario is the lack of synchronization between the vPC peers over the peer link. This behavior causes IGMP snooping to malfunction, which in turn causes multicast traffic to drop. As described previously, a vPC topology intrinsically protects against loops in dual-active scenarios. Each vPC peer, upon losing peer-link connectivity, starts forwarding BPDUs on vPC member ports. With the peer-switch feature, both vPC peers send BPDUs with the same bridge ID to help ensure that the downstream device does not detect a spanning-tree misconfiguration. When the peer link and the peer-keepalive link are simultaneously lost, both vPC peers become operational primary.

vPC Peer-Link Failure
To prevent problems caused by dual-active devices, vPC shuts down vPC member ports on the secondary switch when the peer link is lost but the peer keepalive is still present.

When the peer link fails, the vPC peers verify their reachability over the peer-keepalive link, and if they can
communicate they take the following actions:

● The operational secondary vPC peer (which may not match the configured secondary because vPC is
nonpreemptive) brings down the vPC member ports, including the vPC member ports located on the fabric
extenders in the case of a Cisco Nexus 5000 Series design with fabric extenders in straight-through mode.

● The secondary vPC peer brings down the vPC VLAN SVIs: that is, all SVIs for the VLANs that happen to be configured on the vPC peer link, whether or not they are used on a vPC member port.

Note: To keep the SVI interface up when a peer link fails, use the command dual-active exclude interface-vlan.

At the time of this writing, if the peer link is lost first, the vPC secondary shuts down the vPC member ports. If this failure is followed by a vPC peer-keepalive failure, the vPC secondary keeps the interfaces shut down. This behavior may change in the future with the introduction of the autorecovery feature, which will allow the secondary device to bring up the vPC ports as a result of this sequence of events.

vPC Peer-Keepalive Failure

If connectivity of the peer-keepalive link is lost but peer-link connectivity is not changed, nothing happens; both vPC peers continue to synchronize MAC address tables, IGMP entries, and so on. The peer-keepalive link is mostly used when the peer link is lost, and the vPC peers use the peer keepalive to resolve the failure and determine which device should shut down the vPC member ports.

 Best Practices: 

Define a vPC domain (should match between peers, MUST NOT MATCH BETWEEN 7K and 5K in Double-Sided vPC) This Step is Required! “(config>vpc>domain)#vpc domain <id>
Define Role Priority: Lower Priority wins Primary Role, try and match your STP root bridge with the primary role. If using “peer-switch” the STP root will be the same on both peers. “(config>vpc>domain)# role priority <xxx>”

If roles shift (they are not preemptive) you would need to change the operational primary after a failure to a value of 36767 and shut/no shut the peerlink to restore the originally configured primary. 

If the Peer Switch is also preforming L3 switching the “peer-gateway” command is recommended.

The “vpc peer-gateway” allows HSRP routers to accept frames destined for their vPC peers.  This feature extends the virtual MAC address functionality to the paired router’s MAC address.  The feature is needed when certain storage/load balancing vendors break RFC rules by ignoring the ARP reply by an HSRP active router and reply directly to the host. Without this enabled packets could traverse the peer link and end up being dropped.

Enable vPC AutoRecovery
“(config-vpc-domain)# auto-recovery”

Beginning with Cisco NX-OS Release 5.2(1), you can configure the Cisco Nexus 7000 Series device to restore vPC services when its peer fails to come online by using the auto-recovery command. You must save this setting in the startup configuration. On reload, if the peer link is down and three consecutive peer-keepalive messages are lost, the secondary device assumes the primary STP role and the primary LACP role. The software reinitializes the vPCs, bringing up its local ports. Because there are no peers, the consistency check is bypassed for the local vPC ports. The device elects itself to be STP primary regardless of its role priority and also acts as the master for LACP port roles.

ARP SYNC
The ARP table sync feature overcomes the delay involved in ARP table restoration that can be triggered when one of the switches in the vPC domain goes offline and comes back online and also when there are peer-link port channel flaps. Enabling ARP on a vPC domain improves convergence times for unicast traffic.

To enable Address Resolution Protocol (ARP) synchronization between the virtual port channel (vPC) peers, use the ip arp synchronizecomand. To disable ARP synchronization, use the no form of this command.

(config-vpc-domain)# ip arp synchronize

Content Source:
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/design_guide_c07-625857.pdf
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/command/reference/vpc/n5k-vpc_cmds_i.html#wp1316724

Cisco: Nexus 2000 (FEX) Configurations

Cisco: Nexus 2000 (FEX) Configurations

It’s been way too long since I posted on my blog. Well, I have been studying for the CCIE Data Center lab and wanted to pass on some very critical information on fabric extender (FEX) configurations. One of the most common questions that our Cisco friends ask me is “Why do I need to create a port channel for the FEX-Fabric links?”. Well let’s dive into the WHY, and then explore the HOW.

Let’s first start with a foundation on what FEX is and how it works. FEX is a highly scalable, low latency data center access layer solution. What makes it so awesome? The fact that is managed as a line card vs. a separate ToR/EoR/MoR switch. Take your Nexus 7000 (core) or 5000 (agg/access) and they play the role of the “PARENT” switch. The FEX 2000 is essentially a remote line card to the parent switch. The FEX supports 1/10G and FCoE for consolidated I/O.

Nexus 2000 Comparison:

2248-TP (E) : This is the most common FEX, 48 ports of 100/1000 BASE-T host interfaces and 4 x 10G (SFP+) fabric uplinks. The (E) varient has additional shared buffer (32MB) locally vs. using what’s on the parent switch.

2224-TP: Same as above just 24 host ports instead of 48.

2148T: First generation FEX. The host interface can only operate at 1G. Not recommended any longer, go with the 2248-TP instead.

2232PP:  This is our de-facto 10G FEX. It has 32 x 1/10G and FCoE host interfaces (SFP+) and 8 fabric uplinks (SFP+). This switch also supports DCB.

2232TM: There is also a 10G BASE-T varient of the above FEX. THIS ONE DOES NOT SUPPORT FCoE, but does support DCB. 

OK, now that that is out of the way, let’s get back to the question at hand. How do I configure the connectivity to the parent switch?

I’ll get stright to the point. Use EtherChannel for the fabric uplink interfaces.

So, what are my options anyways?

Static Pinning and EtherChannel are your two options.

Why do we like EtherChannel fabric interfaces anyways?

Well the bottom line is that with Static Pinning you do not have automatic failover capability. Sure it’s deterministic, but if one of those fabric links goes down, so do the associated host interfaces. Let’s say your using two links and set the ‘pinning max-links 2’, half of the FEX host interfaces are mapped to one fabric uplink and the other half of the host ports is mapped to the other fabric uplink. Let’s look at a visual representation of static pinning.

The major issue with this configuration is that if the fabric uplink goes down, so do the ports associated to that interface. There is NO failover. This is WHY we want to use EtherChannel instead of static pinning.

Now let’s talk about the other (preferred) solution. EtherChannel is 110% the way to go. Let’s look at the visual representation.


As you can see from this diagram, we are load balancing the host interfaces (HIF) across all the fabric uplinks. Here is a note from the Cisco Configuration Guide.

Note

A fabric interface that fails in the EtherChannel will not trigger a change to the host interfaces. Traffic is automatically redistributed across the remaining links in the EtherChannel fabric interface.

When you configure the Fabric Extender to use an EtherChannel fabric interface connection to its parent switch, the switch load balances the traffic from the hosts that are connected to the host interface ports by using the following load-balancing criteria to select the link:

  • For a Layer 2 frame, the switch uses the source and destination MAC addresses.
  • For a Layer 3 frame, the switch uses the source and destination MAC addresses and the source and destination IP addresses.
    And finally the configuration (HOW) for all this awesomeness.

    1) ENABLE FEX GLOBALLY:
    N5K(config)# feature fex
    2) Configure the member interfaces for FEX connectivity:
    N5K(config)# interface e1/1, e1/2
    N5K(config-if)# switchport mode fex-fabric
    N5K(config-if)# fex associate 100
    N5K(config-if)# channel-group 100
    3) Create the EtherChannel Interface: This probably is done already based on the previous command, but it doesn’t hurt to make sure.
    N5K(config)# interface po100
    N5K(config-if)# switchport mode fex-fabric
    N5K(config-if)# fex associate 100
    4) Create the FEX:
    N5K(config)# fex 100
    N5K(config-fex)# description FEX_100<<PO100>>
    N5K(config-fex)# pinning max-links 1 (This must 1 for an EtherChannel configuration, if you change this to any other number you cannot use EC and your using static pinning instead)
    FOUR EASY STEPS!!!!
    Here are some good troubleshooting commands.
    sh fex”
    sh fex detail” – This command will show the HIF to Fabric Uplink Mappings and the version of code running on the FEX.
    sh interface fex-fabric”  – This command will display all the FEX units attached to the parent switch.
    sh inventory fex xxx” Display the inventory information about a specific FEX.
    show diagnostic result fex 100″ – Display diagnostic test results for a specific FEX.

NOTE: Nexus 7000 (Parent) FEX

If your trying to use FEX on the N7K be certain to issue the following commands or FEX WILL NOT BE ENABLED.

In the DEFAULT VDC issue this command “install feature-set fex
Now switch to the VDC that you want to enable FEX on and issue this command “feature-set fex

Your all set now.

Cisco.com Configuration Guide for Nexus 2000:
http://tinyurl.com/36uojrv