Browsed by
Tag: Nexus

The Journey to CCIE #2 Starts Now

The Journey to CCIE #2 Starts Now

Game On Old Friend

2015-07-20 10.25.25 pm


It’s hard to believe that it’s been almost 2 years since I passed the R/S lab and my digits (40755) were assigned. I remember the numbers just passed 40k and I was so hoping to get 40007.

This way I could be 007. <GRIN>

Now I’m ready for the next challenge. My motivation for CCIE DC was simple. First I wanted to challenge myself yet again. Second, I feel strongly that a deep understanding of UCS & virtualization helps me stay relevant when it comes to private cloud conversations which all the cool kids are doing. Finally, I suck at storage. If storage was a weakness to me, it would be like green kryptonite to Clark.

2015-07-20 09.34.46 pm









All that said, I also miss the behind the wheel configuration and troubleshooting. I’m a pre-sales SE and spend most of my time these days in design sessions, product updates, and evangelizing new solutions. What better way to get serious hands-on than a CCIE lab?

Right before Christmas 2014, I took the CCIE DC written and failed it by 1-2 questions. I was so upset about carrying that disappointment through the holidays. Jan 8th was my date of redemption and I passed with a 953/1000.

I purchased workbooks from INE and leveraged their all access pass program and have about 1/2 the lab gear in one of our Cisco offices Just don’t have enough juice. <FACEPALM>

I’m also going to leverage VIRL and UCS Emulator for my studies.

Now it’s time to lock down and get this lab banged out in November. T-Minus 4 months… #TickTock


CCIE Data Center Lab Exam v1.0 

Lab Equipment and Software Versions

Passing the lab exam requires a depth of understanding difficult to obtain without hands-on experience. Early in your preparation you should arrange access to equipment similar to that used on the exam, and listed below.

The lab exam tests any feature that can be configured on the equipment and the NXOS versions indicated below. Occasionally, you may see more recent NXOS versions installed in the lab, but you will not be tested on the new features of a release unless indicated below.

  • Cisco Catalyst Switch 3750
  • Cisco 2511 Terminal Server
  • MDS 9222i
  • Nexus7009
    • (1) Sup
    • (1) 32 Port 10Gb (F1 Module)
    • (1) 32 Port 10Gb (M1 Module)
  • Nexus5548
  • Nexus2232
  • Nexus 1000v
  • UCS C200 Series Server
    • vic card for c-series
  • UCS-6248 Fabric Interconnects
  • UCS-5108 Blade Chassis
    • B-200 Series Blades
    • Palo mezzanine card
    • Emulex mezzanine card
  • Cisco Application Control Engine Appliance – ACE4710
  • Dual attached JBODs

Software Versions

  • NXOS v6.x on Nexus 7000 Switches
  • NXOS v5.x on Nexus 5000 Switches
  • NXOS v4.x on Nexus 1000v
  • NXOS v5.x on MDS 9222i Switches
  • UCS Software release 2.x Fabric Interconnect
  • Software Release A5(1.0) for ACE 4710
  • Cisco Data Center Manager software v5.x

ACE!? Really!??!?!?

2015-07-20 10.05.10 pm


ConfigBytes: Nexus 6000/5600 Latency & Buffer Monitor

ConfigBytes: Nexus 6000/5600 Latency & Buffer Monitor


Episode 2
Platforms: Nexus 6000 & 5600 (UPC based ASIC)


Latency Monitor:

Full Documentation

The switch latency monitoring feature marks each ingress and egress packet with a timestamp value. To calculate the latency for each packet in the system the switch compares the ingress with the egress timestamp. The feature allows you to display historical latency averages between all pairs of ports, as well as real-time latency data.

You can use the latency measurements to identify which flows are impacted by latency issues. In addition the statistics generated by the switch latency monitoring feature allow you to plan network topologies, manage incident responses and identify root causes for application issues in the network. You can also use the statistics to provide a Service Level Agreement (SLA) for latency intensive applications.

Configuration Example for Switch Latency Monitoring

Requires 7.x code

This example shows how to configure switch latency monitoring:

switch(config)# hardware profile latency monitor base 800
switch(config)# interface ethernet 1/1
switch(config-if)# packet latency interface ethernet 1/2 mode linear step 40
switch(config-if)# packet latency interface ethernet 1/3-4 mode exponential step 40
switch(config-if)# packet latency interface ethernet 1/5 mode custom low 40 high 1200
switch(config)# interface ethernet 2/1
switch(config-if)# packet latency interface ethernet 1/1 mode exponential step 80

Buffer Utilization Histogram:

Full Documentation

The Buffer Utilization Histogram feature enables you to analyze the maximum queue depths and buffer utilization in the system in real time. Instantaneous or real time buffer utilization information is supported by the hardware. You can use software to obtain the history of the buffer usage by polling the hardware at regular intervals. Obtaining an historic timeline of the buffer usage provides a better picture of the traffic pattern in the system and helps in traffic engineering. Ultimately, you are able to make better use of the hardware buffer resources.

On the Cisco Nexus device, every three ports of 40 Gigabit Ethernet or every 12 ports of 10 Gigabit Ethernet have access to a shared 25 Mb packet buffer. 15.6 Mb are reserved for ingress and 8.6 Mb are reserved for egress. The remaining space is used for SPAN and control packets.

The Buffer Utilization Histogram enables you to do the following:

  • Configure buffer utilization history measurements on the interested ports.
  • View buffer utilization over an interval of time.
  • Configure either a slow or a fast polling mode.
  • Copy collected statistics to the buffer_util_stats file on the bootflash drive every hour to allow for later analysis. The collected statistics are appended to the end of the file after an hour and a timestamp is placed in the header that has the interface name.

Configuration Example for Buffer Utilization:

Requires 7.x code

switch# configure terminal
switch(config)# interface ethernet 1/1
switch(config-if)# hardware profile buffer monitor

Output Examples for Buffer Utilization Histogram

2015-05-14 10.05.56 am

Write Histogram Data to File & Syslog Alert via EEM/Python

Python Script:

import sys
import re
import io
import syslog
from cisco import cli
from sys import argv

def parse_and_print_interface(input_string):
print “Received input – {0}”.format(input_string)
result = re.findall(r’\bEthernet\w+\W+\w+’, input_string)
print result
#print result[0]
show_cli_cmd = “show hardware profile buffer monitor interface ” + result[0] + ” history detail “
# show_cli_cmd = “show hardware profile buffer monitor all history detail”
# Execute the command on the switch
print show_cli_cmd
time1 = cli(“show clock”)
raw_input = cli(show_cli_cmd)
output1 = time1 + raw_input + “\n”
target = open(“/bootflash/EEM_buffer_log”, “a”)
time2 = cli(“show clock”)
raw_input2 = cli(“show interface burst-counters”)
output2 = time2 + raw_input2 + “\n”
target = open(“/bootflash/EEM_burst_log”, “a”)

def main():
print sys.argv

if __name__==”__main__”:

EEM Script:

event manager applet burst_monitor

  event syslog pattern “bigsurusd”

  action 1 cli source -l “$_syslog_msg”

Data Center: Nexus vPC Technology

Data Center: Nexus vPC Technology

Hi Cisco friends! I had a great question from a customer today regarding failure scenarios and vPC. On the surface, I thought this is an easy one. However, when I really gave it deep thought it really depends on the type of failure. Was the failure on the peer-link, peer keepalive, vPC member port, or the worst case dual active/double failure?

Let’s go through some of the failure examples.

vPC Member Port Failure
If one vPC member port goes down – for instance, if a link from a NIC goes down – the member is removed from the PortChannel without bringing down the vPC entirely. Conversely, the switch on which the remaining port is located will allow frames to be sent from the peer link to the vPC orphan port. The Layer 2 forwarding table for the switch that detected the failure is also updated to point the MAC addresses that were associated with the vPC port to the peer link.

vPC Complete Dual-Active Failure (Double Failure)
If both the peer link and the peer-keepalive link are disconnected, the Cisco Nexus switch does not bring down the vPC, because each Cisco Nexus switch cannot discriminate between a vPC device reload and a combined peer-link and peer-keepalive-link failure.

The main problem with a dual-active scenario is the lack of synchronization between the vPC peers over the peer link. This behavior causes IGMP snooping to malfunction, which in turn causes multicast traffic to drop. As described previously, a vPC topology intrinsically protects against loops in dual-active scenarios. Each vPC peer, upon losing peer-link connectivity, starts forwarding BPDUs on vPC member ports. With the peer-switch feature, both vPC peers send BPDUs with the same bridge ID to help ensure that the downstream device does not detect a spanning-tree misconfiguration. When the peer link and the peer-keepalive link are simultaneously lost, both vPC peers become operational primary.

vPC Peer-Link Failure
To prevent problems caused by dual-active devices, vPC shuts down vPC member ports on the secondary switch when the peer link is lost but the peer keepalive is still present.

When the peer link fails, the vPC peers verify their reachability over the peer-keepalive link, and if they can
communicate they take the following actions:

● The operational secondary vPC peer (which may not match the configured secondary because vPC is
nonpreemptive) brings down the vPC member ports, including the vPC member ports located on the fabric
extenders in the case of a Cisco Nexus 5000 Series design with fabric extenders in straight-through mode.

● The secondary vPC peer brings down the vPC VLAN SVIs: that is, all SVIs for the VLANs that happen to be configured on the vPC peer link, whether or not they are used on a vPC member port.

Note: To keep the SVI interface up when a peer link fails, use the command dual-active exclude interface-vlan.

At the time of this writing, if the peer link is lost first, the vPC secondary shuts down the vPC member ports. If this failure is followed by a vPC peer-keepalive failure, the vPC secondary keeps the interfaces shut down. This behavior may change in the future with the introduction of the autorecovery feature, which will allow the secondary device to bring up the vPC ports as a result of this sequence of events.

vPC Peer-Keepalive Failure

If connectivity of the peer-keepalive link is lost but peer-link connectivity is not changed, nothing happens; both vPC peers continue to synchronize MAC address tables, IGMP entries, and so on. The peer-keepalive link is mostly used when the peer link is lost, and the vPC peers use the peer keepalive to resolve the failure and determine which device should shut down the vPC member ports.

 Best Practices: 

Define a vPC domain (should match between peers, MUST NOT MATCH BETWEEN 7K and 5K in Double-Sided vPC) This Step is Required! “(config>vpc>domain)#vpc domain <id>
Define Role Priority: Lower Priority wins Primary Role, try and match your STP root bridge with the primary role. If using “peer-switch” the STP root will be the same on both peers. “(config>vpc>domain)# role priority <xxx>”

If roles shift (they are not preemptive) you would need to change the operational primary after a failure to a value of 36767 and shut/no shut the peerlink to restore the originally configured primary. 

If the Peer Switch is also preforming L3 switching the “peer-gateway” command is recommended.

The “vpc peer-gateway” allows HSRP routers to accept frames destined for their vPC peers.  This feature extends the virtual MAC address functionality to the paired router’s MAC address.  The feature is needed when certain storage/load balancing vendors break RFC rules by ignoring the ARP reply by an HSRP active router and reply directly to the host. Without this enabled packets could traverse the peer link and end up being dropped.

Enable vPC AutoRecovery
“(config-vpc-domain)# auto-recovery”

Beginning with Cisco NX-OS Release 5.2(1), you can configure the Cisco Nexus 7000 Series device to restore vPC services when its peer fails to come online by using the auto-recovery command. You must save this setting in the startup configuration. On reload, if the peer link is down and three consecutive peer-keepalive messages are lost, the secondary device assumes the primary STP role and the primary LACP role. The software reinitializes the vPCs, bringing up its local ports. Because there are no peers, the consistency check is bypassed for the local vPC ports. The device elects itself to be STP primary regardless of its role priority and also acts as the master for LACP port roles.

The ARP table sync feature overcomes the delay involved in ARP table restoration that can be triggered when one of the switches in the vPC domain goes offline and comes back online and also when there are peer-link port channel flaps. Enabling ARP on a vPC domain improves convergence times for unicast traffic.

To enable Address Resolution Protocol (ARP) synchronization between the virtual port channel (vPC) peers, use the ip arp synchronizecomand. To disable ARP synchronization, use the no form of this command.

(config-vpc-domain)# ip arp synchronize

Content Source:

Cisco: Algo Boost Nexus 3548 Preview/Unbox

Cisco: Algo Boost Nexus 3548 Preview/Unbox

I got something very cool last week. It came overnight from my good friend Frank in NY. What we have here is a very special privilege folks. It’s a prototype of the Nexus 3548 ultra low latency switch using our custom ASIC called Algo Boost/Monticello. Instead of killing you with all the details I decided to create a video of the un-boxing and special features walkthrough. Enjoy!



Cisco: Nexus 2000 (FEX) Configurations

Cisco: Nexus 2000 (FEX) Configurations

It’s been way too long since I posted on my blog. Well, I have been studying for the CCIE Data Center lab and wanted to pass on some very critical information on fabric extender (FEX) configurations. One of the most common questions that our Cisco friends ask me is “Why do I need to create a port channel for the FEX-Fabric links?”. Well let’s dive into the WHY, and then explore the HOW.

Let’s first start with a foundation on what FEX is and how it works. FEX is a highly scalable, low latency data center access layer solution. What makes it so awesome? The fact that is managed as a line card vs. a separate ToR/EoR/MoR switch. Take your Nexus 7000 (core) or 5000 (agg/access) and they play the role of the “PARENT” switch. The FEX 2000 is essentially a remote line card to the parent switch. The FEX supports 1/10G and FCoE for consolidated I/O.

Nexus 2000 Comparison:

2248-TP (E) : This is the most common FEX, 48 ports of 100/1000 BASE-T host interfaces and 4 x 10G (SFP+) fabric uplinks. The (E) varient has additional shared buffer (32MB) locally vs. using what’s on the parent switch.

2224-TP: Same as above just 24 host ports instead of 48.

2148T: First generation FEX. The host interface can only operate at 1G. Not recommended any longer, go with the 2248-TP instead.

2232PP:  This is our de-facto 10G FEX. It has 32 x 1/10G and FCoE host interfaces (SFP+) and 8 fabric uplinks (SFP+). This switch also supports DCB.

2232TM: There is also a 10G BASE-T varient of the above FEX. THIS ONE DOES NOT SUPPORT FCoE, but does support DCB. 

OK, now that that is out of the way, let’s get back to the question at hand. How do I configure the connectivity to the parent switch?

I’ll get stright to the point. Use EtherChannel for the fabric uplink interfaces.

So, what are my options anyways?

Static Pinning and EtherChannel are your two options.

Why do we like EtherChannel fabric interfaces anyways?

Well the bottom line is that with Static Pinning you do not have automatic failover capability. Sure it’s deterministic, but if one of those fabric links goes down, so do the associated host interfaces. Let’s say your using two links and set the ‘pinning max-links 2’, half of the FEX host interfaces are mapped to one fabric uplink and the other half of the host ports is mapped to the other fabric uplink. Let’s look at a visual representation of static pinning.

The major issue with this configuration is that if the fabric uplink goes down, so do the ports associated to that interface. There is NO failover. This is WHY we want to use EtherChannel instead of static pinning.

Now let’s talk about the other (preferred) solution. EtherChannel is 110% the way to go. Let’s look at the visual representation.

As you can see from this diagram, we are load balancing the host interfaces (HIF) across all the fabric uplinks. Here is a note from the Cisco Configuration Guide.


A fabric interface that fails in the EtherChannel will not trigger a change to the host interfaces. Traffic is automatically redistributed across the remaining links in the EtherChannel fabric interface.

When you configure the Fabric Extender to use an EtherChannel fabric interface connection to its parent switch, the switch load balances the traffic from the hosts that are connected to the host interface ports by using the following load-balancing criteria to select the link:

  • For a Layer 2 frame, the switch uses the source and destination MAC addresses.
  • For a Layer 3 frame, the switch uses the source and destination MAC addresses and the source and destination IP addresses.
    And finally the configuration (HOW) for all this awesomeness.

    N5K(config)# feature fex
    2) Configure the member interfaces for FEX connectivity:
    N5K(config)# interface e1/1, e1/2
    N5K(config-if)# switchport mode fex-fabric
    N5K(config-if)# fex associate 100
    N5K(config-if)# channel-group 100
    3) Create the EtherChannel Interface: This probably is done already based on the previous command, but it doesn’t hurt to make sure.
    N5K(config)# interface po100
    N5K(config-if)# switchport mode fex-fabric
    N5K(config-if)# fex associate 100
    4) Create the FEX:
    N5K(config)# fex 100
    N5K(config-fex)# description FEX_100<<PO100>>
    N5K(config-fex)# pinning max-links 1 (This must 1 for an EtherChannel configuration, if you change this to any other number you cannot use EC and your using static pinning instead)
    Here are some good troubleshooting commands.
    sh fex”
    sh fex detail” – This command will show the HIF to Fabric Uplink Mappings and the version of code running on the FEX.
    sh interface fex-fabric”  – This command will display all the FEX units attached to the parent switch.
    sh inventory fex xxx” Display the inventory information about a specific FEX.
    show diagnostic result fex 100″ – Display diagnostic test results for a specific FEX.

NOTE: Nexus 7000 (Parent) FEX

If your trying to use FEX on the N7K be certain to issue the following commands or FEX WILL NOT BE ENABLED.

In the DEFAULT VDC issue this command “install feature-set fex
Now switch to the VDC that you want to enable FEX on and issue this command “feature-set fex

Your all set now. Configuration Guide for Nexus 2000: