Cisco Nexus 9000 Family: Nexus 9500 Modular Switches and the Nexus 9300 Fixed Configuration

There are two types of switches in the Nexus 9000 Series: the Nexus 9500 modular switches and the Nexus 9300 fixed configuration switches.
They can run in two modes. When they run in ACI mode and in combination with a Cisco Application Policy Infrastructure Controller (APIC), they provide an application-centric infrastructure. In this case, the design follows the spine-leaf architecture shown in Figure1. When they run in NX-OS mode and use the enhanced NX-OS software, they function as a classical Nexus switch. Therefore, the design follows the standard three-tier architecture.
Figure1. Nexus 9000 Spine-Leaf Architecture

Cisco Nexus 9500 Family
The Nexus 9500 family consists of three types of modular chassis, as shown in Figure2: the 4-slot Nexus 9504, the 8-slot Nexus 9508, and the 16-slot Nexus 9516.
Figure2. Nexus 9500 Chassis Options

The Cisco Nexus 9500 Series switches have a modular architecture that consists of the following:

Switch chassis

Supervisor engine

System controllers

Fabric modules

Line cards

Power supplies

Fan trays

Optics

Among these parts, supervisors, system controllers, line cards, and power supplies are common components that can be shared among the entire Nexus 9500 product family. Table1 shows the comparison between the different models of the Nexus 9500 switches.
Table1. Nexus 9500 Modular Platform Comparison

	Nexus 9504 4-Slot	Nexus 9508 8-Slot	Nexus 9516 16-Slot
Height	7RU	13 RU	20 RU
Supervisor Slots	2	2	2
Fabric Module Slots	6	6	6
I/O Module Slots	4	8	16
Max BW per Slot (Tbps)	3.84 Tbps	3.84 Tbps	3.84 Tbps
Max BW per System (Tbps)	15 Tbps	30 Tbps	60 Tbps
Max 1/10/40 Ports	192/576/144	384/1152/288	768/2304/576
Air Flow	Front-to-back	Front-to-back	Front-to-back
Power Supplies	4 × 3kW AC PSUs	6 × 3kW PSUs	10 × 3kW PSUs
Fan Trays	3	3	3
Application	EoR or Core	EoR or Core	EoR or Core

Chassis
The Nexus 9500 chassis doesn’t have a midplane, as shown in Figure3. Midplanes tend to block airflow, which results in reduced cooling efficiency. Because there is no midplane with a precise alignment mechanism, fabric cards and line cards align together.
Figure3. Nexus 9500 Chassis

Supervisor Engine
The Nexus 9500 modular switch supports two redundant half-width supervisor engines, as shown in Figure4. The supervisor engine is responsible for the control plane function. The supervisor modules manage all switch operations. Each supervisor module consists of a Romely 1.8GHz CPU, quad core, and 16GB RAM, upgradable to 48GB RAM and 64GB SSD storage. The supervisor has an external clock source; that is, pulse per second (PPS). There are multiple ports for management, including two USB ports, an RS-232 serial port (RJ-45), and a 10/100/1000MBps network port (RJ-45).
Figure4. Nexus 9500 Supervisor Engine

System Controller
A pair of redundant system controllers can be found at the back of the Nexus 9500 chassis, as shown in Figure5. They offload chassis management functions from the supervisor modules. The system controllers are responsible for managing power supplies and fan trays. They host two main control and management paths—the Ethernet Out-of-Band Channel (EOBC) and the Ethernet Protocol Channel (EPC)—between supervisor engines, line cards, and fabric modules. The EOBC provides the intrasystem management communication across modules, and the EPC channel handles the intrasystem data plane protocol communication.
Figure5. Nexus 9500 System Controllers

Fabric Modules
The platform supports up to six fabric modules. The packet lookup and forwarding functions involve both the line cards and the fabric modules; both contain multiple network forwarding engines (NFEs). The NFE is a Broadcom trident two ASIC (T2), and the T2 uses 24 40GE ports to guarantee the line rate. All fabric modules are active; each fabric module consists of multiple NFEs, as shown in Figure6. The Nexus 9504 has one NFE per fabric module, the Nexus 9508 has two, and the Nexus 9516 has four.
Figure6. Nexus 9500 Fabric Module

When you use the 1/10G + four 40GE line cards, you need a minimum of three fabric modules to achieve line-rate speeds. When you use the 36-port 40GE line cards, you will need six fabric modules to achieve line-rate speeds.
NOTE: The fabric modules are behind the fan trays, so to install them you must remove the fan trays.

Line Cards
It is important to understand that there are multiple types of Nexus 9500 line cards. There are cards that can be used in standalone mode when used with enhanced NX-OS, in a classical design. There are line cards that can be used in application-centric infrastructure mode (ACI) only. There are also line cards that can be used in both modes: standalone mode using NX-OS and ACI mode.
All line cards have multiple NFEs for packet lookup and forwarding. In addition, the ACI-ready leaf line cards contain an additional ASIC called an application leaf engine (ALE). ALE performs the ACI leaf function when the Nexus 9500 is used as a leaf node when deployed in ACI mode.
The ACI-only line cards contain an additional ASIC called an application spine engine (ASE); the ASE performs ACI spine functions when the Nexus 9500 is used as a spine in ACI mode. Figure7 shows the high-level positioning of the different cards available for the Nexus 9500 Series network switches.
Figure7. Nexus 9500 Line Cards Positioning

Nexus 9500 line cards are also equipped with dual-core CPUs, which are used to speed up some control functions, such as programming the hardware table resources, collecting and sending line card counters, statistics, and offloading BFD protocol handling from the supervisors. Table2 shows the different types of cards available for the Nexus 9500 Series switches and their specification.
Table2. Nexus 9500 Modular Platform Line Card Comparison

Power Supplies
The Nexus 9500 platform supports up to 10 power supplies; they are accessible from the front and are hot swappable. Two 3000W AC power supplies can operate a fully loaded chassis; they support N+1 and N+N (grid redundancy). The 3000W AC power supply shown in Figure8 is 80 Plus platinum rated and provides more than 90% efficiency.
Figure8.Nexus 9500 AC Power Supply

NOTE:The additional four power supply slots are not needed with existing line cards shown in Table2; however, they offer head room for future port densities, bandwidth, and optics.

Fan Trays
The Nexus 9500 consists of three fan trays; each tray consists of three fans. Dynamic speed is driven by temperature sensors and front-to-back air flow with N+1 redundancy per tray. Fan trays are installed after the fabric module installation, as shown in Figure9.
Figure9. Nexus 9500 Fan Tray

NOTE: To service the fabric modules, the fan tray must be removed first. If one of the fan trays is removed, the other two fan trays will speed up to compensate for the loss of cooling.

Cisco QSFP Bi-Di Technology for 40 Gbps Migration
As data center designs evolve from 1G to 10G at the access layer, access to aggregation and spine-leaf design at the spine layer will move to 40G. The 40G adoption is slow today because of multiple barriers; the first is the cost barrier of the 40G port itself. Second, when you migrate from 10G to 40G, you must replace the cabling. 10G operates on what is referred to as two strands of fiber; however, 40G operates on eight strands of fiber. Bi-Di optics are standard based, and they enable customers to take the current 10G cabling plant and use it for 40G connectivity without replacing the cabling. Figure10 shows the difference between the QSFP SR and the QSFP Bi-Di.
Figure10. Cisco Bi-Di Optics

Cisco Nexus 9300 Family
The previous section discussed the Nexus 9500 Series modular switches. This section discusses details of the Cisco Nexus 9300 fixed configuration switches. There are currently four chassis-based models in the Nexus 9300 platform. Table3 summarizes the different specifications of each chassis. The Nexus 9300 is designed for top-of-rack (ToR) and mid-of-row (MoR) deployments.
Table3. Nexus 9500 Fixed-Platform Comparison

	Nexus 9396PX 48-Port 1/10GE	Nexus 9396TX 48-Port 1/10GE	Nexus 93128TX 96-Port 1/10GE	Nexus 9336PQ 36-Port 40GE
Height	2 RU	2RU	2RU	RU
I/O Module Slots	1 GEM (12 QSFP)	1 GEM (12 QSFP)	1 GEM (8 QSFP)	Fixed
Max BW/ System	480 Gbps (nonblocking)	480 Gbps (nonblocking)	3:1 (oversubscription)	1.44 Tbps (nonblocking)
Max 1/10/40/ Ports	48/48/12	48/48/12	96/96/8	0/0/36
Air Flow	Front-to-back Back-to-front	Front-to-back Back-to-front	Front-to-back Back-to-front	Front-to-back
Power Supply Configurations	(1+1) x 650W AC	(1+1) x 650W AC	(1+1) x 650W/1200W AC	(1+1) x 1200W AC
Application	ToR	ToR	ToR	EoR
Software Mode	NX-OS or ACI mode (leaf)	NX-OS or ACI mode (leaf)	NX-OS or ACI mode (leaf)	ACI mode only (spine)

The 40Gbps ports for Cisco Nexus 9396PX, 9396TX, and 93128TX are provided on an uplink module that can be serviced and replaced by the user. The uplink module is the same for all switches. If used with the Cisco Nexus 93128TX, eight out of the 12 × 40Gbps QSFP+ ports will be available.
As shown in Table3, the Nexus 9396PX, 9396TX, and 93128TX can operate in NX-OS mode and in ACI mode (acting as a leaf node). The Nexus 9336PQ can operate in ACI mode only and act as a spine node.
Figure11 shows the different models available today from the Nexus 9300 switches.
Figure11. Cisco Nexus 9300 Switches