The FI 6332 has 26 QSFP+ Ports and 6 QSFP+ Modul Ports. The former can be broken out using QSFP-SFP break-out cable. According to the documentation, this results in 98 10G ports, which is odd, because 26 * 4 is 104 and 26 is a unusual number anyway. Obviously, ports 13-14 cannot be broken out but instead can use the QSA to get one 10G port. Therefore 24 * 4 + 2 * 1 is 98, which matches the documentation.
Can anyone show me, how these ports are connected to the NFE and ALE internally which should explain above port configuration. The same for the FI 6332-16UP if possible.
The port 13-14 no-breakout exception also exists on the 9332PQ switch, so you might look for its architecture. I think the 13/14 limitation is entirely a function of the Broadcom T2.
Do mind another scale limitation from the UCS 3.1 release notes -- if you're using jumbo frames, only four QSFP ports can be configured for breakout. I suspect this is due to the small 12MB buffers on the Broadcom ASIC.
To help understand cabling possibilities with the new FI's, I've put together a web tool that tries to honor all the rules about bkreaout, QSA, port licenses, etc. It may be useful to you: http://beeline.org/ucs/ucslayout.cgi
Very nice. Thanks Craig.
So if you use jumbo frames AT ALL in the system, you can only break out FOUR of the QSFP ports? Most VMware deployments that I have seen use Jumbo frames, so this appears to be a fairly serious "caveat". Am I correct in my understanding?
I dug into it more closely in our lab. Here is the default QoS system class from a UCS 6332-16UP running 3.1(1e):
Note the Platinum class is a no-drop class ("Packet Drop" unchecked) and it has a jumbo MTU. When I try to configure more than four breakout ports, this error displays:
But if I make Platinum a drop class, or if I lower the Platinum MTU to 2240 or less, or if I disable the Fibre Channel class, then I can configure as many breakout ports as I want to.
So, the short answer is you can do breakout with jumbo frames, but only one traffic class can receive a no-drop guarantee. This should only affect those running multiple storage protocols, like FC/FCoE and iSCSI. (the iSCSI traffic either needs to use a regular MTU or it can't be no-drop).
Other types of jumbo traffic like vMotion, replication, and VXLAN don't need the no-drop guarantee.