Check your QoS policies before upgrading to 2.2(1b)

I recently upgraded our UCS domain from 2.1(3a) to 2.2(1b). We immediately ran into some critical errors which pointed to a few QoS Policies. The Rate(Kbps) setting cannot be set higher than 10000000. Be aware that it can cause quit some issues! Allow me to explain what I have noticed:

Hardware:
Cisco UCS 6248UP Fabric Interconnects
Cisco UCS 5108 Blade Chassis
Cisco UCS B200 M3 Blade Servers (including VIC1240 + Port Expander)

Problem Description:
When you look at a QoS Policy you can set a Rate(Kbps) setting:

This value can be set from 0 to 40000000. A value 0 equals to ‘line-rate’ and has no limit. For example if I would set this to 20000000 it would result in a 20Gbps network interface in the Operating System. In our case a 0 or ‘line-rate’ value would result in a 40Gbps network interface because our hardware supports that. With version 2.1(3a) I was able to set this value as I want. Since we upgraded to 2.2(1b) the following occurs:

1. Every QoS Policy set to ‘line-rate’ results in a 10Gbps interface in the Operating System. In our case that used to be 40Gbps.
2. Any value higher than 10000000 result in the following critical alert:

This results in the following critical alert:

Please note that this critical alert are sometimes only generated once you shutdown a blade server. And it takes some while before they pop-up. Actually I find it quit normal to have only one or multiple 10Gbps interface instead of 40Gbps interfaces. Because if you ask me I would think you have a 4x10Gbps backplane. But then again it is a different behavior since 2.2(1b).

IMPORTANT: If you have set these values and upgrade to 2.2(1b). This is what we encountered. Once your first Fabric Interconnects reboots you loose connectivity to that Fabric Interconnect. And when you shutdown and then start a Blade Server it is unable to boot because it cannot associate with the Fabric Interconnect. Set the values back to line-rate and the problem is solved.

I hope this information might help you out if you run into the same problem.

Boudewijn

I have added a Cisco Support Case for this issue. We were able to re-produce it. The Cisco engineer also found this information which is very usefull information:

As I thought it is only possible to configure a maximum of 10Gbps with the QoS policy. The Cisco engineer confirmed with Cisco that something has changed. They are now working on it to provide us with more information. If there is any update they will update this post.

Please note; once your UCS domain is already running on 2.2(1b) it causes no issues (as far we have seen). But you have to be sure to set it correct before upgrading to 2.2(1b), otherwise it will cause issues. If you have already configured every value with 'line-rate' you don't have to worry about anything.

Hi Boudewijn Plomp,

I have logged a new BUG : CSCum40629 for correcting the documentation of the Cisco.com

As of now, i can confirm to you that the traffic can be shapped only between 8 Mbps to 10 Gbps and nothing beyond that.

Best Regards

Moulie G J C

Thanks! That is good news.

I have changed the title of this post so that it more reflects to this case.

Boudewijn