Friday, October 10, 2014

2 NIC ports disappeared in a quad core NIC port after HP SPP drivers update

Good day! All,


As part of the BIOS and Firmware upgrade using the June HP Support pack(SPP) we started to upgrade a 2 NODE Windows 2008 R2 Failover Cluster , so the first step we did was failed over all the resources from one Node to the another and then started to push the updates, all went well and Server was rebooted and it came back online.

When we started to do post verification we found that 2 NIC cards of the 4 Port NIC card just disappeared but we where able to see the other 2 NIC's when we open the HP Teaming Software.

Well this is the first time we have seen something like this because if there was NIC card Firmware all 4 NIC's should disappear but we are seeing only 2 missing..

So as part of first troubleshooting step , we dissolved the teaming and re-installed the NIC's card drivers and Firmware again along with Teaming drivers.. After doing this all NIC's showed up and we re-configured the teaming again.

Now we had to do the other NODE so we failed over all the resources the other NODE and for some moment we paused to check how we wanted to proceed , disallow all the Teaming and do the NIC's card update first or let the SPP do the magic.. We took the risk of allowing SPP do his magic and to our surprise nothing happened and everything came back clean..

I am not sure what really happened but after logging a case with HP, there was no concrete answer to what really happened but one thing they Pointed out was if its Windows Sever the best way to upgrade the Firmware is using SPP as its smart enough to follow a order of updating BIOS,NIC cards and Smart Array controller etc..

Hope this help some one and in case some one has a better ideas please share to me...

Thursday, October 9, 2014

WEARIED!!! NIC card status shows Network Cable unplugged

Good day! All,


We started to use June 2014 HP Support Pack for upgrading BIOS and Firmware for all our HP Physical and Blades Servers..The other day we started to do a update on a Blade BL 460 G6 Server,
As we did so many Firmware updated with NEW SPP it took about 20-25 mints per Server but this one started to act and we waited for like a HOUR and it came back saying Firmware update Failed.

So after further investigating the SPP log found that it go struck updating the NIC card drivers and it just failed moving forward... well hopped on the Server using the KVM and when checked all my NIC's in the Server started to show as Network cable unplugged.

It was very wearied and we tried so many option still with no Luck, then decided to revert back the NIC card drivers , uninstalled every thing and went back to old drivers and still the same issue..


Now after uninstalling the new drivers and reverted to OLD NIC drivers the issue didn't resolve, so we started to check the System logs in Virtual Connect Module.. so to our surprise we saw as below..


Port enc0:iobay1:d4 pause-flood detected and automatically disabled

So we see something disabled and started to google around to see why and what caused the issue.. found a article in HP website similar to this

http://h20565.www2.hp.com/portal/site/hpsc/template.PAGE/public/kb/docDisplay/?javax.portlet.begCacheTok=com.vignette.cachetoken&javax.portlet.endCacheTok=com.vignette.cachetoken&javax.portlet.prp_ba847bafb2a2d782fcbb0710b053ce01=wsrp-navigationalState%3DdocId%253Demr_na-c02623029-6%257CdocLocale%253D%257CcalledBy%253D&javax.portlet.tpst=ba847bafb2a2d782fcbb0710b053ce01&sp4ts.oid=3794423&ac.admitted=1393783592336.876444892.492883150

Still poking around we saw this under ETHERNET,ADVANCE SETTINGS UNDER PORT PROTECTION


At this point we didn't Re-enable Ports but we went back to HP and raised a case.. So they suggested to first upgrade all the NIC's on the blade to current version and then click the Re-enable Ports..

After we upgrade the NIC drivers and then Re-enabled Ports all my NIC's started to connect and every thing was up and running..

Hope this helps someone.. ...