Steps replacing Failed Virtual Connect Module on a C Class Frame

Good day All,

Welcome back!!!! recently we had a failed Virtual Connect Module and we had to replace it. Its hot swap-able but there are certain things you need to follow before you replace it..

Important Step is to identify what the current Firmware version the new VC Module is? only if its matches to the Firmware Version of the VC Module which is failed then only you can replace.
If the Firmware Version doesn't match then you will have to either upgrade Current Firmware or downgrade the Firmware depending on your scenario.In our case the VC module we got was 4.10 Version and we had to upgrade to 4.20 Version

Another Important note is we need a IP for the new VC when inserting into Spare bay.You can use the failed bay IP make sure to un-check it or else you will get duplicate IP error.

Steps:

1. So ask the FE to insert the VC in spare bay
2. Login to OA and under inter connect bays you will see the new VC, expand and click on Information tab you will see the current Firmware Version, in our case it was 4.10 .So we had to upgrade the VC.
3.Now go to Enclosure Settings\Enclosure Bay IP Addressing\IP4\Interconnect Bays and then enable checkbox on the bay the new VC is inserted, type in the EBIPA IP that is IP address and other details in the column and click Apply

4.We have seen sometime that it doesn't show ip in the Interconnect Bays information tab under Management IP Address ,so try resetting the module and check,

5.if you still don't see the IP then you will have to login to Primary OA using putty and do the following commands

show EBIPA interconnect 

set ebipa interconnect x.x.x.x x.x.x.x baynumber     (Applies IP and mask for bay 1)

set ebipa interconnect gateway x.x.x.x baynumer      (Applies gateway for bay 1)

6. Login to Windows machine were you installed the VCSU Utility and run the command in interactive mode to check healthcheck under start Program files

Please enter action ("help" for list): healthcheck
Please enter Onboard Administrator IP Address: x.x.x.x
Please enter Onboard Administrator Username: *************
Please enter Onboard Administrator Password: *************

7.Again start the VCSU Utility in interactive mode and it will ask series of questions..

Please enter action ("help" for list): update
Please enter Onboard Administrator IP Address: x.x.x.x
Please enter Onboard Administrator Username: *************
Please enter Onboard Administrator Password: *************
Please enter firmware package location: C:\vc\vcfwall420.bin
Please enter Configuration backup password (Optional):
Please enter Force Update options if any (eg: version,health): health
Please enter VC-Enet module activation order if any (eg: parallel or odd-even
or serial or manual. Default: odd-even):
Please enter VC-FC module activation order if any (eg: parallel or odd-even or
serial or manual. Default: serial):
Please enter the time (in minutes) to wait between activating or rebooting
VC-Enet modules (max 60 mins. Default: 0 mins):
Please enter the time (in minutes) to wait between activating or rebooting
VC-FC modules (max 60 mins. Default: 0 mins):
The target configuration is integrated into a Virtual Connect Domain. Please
enter the Virtual Connect Domain administrative user credentials to continue.
User Name: ************
Password: *************

Note: All the steps remain the same if you are trying to downgrade or upgrade Firmware for a VC just that in the steps highlighted in RED above if you upgrading as in my case you need to put as health.If downgrading you need to type in there as version.

It takes about 30-40 mints depending on how many VC modules present and it will show at the end updated version

8.After confirming that Version is at the same level as the failed VC Module now ask the FE to replace the failed VC Module.

9.Wait couple of mints and you will see that new VC Module settles down and will show green if you see on the OA screen.

So this was easy, any questions free to ask!!!!!

If you don't have links to VCSU utility or Version Please find below:

Note: You can use SPP as well but VC are critical so best bet is to use VCSU utility which also takes a back up of VC domain before upgrade how cool is that.

Virtual Connect Support Utility User Guide:

https://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=4144084&docId=emr_na-c04567803&docLocale=en_US

Virtual connect Support Utility Version 1.11

http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_5e16cbb76d9e46e891ca04048d

 Download the Bin file you needed, In our case it was Version 4.20

     Download Virtual Connect firmware level of your choosing. (in your case version 4.20)

http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3adcc3c4275f460c8d97cad17e

Excellent article which goes over if the module you received is having higher version and want to downgrade to replace failed module
http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_sf-EN_US000005572


Hopefully this helps someone, until next one you all have a good day!!!!

How to trace a process which gets created and disappears!!!!

Good day All,

Welcome back!!!

We had little issue on a Server where we had to find out which process taskkill,exe is calling it.
Any ideas?????
If you saying Process Explorer, yell yes you can but if the new Process getting created and getting terminated is so fast you will not be able to trace it .

So nice little tool from System Internals is ProcMon, it has so many benefits and i believe if you want to be a successful System Admin you should always carry System Internal tools along with you and google on how these tools help!!! very very useful tools :)


So coming back to the point, i needed to trace what is calling taskkill,exe ,so ran procmon.. let it run for 10 mints and saved the logs...Just did ctrl +F to open Find tool and typed in taskkill,exe, boom it took me right there

Hopefully this helps someone!!! until next one you all have a good day!!!

VMSS2Core and Windbg saved the day!!!!

Good day All,

Welcome back!!! As part of root cause analysis we investigated 2 VM's issues and fixed them so thought of sharing this to all so that little steps from us help in fixing lot of problems.

We had 2 VM's ,one Windows 2008 and the other one is Windows 2012 both was in hung state.

When the Server was hung took a snapshot of the Server and using the vmss2core.exe generated the dump for it.
If you need more on vmss2core.exe check my other post

Windows 2008:
So using Windbg started to analyze the dump, the command i gave this time was !memusage

So out of 4 GB memory only 21 MB free space and only 448KB in standby causing the Server to run out of memory.


Windows 2012:

In Windbg i used this time !vm and this showed me like 500+ process of wmiprvse.exe.. So we know the issue what caused Server to hung.

Please bookmark this link that gives you a list of commnon Windbg command

https://blogs.msdn.microsoft.com/willy-peter_schaub/2009/11/27/common-windbg-commands-reference/

Hopefully this helps someone!!! and until next one you all have a good day!!!!