c7000/c3000 Enclosures - VIRTUAL CONNECT INTERCONNECT MODULES FIRMWARE UPDATE

Good day! All,

Welcome back!, Today i will share with you all the steps we followed to update Firmware for Virtual Connects on either c7000 or c3000 Enclosures.

Before you think of updating Firmware its very much necessary that you take a backup for the Virtual Connect domain

We have 2 ways to accomplish the task

1. Using HP Support pack(HP SPP)
2.Virtual Connect Support Utility - (vcsu-1.9.0)

Assumption is that you are having 2 Virtual Connect Interconnect Modules in bay 1 and bay 2 for redundancy... If not the case then you are looking at lot of downtime because this Firmware updates on each Interconnect Module takes about 20-25 approximately .

So the next question would be how much of downtime is required if we have 2 Interconnect Modules and using VCSU Utility, well i did updates for close to like 20 Enclosures and we always saw like 5-10 Pings packs drops.The beauty with VCSU utility is that it first finishes the standby Interconnect Module and then it fails-over and when doing Active Interconnect Module we have seen the packet drops.

Option 1: Using HP SPP i already posted a article on how to apply Firmware for a Windows Server, the steps are the same just that while Adding Node you type in the IP Address of the Virtual IP of Interconnect Module and chose the right type as below.

If anyone has a question did we ever update Firmware using the HP SPP then the answer is yes only 1 time for like 20 Enclosures and believe me that is not fun at all.. After you hit enter and say re-mediate then you are like watching the browser with no update and no proper information, hopefully in near future this may change but till i writing this article the safe bet would be using VCSU utility..

Option 2: Virtual Connect Support utility

1. Just search the HP site and download the latest Virtual Connect Support utility , the version we used was 1.9 , its just next next and you should be able to install the tool
2. On the desktop you will see shortcut, if not just browse to this location C:\Program Files\Hewlett-Packard Company\Virtual Connect Support Utility and right click on VCSU-CommandPrompt and say create a short cut on the desktop.
3. Go out to HP site and download the Firmware for interconnect modules, usually a BIN file.
4.Click the VCSU-CMD and then type the following command

vcsu -a healthcheck -i IP Address of OA -u Username of OA -p Password of OA -vcu Username of VirtualConnect Manger -vcp Password of Virtual connect Manager

ex. vcsu -a healthcheck -i 192.168.1.1 -u localadmin -p password -vcu Administrator -vcp password

Before you proceed to update the firmware make sure the health check result is all passed, if not re-mediate it first and then only proceed further.

5. Now type the following command to update the firmware..

vcsu -a healthcheck -i 192.168.1.1 -u localadmin -p password -vcu Administrator -vcp password -l "Bin file you just downloaded"

You will prompted with yes/no option to proceed, just type yes..

Lesson we learnt is that after clicking yes it goes fast with Percentage and around like 20 %  for like 15-20 mints it just stops so just be patient and don't close the window.. this is when we have seen that  it does the Firmware update on the stand by Module and then  fail-overs to Active module and if you have started any continues Pings you will see the packet drops now..
After like 40-45 mints if everything goes well you will see something like this..

Well updating Firmware on Interconnect modules are not that cumbersome but certainly needs some planning and if there is critical applications which cant even with stand 5-10 ping drops then you need to be very careful.. Also on a Unix Server we lost nic connectivity and had to be rebooted to have it fixed.. i know this is weird so just play safe when taking downtime always unexpected happens so if we do the maths out of 20 Enclosures we did Firmware update of which we had 320 Blades(20*16 blades) 1 such incident, you do the percentage.

Hope this helps someone!!!

.

(OA)On-Board Administrator on C7000/C3000 - FIRMWARE UPDATE

Good day!All,

Welcome Back! Today i will share with you all the steps we followed to updated OA for c7000/c3000.

Basically there are couple of ways to accomplish the task

1. Using HP SPP , pointing to Active OA IP and re-mediating it
2. Manual download of bin file and updating it..

Before you even start thinking for updating Firmware always safe to take a backup, so login to Active OA using the IP address and on the left hand side expand Enclosure Settings, Configurations and click Show Config, a Text File will just pop up, save it which will be your backup file in-case you have to restore it.

Option 1 using HP SPP i already posted a article on how to apply it for a Windows Server, the steps are the same just that While Adding Node you type in the IP Address of the Active OA and choose the right type as below.

Note: I am using June HP SPP ..

Option 2 :

1. Download the Bin file from HP Site
2.Expand Active Administrator and click on Firmware update , it will show the current Firmware level

3.Click on Browse and point it to the bin file you downloaded and click Update..
4. Click ok on the below pop-up

5. You will see something like this below saying updating Firmware...

6.That was easy we just updated the OA Firmware , to double check click Rack Firmware and see if it shows updated Firmware version

In-case you guys wondering if this required downtime, well after doing so many updates i can confidently say that this doesn't require downtime and can be updated any time.
If anyone out there had any downtime doing this please share the information to me..


Hope this helps someone!!!!

LESSON LEARNED AFTER COMPLETING FIRMWARE UPGRADES FOR C7000/C3000 ENCLOSURES

Good day! All,

Welcome back! We recently completely Firmware upgrades on all our c7000/c3000  Enclosures and thought would share  all the lesson we have learnt and hopefully this helps some one..


1. Always a confusion on the Order of Applying Firmware to different components on the Enclosures, so below is the order we always followed and it works like a charm

a.Apply ilo Firmware for all the Blades in the enclosure
b.Apply Firmware update for the Physical Blades and if its Windows OS\ESXi apply all the drivers/Firmware as well
c.Apply On-board Administrator Firmware update
d. Last Apply the Virtual Connect Firmware update


2. IF you ever have to replace MidPlane make sure you make a note of the Old serial number and the same number needs to be updated using a putty session by connecting to new On-board Administrator, if you Virtual Connect domain will not connect and all the blades will not power on. A more deep article is out there on internet and also i have posted a article on the same too

3. The best way to do Firmware update on a Windows Server is using HP Support pack

4. While doing physical Servers it always best to copy the HP SPP on to local hard disk and run locally and also if you updating NIC card drivers make sure you keep a copy of old NIC drivers and new drivers handy

5.Always a confusion that when doing NIC card drivers update do we really have to break the teaming and do the update... Well we completed close to 80 odd Physical servers and we did all using HP SPP and we didn't break any NIC teaming and HP SPP is smart enough to update the Firmware. If you closely watch the steps it does, first it updates BIOS/NIC drivers and then it updates the rest..

6. Firmware updates on Windows Clusters you need to be extract careful.. HP SPP worked like a charm expect one instance where the NIC card update got struck and it basically disappeared NIC's from the Server.. so this is the reason to be safer side always make a backup of all the IP Settings and also Keep the NIC drivers locally on the Server.

7. Virtual connect Firmware update, don't use HP SPP because it just takes too long and there is no information at all on the Screen what is going-on on the back ground.. so the safe bet is using Virtual Connect update utility

Note: Firmware update on the Blades need downtime and Firmware update on the Virtual connect you will see packet drops even though you have 2 Virtual connect modules so plan carefully with downtime.

EVENT ID 47 Corrected Machine Check

Good day! All,

HP DL 580 G7 recently had a unexpected reboot , we couldn't identify the root cause of the unexpected reboot as there was no Memory dump so we started to take a look in patterns before the reboot then we can across this Event ID 47 - memory correction errors all over the logs..
So we went ahead and logged a case with HP and they said they have seen this errors for HP servers having XEON E7 family processor and they gave this link below for more information and steps to be modified to have this corrected and also pointed out that we have some Firmware to be updated as well.. which we are planning to do it soon..

http://h20564.www2.hp.com/hpsc/doc/public/display?docId=emr_na-c03282091


Hope this helps some one!!!!!!!!

Failover Clustering Features in vNEXT Server

Good day All,

I was checking the other day on the Clustering blog and stumbled across the vNEXT Failover clustering Video on the Channel 9

Here's the link:
http://channel9.msdn.com/Shows/Edge/Edge-Show-125

All the cluster fans should go check the new features which are going to come in the next release of Windows, in-case you can't view it below are the 2 cool new features

1. Windows Clustering with no Shared storage(kind of answer to VMWARE vSAN)
2. For Multi-site clustering no third part SAN Replication required, in-build features available..

I would encourage every one to go check the video and also Technical preview is out there so start testing.. and i will be doing the same.. so stay tuned for step- step process on both the 2 top features soon..

thanks...................

UNABLE TO CREATE NEW RAID ARRAY

Good day! All,

The other day someone showed me the below screen shot and started saying that i have tried all the things like Run as Administrator, Local admin etc. still when i open  HP Smart Storage Administrator don't see the option of New ARRAY creation

Can any one guess? i looked it and i know the answer? well i said guys go check if someone already have the same screen kept open on the Server.. so they checked there was Active RDP session but was not sure if it was kept open, i said disconnect the session and try it and well now we know the answer

Hope this helps some one :)

Outlook Email Registration Fails, Certificate root error

Good day All,

Our Citrix team recently installed Outlook on a Citrix Server and when they started configuring the Outlook client , the Client E-mail registration was failing with the below error ..

Well they tried the best and it was brought to my attention, seeing the error i said its Certificate root issue, they said to me yes we know but its from Verizon root certificate and it should take automatically..

I said hold on, let me show what i was referring.. so i browsed the website on my laptop and clicked on the certificate icon , so i opened and went to Certification path (highlighted in yellow) asked can you check on the Server if both Root and Intermediate Certificate is available in the Certificate.MMC console.

After some search they confirmed back to me that they don't see the Certificates, so i exported them from my laptop and applied on the Server.. then i asked them to test it..

While testing, smile on the face understood that it worked.. and then they came back to me and asked why this is Internet facing Certificate authority we never have to install certificate they said..

So i asked them one question, did you guys check if that Server has access to internet because all our Servers are locked down, well the obvious answer was No. Still they had question saying root certificate should all be Pre-installed on the Server..
I said very true but  as the Server is Windows 2003 OS , the Certificate root looks like changed at some point and root certificates was never updated, finally they got convinced.

On the side note please note that Outlook 2010 and above when Adding client , will use the Exchange web client URL to authenticate..

Hope this helps some one !!!