NETWORK UNPLUGGED ON BLADE 460G7

Good day All,

Last couple of weeks has been extremely busy couldn't share much.. promised to myself saying to do at-least 2-3 post, let see how it goes..

Well the issue i am going to share has went for like 3 weeks before we buried to bed.
Ok this all started on a Blade in C7000 with 3 NIC's showing unplugged..
Initial Troubleshooting steps we performed still the issue persisited are
1. Reset the blade in OA
2. Re seating the blade Physically
3.Firmware Upgrade for NIC's
4.Motherboard replacement

After all the initial troubleshooting failing we decided to move to next level that is replace Virtual Connect Module.. As the NIC which was failing was going through one VC module 2 we thought we will replace that and give it a try.. Before we went to Business to get downtime, one more alert popped up and this time one more Server has same issue but this time the traffic is going through the other VC Module which is 1 in our case.

So like weeks of discussing with Vendor and internally we came up with the below plan

1. VC Domain and OA Backup
2. Reset the VC Modules
3. Replacing the VC Modules.. and for every one benefit replacing VC Module is not that easy..
Please find the link which will be useful
http://h20564.www2.hp.com/hpsc/doc/public/display?docId=mmr_kc-0115081
As we had to replace 2 VC Modules we had to get 2 Temp IP's and 2 Free slots in the c7000 enclosure to Flash the VC Modules to 4.20 and as our Frames was linked while using the vscu utility to update the Firmware you have to use -f option to force upgrade.
4. All the 3 steps fails last try was to replace Midplane.. and i already have post which i did, the challenges we faced to replace Midplane, click the Link and also more detail information on the vendor website

As it was our critical Server infrastructure we had to do lot of Pre-works to make sure we are all ready , like
1. Screen shots for all the Server profiles and VLAN Information
2. Shared up links
3.IP's etc..

So the big day came we anxiously started with our step which was to Reset the VC Modules..
Way to do it:

1. Please check which is the active VC Module( Incase you not aware , login to Virtual Connect Manager and click on Help and then click "About Virtual Connect Manager' it will list all the VC Modules listing which is Primary
2. Failover the VCM from Primary to Standby and reseat the VC Module
3.Failed over now Primary to Standby and reseat the VC Module
4. Guess what all the NIC's got connected and issue got resolved..

We are glad that we never had to do rest of the steps :)

Hope this helps someone and if anyone as question free to post it and glad to help!!!!

F5 load Balancer and IIS certificate issue

Good day All,

Today i will share with all F5 and IIS certificate issue i was pulled into sometime ago.. In-fact when the issue came to me i was said that all configuration on F5 as been set properly and certificate as been installed on IIS Web server but still the page is unavailable when the tried to access the Server  using https from external network.
Couple of questions i asked

1. is the certificate working properly internally and the answer was Yes
2.I asked what kind of offloading we have configured on F5, Client-side SSL or Server-side SSL and the answer was Server-side SSL
3.last question was if Server-side SSL, are we using 2 certificates 1 for F5 and 1 for IIS web server or we are installing 1 Certificate on F5, exporting and importing the same certificate with Private key on Web server and the answer was only 1 certificate.

for starters if you wondering what is Client-side and Server-side, please check this Manual of F5 it goes in depth on the same..

So i said please hold on and let me check something because i kind of know what is the issue? any guess from anyone? well here you go the answer see below? can you tell me what is missing?

If any one guessed don't see the Private Key then the answer is Yes.. there is no Private key for this certificate so all the traffic from F5 to IIS Web server was encrypted but IIS doesn't have the Private key to decrypt. Why we don't see the Private key, the possible reasons
As the certificate request was generated from F5 and when it receives the certificate it gets the certificate and Private key separately and F5 team passed the same certificate without Private Key to be applied on IIS not knowing Server would need Private key.

So how do we fix the issue well there is neat little tool call openSSL , simple download and install which basically merge Certificate and Private Key in a PFX format to be imported to IIS..


C:\OpenSSL-Win32\bin>openssl.exe pkcs12 -export -out name.pfx -inkey w.key -in w.
crt -name test
WARNING: can't open config file: /usr/local/ssl/openssl.cnf
Loading 'screen' into random state - done
Enter Export Password:
Verifying - Enter Export Password:


well the above command will create a PFX , so we re-imported the certificate to Certificate store and reapplied the certificate to IIS.

Hope this helps someone!!!!!!!!!!

The backup operation that started at ... has failed with following error code '2155347997' - SYSTEM STATE BACKUP- WINDOWS 2008

Good day All,

We started updating Firmware on lot of Physical Servers and as part of Pre-requisites we started taking System state backup's and lot of 2008 Severs system state backup was failing.
In-case you guys are wondering what we will do with system state backup? can we restore the Server back in case of disaster well, check this article i have posted, 2 Servers was successfully restored with the process in this link
After some search and couple of these blogs really helped so what really the issue is?
Well let me see i can put in simple and there are lot of blogs which tells in-depth if you google on enumeration errors.. so what basically happening while taking System state backup is Backup process is making sure that all the image paths for the Service is correct and in that process if it sees any invalid paths it stops the backup process with error code...

http://blogs.technet.com/b/askcore/archive/2010/06/18/ps-script-for-blog-enumeration-of-the-files-failed.aspx
http://h10025.www1.hp.com/ewfrf/wc/document?docname=c03921757&cc=ca&dlc=en&lc=es

So for starters download the Powershell script, enable Script execution in PS and run the PS script.. you will see output something like this..

     Service Name    :
     Service Caption :
     Registry key    :
     Value           :
     Reason          : The service path contains spaces, the whole path needs to be enclosed using double      quotes

The good part about the script is it tells what needs to be done so be cautious when changing something in registry .. below are the few things i have seen as errors..

1. The service path contains spaces, the whole path needs to be enclosed using double quotes
2.The service path contains a forward slash. Only paths containing an inverted slash are supported
3.The service path does not have a proper path format. Only paths beginning with [<Drive>]:\ format
are supported.
4.The service path contains invalid characters. Characters < > : " | ? cannot be used in a file path
5.Non-Existed Volume

Couple of things i have noticed

1. Cluster Servers, the server is active on one node and the respective Service is active on that node, you will see that Service warning failure in the PS script you run.. in that case we couldn't never fix it.. before we do activity we failed over the node took the system state backup and then proceeded with firmware update..

2. Warnings in the Script can be ignored

3. the service path contains a reparse point. Paths containing a reparse point are not supported - can be ignored too..

As I progress and if i encounter more backup failure errors will update this post.. 

Hope this helps someone!!!!

WHICH PROCESS ID LISTENING ON WHICH PORT? ???

Good day All,

Welcome back and a very Happy New Year.. this is my first post this year and hopefully will share lot more than last year.

The other day some one asked me i have a process running and need to find out which port is it listening? Well that was easy i know sometimes little things are hard to remember, so i said you can get it from netstat..
I don't want to go over different options available in netstat, let you guys explore but for this to be achieved you should use something like this
netstat -nao
this will scroll through the list of all the tcp/udp ports listening and which Process id is being used.

You will realize that its the big list , so if you know which port number you trying to search then you do something like this
ex: netstat -nao | find "161"

What i am trying to do is find out which process ID\Process is listening on Port 161..If any one have a question how did you correlate PID and Process then open task manger add a column PID , there you know now.
So in the below example i clearly see that PID 1960 is used by SNMP on the default port 161.

.
Hope this helps someone!!!

c7000 - PORT MAPPINGS

Good day! All,

Welcome back.. the other day someone asked me how the port mappings of Nic's and Interconnect modules work for a Half-height and Full-height blades i showed them 2 screen shots i always carry around..

• Half-height server blades typically will have 2 embedded Gigabit NICs and two c-Class PCIe mezzanine option connectors. A half-hright server configured with one dual-port Gigabit NIC mezzanine card and one quad-port NIC mezzanine card provides eight independent NICs
• Full-height server blades typically have four embedded Gigabit NICs and three c-Class PCIe mezzanine option connectors. A full-height server configured with three quad-port Gigabit NIC mezzanine cards provides 16 independent Gigabit NICs

Half-Height Server:

Full-Height Server:

i know i am just sharing little bit of so much around Blades and enclosures and i think HP has very good guides up there explaining more on the same..

Hope this helps someone!!!

c7000/c3000 Enclosures - VIRTUAL CONNECT INTERCONNECT MODULES FIRMWARE UPDATE

Good day! All,

Welcome back!, Today i will share with you all the steps we followed to update Firmware for Virtual Connects on either c7000 or c3000 Enclosures.

Before you think of updating Firmware its very much necessary that you take a backup for the Virtual Connect domain

We have 2 ways to accomplish the task

1. Using HP Support pack(HP SPP)
2.Virtual Connect Support Utility - (vcsu-1.9.0)

Assumption is that you are having 2 Virtual Connect Interconnect Modules in bay 1 and bay 2 for redundancy... If not the case then you are looking at lot of downtime because this Firmware updates on each Interconnect Module takes about 20-25 approximately .

So the next question would be how much of downtime is required if we have 2 Interconnect Modules and using VCSU Utility, well i did updates for close to like 20 Enclosures and we always saw like 5-10 Pings packs drops.The beauty with VCSU utility is that it first finishes the standby Interconnect Module and then it fails-over and when doing Active Interconnect Module we have seen the packet drops.

Option 1: Using HP SPP i already posted a article on how to apply Firmware for a Windows Server, the steps are the same just that while Adding Node you type in the IP Address of the Virtual IP of Interconnect Module and chose the right type as below.

If anyone has a question did we ever update Firmware using the HP SPP then the answer is yes only 1 time for like 20 Enclosures and believe me that is not fun at all.. After you hit enter and say re-mediate then you are like watching the browser with no update and no proper information, hopefully in near future this may change but till i writing this article the safe bet would be using VCSU utility..

Option 2: Virtual Connect Support utility

1. Just search the HP site and download the latest Virtual Connect Support utility , the version we used was 1.9 , its just next next and you should be able to install the tool
2. On the desktop you will see shortcut, if not just browse to this location C:\Program Files\Hewlett-Packard Company\Virtual Connect Support Utility and right click on VCSU-CommandPrompt and say create a short cut on the desktop.
3. Go out to HP site and download the Firmware for interconnect modules, usually a BIN file.
4.Click the VCSU-CMD and then type the following command

vcsu -a healthcheck -i IP Address of OA -u Username of OA -p Password of OA -vcu Username of VirtualConnect Manger -vcp Password of Virtual connect Manager

ex. vcsu -a healthcheck -i 192.168.1.1 -u localadmin -p password -vcu Administrator -vcp password

Before you proceed to update the firmware make sure the health check result is all passed, if not re-mediate it first and then only proceed further.

5. Now type the following command to update the firmware..

vcsu -a healthcheck -i 192.168.1.1 -u localadmin -p password -vcu Administrator -vcp password -l "Bin file you just downloaded"

You will prompted with yes/no option to proceed, just type yes..

Lesson we learnt is that after clicking yes it goes fast with Percentage and around like 20 %  for like 15-20 mints it just stops so just be patient and don't close the window.. this is when we have seen that  it does the Firmware update on the stand by Module and then  fail-overs to Active module and if you have started any continues Pings you will see the packet drops now..
After like 40-45 mints if everything goes well you will see something like this..

Well updating Firmware on Interconnect modules are not that cumbersome but certainly needs some planning and if there is critical applications which cant even with stand 5-10 ping drops then you need to be very careful.. Also on a Unix Server we lost nic connectivity and had to be rebooted to have it fixed.. i know this is weird so just play safe when taking downtime always unexpected happens so if we do the maths out of 20 Enclosures we did Firmware update of which we had 320 Blades(20*16 blades) 1 such incident, you do the percentage.

Hope this helps someone!!!

.

(OA)On-Board Administrator on C7000/C3000 - FIRMWARE UPDATE

Good day!All,

Welcome Back! Today i will share with you all the steps we followed to updated OA for c7000/c3000.

Basically there are couple of ways to accomplish the task

1. Using HP SPP , pointing to Active OA IP and re-mediating it
2. Manual download of bin file and updating it..

Before you even start thinking for updating Firmware always safe to take a backup, so login to Active OA using the IP address and on the left hand side expand Enclosure Settings, Configurations and click Show Config, a Text File will just pop up, save it which will be your backup file in-case you have to restore it.

Option 1 using HP SPP i already posted a article on how to apply it for a Windows Server, the steps are the same just that While Adding Node you type in the IP Address of the Active OA and choose the right type as below.

Note: I am using June HP SPP ..

Option 2 :

1. Download the Bin file from HP Site
2.Expand Active Administrator and click on Firmware update , it will show the current Firmware level

3.Click on Browse and point it to the bin file you downloaded and click Update..
4. Click ok on the below pop-up

5. You will see something like this below saying updating Firmware...

6.That was easy we just updated the OA Firmware , to double check click Rack Firmware and see if it shows updated Firmware version

In-case you guys wondering if this required downtime, well after doing so many updates i can confidently say that this doesn't require downtime and can be updated any time.
If anyone out there had any downtime doing this please share the information to me..


Hope this helps someone!!!!