HP BLADE WILL NOT POWER UP AFTER MIDPLANE REPLACEMENT

All,

So other day we started with a planned activity of replacing a midplane on a C3000 enclosure..
what was the issue, let me elaborate more..

Issue: Fan speeds on the enclosure was always high at 90-95% and even 100% and 1 or 2 fans always keep failing..

Troubleshooting done before we wanted to replace midplane:

1. We update ILO firmware that did't help
2. replace fans couple of times and that failed too
3.change HP power settings that did't help

So last option the next step was to replace mid-plane..

We had about 150 VM's but fortunately we had a other ESX host on the cluster on different chassis and we moved all the VM's to the host and we had no down time...

Well we started with replacing the mid-plane that was pretty quick but after that none of the Blades started to power up..
So steps done to troubleshoot

1. We did a reset of all the VC Flex modules and that did't help
2. We tried power up the blades and still not luck.


so we decided to move the blade to a different enclosure then we tried powering up and i was able to power it on.. then we where confident that its VC issue..


Seeing that we said oh man looks like VC database is gone...


After some research we say link on HP website and said to our self what a miss... it says that we need to put back the old serial number of midplane so that VC can see the enclosure settings.. well the article made sense..

So we removed all the blades out, removed all the VC Flex modules and opened a Putty session to Active on-board administrator.. and run the following command
SET ENCLOSURE SERIAL_NUMBER

and then we put back the active VC Flex module and went to Management Console and said

1. Create a new VC domain
2.  In the Next screen  say  to repair the VC database with  changed OA module
click yes..

that's all it started to repair the VC database and all the settings started to come all..

The good part of all these is i took a VC database backup before we proceed with change.. so even though it was a mess we were confident that anything mess up we can recovery it using the restore....


So the big lesson learned is always check all possible scenarios .. it does't matter how small the change is we should always think about the worst possible scenario and always take good backups...

Till i encounter the next issue you all have a good day..