Windows 2003 SAN Migration from one storage to another.

Good day!

Welcome back!!! and Happy Diwali to all!!!

Recently we had a request to migration SAN from existing storage to a new Storage.As the SAN was pretty old SAN level migration didn't work and we manually had to do xcopy.

If anyone thinks this is pretty easy well to be frank nope, lot of risk involved with no support from MS and also remember drive details are in registry in the form of Signature, so if you change drive letter like we do in Windows 2008 and expect Cluster disk to come online it will fail.

Prerequisites before you start:

1.Take Full Back of both the Nodes
2.Take Registry backup on both the Nodes
3.Take System state backup for both the Nodes.
Note: if anyone things taking system state backup can we restore a Windows 2003, well absolutely and we have done it 2 times successful.
4. Make sure you have the password for Cluster Service account it is set under.Usually 2003 cluster are always set under Service accounts.
5. Make sure you have 2003 resource toolkit is installed and navigate to make sure you can see Clusterrecovery.exe under C:\program files\Windows Resource Kits\Tools.

Steps to perform:

Note: All these steps should be performed by login in on the Node with Cluster Service Account.
If your security doesn't allow to login using the Service Account then make sure you have ID which is part of Local Administrative group before you proceed.

1. Request storage team to assign Luns to both nodes
2. On confirmation that you can see the disk on all Nodes,Shutdown the passive node
3. Format the disk on the active node and assign a drive letter
4. Go to Cluster Admin, right click and add a disk resource and give some dummy name and select the defaults and click and select the new Formatted drive letter and finish it.
5. Bring the disk online and then power on the passive node online
6.Try failing over the disk to another node to make sure you can see the disk on the other node as well.
7. Make sure you do some read/write operations on the disk on the node which is active and confirm that the same can be seen on the other Node.
8. After confirmation disk looks good then perform the xopy from Source to destination disk and after completion make sure all the contents and folders are the same as Source disk.

Command we used:

xcopy Source_drive: destination_drive: /e/v/c/h/k/o/y

9. Now navigate to the clusterrecovery.exe and run it.It will ask you to connect to Cluster, in-case you have stopped the cluster make sure its running and online.After connecting when click next you will see a window asking if you want to replace a disk,Click next and in the final window you need to select Source disk and destination disk what you want to replace it with and Click Finish.
10.What this tool will do is update registry signature of the new disk.
11. Now if you open cluster admin, you will see that new Disk is renamed with the drive letter the old disk had and the old disk is commented as lost.
12. Right click the lost disk in clusteradmin and delete it.
13. Go to disk management and remove the drive letter Q to old disk and change the new Disk to Q.If you get a warning saying reboot is needed to take effect just say ok and be patient it will take sometime and then will show Q assigned to new SAN disk.
14.Verify in Cluster, try failing over to see if Q works fine and both nodes.
15. Last just reboot both the nodes and finally node fail over testing
16. We did 2 Clusters and above steps worked both the time.

The disk we performed had Shared drive, no issue reported after disk replacement and even shares showed up just fine.
Quorum was replaced that worked with no issues.
On google some reported disk signatures issues and had to do some registry fix etc was so not sure under what scenarios, but the steps above is what i have followed and was successful to migrate close to 6 drives.

Hopefully this will help someone, until next one you all have good day!!!!!!!!!!!!

Roaming Profiles, Terminal Server Profiles and Profile Versions

Good day All,

Welcome back, it been some time i did my last posting.. quite busy these days with so many things going on..

We are in the process of rolling out Folder Redirection for our Citrix users so during the process i had lot of confusions on Roaming profile and Terminal Server profile which loads when and why there are different version etc....

So i did some homework and thought to share the same so it may help someone else too..

Please the the screenshot below not sure how many of them understood

Let me tell you i didn't understand fully either and started to try different combinations to really understand this chart.

Before i post the result, the question why would you care??? well if you one of those users you need to implement Citrix UPM or Roaming Profiles Or if you introducing Windows 2012 R2 or Windows 2016 Citrix App Servers then yes you should have this knowledge because if no proper care taken you will hear lot of Profile corruption and also document missing in Profile issues.

How do we avoid it, well you should look at introducing folder redirection , that way Users My documents, desktop etc move along with User on any version of Windows they log in


Roaming Profile set for a User when logs on Windows 7:

A user when logs to a Windows 7 desktop/laptop gets a V2 profile created.

Roaming Profile set for a User when logs  on Windows 10:

A user when logs to a Windows 10 desktop/laptop gets a V5 profile created.


Roaming Profile V2 user logs to Windows 10 desktop/laptop:

A V2 Roaming Profile user when logs to a Windows 10 desktop/laptop then a new V5 profile will be created


Roaming Profile V5 user logs to Windows 7 desktop/laptop:

A V5 Roaming Profile user when logs to a Windows 7 desktop/laptop then a new V2 profile will be created



Roaming profile user on Windows 7 and launching Citrix\RDP on 2008\2012:


A V2 roaming profile user when launches Citrix or RDP on Windows 2008\2012 then same V2 profile will be loaded when no hotfix and registry changes are done.

A V2 roaming profile user when launches Citrix or RDP on Windows 2012 R2 then  V4 profile will be created if hotfix installed and registry changes are done.


Roaming profile user on Windows 10 and launching Citrix\RDP on 2008\2012:

Roaming Profile user on Windows 10 will have a V5 profile, so when he launches Citrix or RDP on Windows 2008\2012 then V2 profile will be created and loaded when no hotfix and registry changes are done.

A V5 roaming profile user when launches Citrix or RDP on Windows 2012 R2 then  V4 profile will be created and loaded if hotfix installed and registry changes are done.


Roaming Profile with TS Profile:

A V2 roaming Profile user if TS profile is attached then when launching Citrix or RDP on Windows 2008 then a new V2 TS profile will be created and loaded

A V2 roaming Profile user if TS profile is attached then when launching Citrix or RDP on Windows 2012 then a new V4 TS profile will be created if hotfix is installed and registry change are done.

A V5 roaming Profile user if TS Profile is attached then when launching Citrix or RDP on Windows 2008 then a new V2 profile will be created and loaded

A V5 roaming Profile user if TS profile is attached then when launching Citrix or RDP on Windows 2012 then a new V4 TS profile will be created and loaded if hotfix is installed and registry change are done.


Only TS Profile :

A new User with TS profile configured logs to a Citrix or RDP on Windows 2008 , then V2 profile will be created

A new User with TS profile configured logs to a Citrix or RDP on Windows 2012 , then V4 profile will be created if hotfix is installed and registry change is done ,if not it will search for any exsisting V2 profile to load or Will create a new V2 profile and will load.


Hotfix links:

Windows 8/Server 2012 (KB 2887239)
Windows 8.1/Server 2012 R2 (KB 2887595)

Registry Changes:

1. Locate and then tap or click the following registry subkey: 
   HKEY_LOCAL_MACHINE\System\CurrentControlset\Services\ProfSvc\Parameters
2. On the Edit menu, point to New, and then tap or click DWORD Value.
3. Type UseProfilePathExtensionVersion. 
4. Press and hold or right-click UseProfilePathExtensionVersion, and then tap or click Modify.
5. In the Value data box, type 1, and then tap or click OK.
6. Exit Registry Editor

Last Step: Make sure to reboot the Server. If anyone has question on which Server you going to do? well all the Server where you will RDP or Citrix App will be published .

I wanted to end with a example let say we have a new User, Roaming Profile has been set and the requirement is he will log to Windows 7 laptop, Windows 10 laptop and also will launch Citrix Application on Windows 2008, Windows 2012 how many profile will get created????? if hotfix and registry change is done on Windows 2012

If anyone say 3 then yup you got the concept right.. let me elaborate..

When User logs to Windows 7 he will get a V2 profile.
When the same User logs to Windows 10 he will get a V5 Profile
Same user When launch Citrix App on Windows 2008 then already created V2 profile will load.
Same user When launch Citrix App on Windows 2012 then new V4 profile will be created and loaded.

Hopefully this help some, until next one you all have a good day!!!!!!!!!!!!!!!!!!!!!!!!

Convert to .PFX certificate for Windows if you receive Server certificate (.crt) and Private key(.key)

Good day All,

Welcome back!!! We had a request to apply SSL for a website during the conversation it was suppose to be applied at F5 Load Balancer so the LB team took care of generating the certificate.

Little later client requirement changed and it was decided that SSL should be applied at Windows Web server and not on Load balancer and it should just redirect the traffic to web server.

So the LB team sent me the Certificate file which is .crt and a very confidential file is the .key file which has the private key for the certificate was sent only to authorized people.

Now i had to find a way to merge both so that i can generate a web server certificate with Privatekey embed.

Follow the below steps:

1.Got to following link https://slproweb.com/products/Win32OpenSSL.html and download either 32 bit or 64 depending on the OS



2. Its simple next,next installer and then you will see a folder called C:\OpenSSL-Win64 depending on which version you installed

3. Copy the .cert and .key file to following location C:\OpenSSL-Win64\bin

4.Open a Command Prompt with Administrative rights and change path to C:\OpenSSL-Win64\bin and run command as below in screen shot



Format of Certificate should be pkcs12
Dummy name to export the certificate as PFX
Private key path
Server certificate path
Friendly name
When you hit enter it will ask you set a password, remember that password or make a note if it..
After verifying then you will see that there is a file with .pfx extension generated as below



5. Now open certificate.MMC and import the .pfx and note during the import it will ask for the password you set during Step 4.



Enable the checkbox which says this key as exportable , in case for future use you want to export the certificate from certificate.mmc store.

6. Now open the certificate under personnel store and you will see now that the Server certificate has private key.

Let's assume you want to do vice versa that is you have .PFX certificate and you want to extract Private key(.key) for say Load balancer either F5 or Netscaller then you will have to follow the below steps

Import password is the password for the pfx
Enter Pem pass phrase is just a some password you will have to give

Note: the reason we have to do rsa temp key to private key is that it's observer without rsa command some spaces are in the key which when added to load balancer will through error.



The steps above helped and hopefully this will help someone too!!!!
Until next one all have great day!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

SSL on 2 Node Storefront 3.6 Load Balancing Servers

Good day All,

Welcome back!!!

We got a request to apply SSL to Storefront Servers recently even for internal users and also add 1 more Server for Load balancing .The current Setup we had was 1 Node Storefront Server with 2.x version running on it

Below are the steps we followed to fulfill the request

Step 1. We add a new Storefront Server with same 2.x version and added the Server to Load balancer. The steps are pretty straight forward..

a.Installed Storefront on the new node
b.Go to primary node, Click Add Server under Server Group in Citrix Storefront MMC and it will show a Authorization code as below

c. Login to new Secondary Storefront Server and click join to Storefront farm in welcome screen when you open Citrix Storefront MMC and then type in the Primary Node Server name and Authorization code , OLA the Server is added to Load Balancer
c. Requested a new Virtual IP(VIP) from F5 Load Balancer team and it was configured to Load Balance traffic on port 80 between both Storefront Servers.
d. Last step was to update the DNS record as it was earlier 1 node and it was pointing to the Primary Storefront Server IP. So we changed the DNS record to point to F5 Load Balancer VIP


Step 2: We wanted to upgrade the Storefront Servers from 2.x to 3.6 before we apply SSL. If you have a huge user base and can't afford to have users downtime for very long time then you probably will have to make sure you involve Load Balancer team during the upgrade process

a. Request Load Balancing team to remove Primary Server from Load Balancing.
b. Download the Setup and run the upgrade, simple straight forward upgrade.
c.After testing upgrade , add the upgraded Server to Load Balancing and remove the old Server from it
d.Upgrade the Secondary node
e.Request Load Balancing team add the other Server

As we had required downtime we didn't involve Load Balancer team and Servers was upgraded 1 Node at a time.

Step 3: Final step was to apply SSL on both the Storefront Servers.
As our requirement was to apply SSL for internal users we wanted to have the SSL traffic get terminated at Storefront Servers and not at F5 Load Balancer. Also we wanted if any users types in url then it should get auto redirected to 443 traffic and send to Storefront Servers.

a. A certificate was requested .Generating certificate etc are pretty straight forward process and there are so many articles out there so will not be covering it.
b.Requested the F5 Load Balancer team to reconfigure the Virtual IP(VIP) so that HTTP to HTTPS redirection works and HTTPS traffic is sent to both Storefront Servers.
c. Certificate as uploaded to Certificate.MMC store and also root and Intermediate Certificate was added to both Storefront Servers.
c.On both Citrix Storefront Servers under IIS, new binding was added for 443 as below under Default website and under the tab which says SSL Certificate , the certificate we processed earlier was pointed and applied.

d.Same steps as above needs to be done on the other node as well.
e. last Step was under Primary  Citrix Storefront Server MMC, right click Server Group and click change Base URL and change the record from HTTP to HTTPS
f. During the testing we started to see the below error when we browsed the URL.

Troubleshooting Steps performed:

1. We know that before we applied SSL storefront was working fine so to identify if this error is on both nodes or 1 single node , so i went ahead and shutdown Secondary Server.
2. When tested, Storefront was working fine and we able to browse the Apps.
3. So this time i powered off Primary and brought online Secondary, now we Started to see the same error. Now i know the issue is with Secondary node. As we had load balancer sending traffic to both SF Servers so we seeing the above error when we hinting the Secondary node.
4. To fix the issue i brought the primary node online and went to Server group and started to check around.
5. Anyone any guess? what would be the fix? well guess what when i checked the Last Synchronization time it was showing couple of days ago, so clicked under Actions to propagate changes to all the other Nodes, and ola the issued got fixed.

To make sure we do a through testing , following things was tested

1. Primary Node was shutdown and Secondary Node was tested with Node IP, Load Balancer IP and HTTPS and application was tested.
2. Vise versa was tested
3. Last step was both Servers was brought online and tested both Servers with IP, VIP IP,HTTPS.

So this is how we completed this request, hopefully this helps someone...........

Until next one you all have great day!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

RDP-TCP recreation on Windows 2012/R2

Good day All,

Welcome back!!!

Recently on a Windows 2012 R2 domain controller unable to RDP. We tried almost everything and eventually rebooted it still we had the same issue unable to RDP Server and using the  KVM we able to see everything was healthy.
So decided that we should try deleting RDP-TCP connection and see if this helps.
Well i remember in old Windows 2008 days you go into Remote desktop Session Host configuration (tsconfig.msc) and delete it and recreate it .. simple right well that is gone in Windows 2012,R2 .. as MS moved to a improved version of RDSH they incorporated all this to GPO Settings

Windows 2008:

Windows 2012:

After searching for a while there is absolutely no way we could recreate the RDP-TCP using gui so came across this excellent article which talks about how to re-create it by deleting and recreating the registry Key and it worked like a charm and we able to RDP back.

Note: If any one would like to thank , then follow the link and convey it, i take no credit for this one.

Recreate the default RDP Listener

How to recreate the RDP listener.

1. Export the following registry key:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp
2. Delete the following registry key:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp
3. Copy and paste the below text into notepad, and save the file as RDP-Tcp.reg. Additionally, if the operating system is 2012 R2, another file will be required with the contents of the second box.
   
   

   Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp] "fInheritMaxIdleTime"=dword:00000001 "fPromptForPassword"=dword:00000000 "fResetBroken"=dword:00000000 "PdClass"=dword:00000002 "LoadableProtocol_Object"="{5828227c-20cf-4408-b73f-73ab70b8849f}" "UserAuthentication"=dword:00000001 "fDisableCam"=dword:00000000 "fInheritAutoLogon"=dword:00000001 "InteractiveDelay"=dword:00000032 "Domain"="" "fInheritReconnectSame"=dword:00000001 "SelectTransport"=dword:00000000 "MinEncryptionLevel"=dword:00000002 "fInheritShadow"=dword:00000001 "WFProfilePath"="" "fReconnectSame"=dword:00000000 "PdDLL"="tdtcp" "PortNumber"=dword:00000d3d "PdFlag1"=dword:00000000 "WdName"="Microsoft RDP 8.0" "fInheritMaxSessionTime"=dword:00000001 "WdFlag"=dword:00000036 "SelectNetworkDetect"=dword:00000000 "fLogonDisabled"=dword:00000000 "MaxDisconnectionTime"=dword:00000000 "Callback"=dword:00000000 "PdDLL1"="tssecsrv" "NWLogonServer"="" "MaxIdleTime"=dword:00000000 "fDisableEncryption"=dword:00000001 "fInheritCallback"=dword:00000000 "fDisableCcm"=dword:00000000 "ColorDepth"=dword:00000003 "PdName"="tcp" "fEnableWinStation"=dword:00000001 "OutBufLength"=dword:00000212 "PdFlag"=dword:0000004e "CallbackNumber"="" "CdClass"=dword:00000000 "Shadow"=dword:00000001 "fDisableCdm"=dword:00000000 "PdName1"="tssecsrv" "fInheritSecurity"=dword:00000000 "CdDLL"="" "LanAdapter"=dword:00000000 "fInheritResetBroken"=dword:00000001 "CfgDll"="RDPCFGEX.DLL" "InitialProgram"="" "fDisableClip"=dword:00000000 "InputBufferLength"=dword:00000800 "fAllowSecProtocolNegotiation"=dword:00000001 "fDisableAudioCapture"=dword:00000000 "Password"="" "CdName"="" "fDisableLPT"=dword:00000000 "CdFlag"=dword:00000000 "PdClass1"=dword:0000000b "fAutoClientLpts"=dword:00000001 "fAutoClientDrives"=dword:00000001 "fInheritCallbackNumber"=dword:00000001 "OutBufCount"=dword:00000006 "fInheritMaxDisconnectionTime"=dword:00000001 "MaxInstanceCount"=dword:ffffffff "KeyboardLayout"=dword:00000000 "fDisableExe"=dword:00000000 "AudioEnumeratorDll"="rdpendp.dll" "Username"="" "KeepAliveTimeout"=dword:00000000 "fUseDefaultGina"=dword:00000000 "fHomeDirectoryMapRoot"=dword:00000000 "fInheritColorDepth"=dword:00000000 "fForceClientLptDef"=dword:00000001 "WorkDirectory"="" "SecurityLayer"=dword:00000001 "DrawGdiplusSupportLevel"=dword:00000001 "WdPrefix"="RDP" "fInheritAutoClient"=dword:00000001 "fDisableCpm"=dword:00000000 "Comment"="" "OutBufDelay"=dword:00000064 "fInheritInitialProgram"=dword:00000001 "MaxConnectionTime"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp\VideoRemotingWindowNames] "AGFullScreenWinClass"="*" "MacromediaFlashPlayerActiveX"="*" "EVRVideoHandler"="*" "MicrosoftSilverlight"="*" "ShockwaveFlashFullScreen"="*"

   
   Additional 2012 R2 values:

   Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp] "UserAuthenticationBackup"=dword:00000000 "MaxMonitors"=dword:00000004 "MaxXResolution"=dword:00000a00 "MaxYResolution"=dword:00000640

4. Double-click the RDP-Tcp.reg file and click Yes at the prompt.

So later after sometime started to poke around to see if there is any other way rather than deleting and creating the registry setting , well none in Windows 2012.

Well then i went back to Windows 2008 Server opened tsconfig.msc and tried to connect to a Windows 2012 R2 Server and see if it connects and sure enough i was able to connect and it showed the RDP-TCP settings and tried deleting and recreating it and it worked like a charm.

In Windows 2016 TP4 i don't see any option to delete it from MMC, tested it from Windows 2008, was able to connect and recreate the Listener as well..

Windows 2016 TP4:

                                                                                                                                                               

Let's hope final release of Windows 2016 probably this option comes back which i highly doubt it .
As long as Windows 2008 around it works ,if not we have to work with registry, no much choice :)

Hopefully this helps someone, until next one you all have a great day ahead!!!!!!!!!!!!!!!!!!!!!

SSL Offloading Where to do it? in Citrix Storefront 7.6

Good day All,

Welcome back!!!

We are trying to setup SSL for our new Citrix 7.6 farm and we had a  question from our Network guy asking how is password been sent when user types the user name and password on the Citrix URL.
Well  Citrix support was called they kept saying it was clear text so to double confirm it i setup a lab and installed Netmon on the Storefront server.

My Source IP was 192.168.1.5 and my destination Storefront IP was 192.168.1.72 and by delivery controller was 192.168.1.73.

So opened the URL , typed it username and password before i hit enter logged on to the Store front Server and installed the Netmon and started the capture.

Logged into client machine 192.168.1.5 and at Citrix URL login screen hit entered and i was logged in to Citrix and my published app showed up.So i quickly jumped to my storefront and stopped the Netmon.

Microsoft Netmon  is very simple and powerful tool , all you have to do is Click All Traffic you will see it beautifully segregates traffic between 2 hosts..

Well i high-lighted in yellow, now you know my username and password for my Citrix login.

So i was curious and wanted to check how is password been sent from Storefront to Delivery Controller and my delivery controller IP was 192.168.1.73 and for every one knowledge its is been set at port 80 for communication

if you see the screen shot Storefront is sending a xml query to delivery controller on port 80 and good thing is password is not been sent as clear text but its been de-crypted .
Well there are tools out there which can help in de-crypting so at least it not clear-text.

So the big question becomes how far we should go to encrypt the traffic?????

1. It depends on how the client is connecting? if external then SSL is a must on for Citrix URL
2.If all Client communication is internal probably we can get away with no SSL
3.Is SSL needed between storefront and delivery controller, it would depend on company to company how far we need to go and how secured you want.. understand there will always be over head associated to it.
4.Most of the companies i have seen is they offload SSL on load balancer either on F5 or Net scalar to avoid over head on the Storefront, that means traffic from Client to F5 or Net Scalar will be 443 and from there it will be port 80 to Storefront.


So testing,testing and more testing how secure and how how fast you need the apps to users will determine how much secure you need it.


before i conclude i added SSL for storefront so now see the communication from user desktop to Storefront.. its all been encrypted on port 443 and Secure..

Let me know how secure you have implemented in your environment , so until next one you all have good day!!!!!!!!!!!!!!!!!!!!

Windows Performance Analysis pdf for a grab!!!!

Good day All,

Welcome back!!!
 I came across this link i see ebook of Client Huffman Windows Performance Analysis is up for a grab

As of this posting this link is available

http://www.rccsonline.com/library/backend/books/Syngress%20Publishing%20Windows%20Performance%20Analysis%20Field%20Guide%20(2015).pdf

 Don't miss to download a copy, it is must read if you are a Windows Admin :)

 Untill next one all have a good day!!!!!

Steps replacing Failed Virtual Connect Module on a C Class Frame

Good day All,

Welcome back!!!! recently we had a failed Virtual Connect Module and we had to replace it. Its hot swap-able but there are certain things you need to follow before you replace it..

Important Step is to identify what the current Firmware version the new VC Module is? only if its matches to the Firmware Version of the VC Module which is failed then only you can replace.
If the Firmware Version doesn't match then you will have to either upgrade Current Firmware or downgrade the Firmware depending on your scenario.In our case the VC module we got was 4.10 Version and we had to upgrade to 4.20 Version

Another Important note is we need a IP for the new VC when inserting into Spare bay.You can use the failed bay IP make sure to un-check it or else you will get duplicate IP error.

Steps:

1. So ask the FE to insert the VC in spare bay
2. Login to OA and under inter connect bays you will see the new VC, expand and click on Information tab you will see the current Firmware Version, in our case it was 4.10 .So we had to upgrade the VC.
3.Now go to Enclosure Settings\Enclosure Bay IP Addressing\IP4\Interconnect Bays and then enable checkbox on the bay the new VC is inserted, type in the EBIPA IP that is IP address and other details in the column and click Apply

4.We have seen sometime that it doesn't show ip in the Interconnect Bays information tab under Management IP Address ,so try resetting the module and check,

5.if you still don't see the IP then you will have to login to Primary OA using putty and do the following commands

show EBIPA interconnect 

set ebipa interconnect x.x.x.x x.x.x.x baynumber     (Applies IP and mask for bay 1)

set ebipa interconnect gateway x.x.x.x baynumer      (Applies gateway for bay 1)

6. Login to Windows machine were you installed the VCSU Utility and run the command in interactive mode to check healthcheck under start Program files

Please enter action ("help" for list): healthcheck
Please enter Onboard Administrator IP Address: x.x.x.x
Please enter Onboard Administrator Username: *************
Please enter Onboard Administrator Password: *************

7.Again start the VCSU Utility in interactive mode and it will ask series of questions..

Please enter action ("help" for list): update
Please enter Onboard Administrator IP Address: x.x.x.x
Please enter Onboard Administrator Username: *************
Please enter Onboard Administrator Password: *************
Please enter firmware package location: C:\vc\vcfwall420.bin
Please enter Configuration backup password (Optional):
Please enter Force Update options if any (eg: version,health): health
Please enter VC-Enet module activation order if any (eg: parallel or odd-even
or serial or manual. Default: odd-even):
Please enter VC-FC module activation order if any (eg: parallel or odd-even or
serial or manual. Default: serial):
Please enter the time (in minutes) to wait between activating or rebooting
VC-Enet modules (max 60 mins. Default: 0 mins):
Please enter the time (in minutes) to wait between activating or rebooting
VC-FC modules (max 60 mins. Default: 0 mins):
The target configuration is integrated into a Virtual Connect Domain. Please
enter the Virtual Connect Domain administrative user credentials to continue.
User Name: ************
Password: *************

Note: All the steps remain the same if you are trying to downgrade or upgrade Firmware for a VC just that in the steps highlighted in RED above if you upgrading as in my case you need to put as health.If downgrading you need to type in there as version.

It takes about 30-40 mints depending on how many VC modules present and it will show at the end updated version

8.After confirming that Version is at the same level as the failed VC Module now ask the FE to replace the failed VC Module.

9.Wait couple of mints and you will see that new VC Module settles down and will show green if you see on the OA screen.

So this was easy, any questions free to ask!!!!!

If you don't have links to VCSU utility or Version Please find below:

Note: You can use SPP as well but VC are critical so best bet is to use VCSU utility which also takes a back up of VC domain before upgrade how cool is that.

Virtual Connect Support Utility User Guide:

https://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=4144084&docId=emr_na-c04567803&docLocale=en_US

Virtual connect Support Utility Version 1.11

http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_5e16cbb76d9e46e891ca04048d

 Download the Bin file you needed, In our case it was Version 4.20

     Download Virtual Connect firmware level of your choosing. (in your case version 4.20)

http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3adcc3c4275f460c8d97cad17e

Excellent article which goes over if the module you received is having higher version and want to downgrade to replace failed module
http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_sf-EN_US000005572


Hopefully this helps someone, until next one you all have a good day!!!!

How to trace a process which gets created and disappears!!!!

Good day All,

Welcome back!!!

We had little issue on a Server where we had to find out which process taskkill,exe is calling it.
Any ideas?????
If you saying Process Explorer, yell yes you can but if the new Process getting created and getting terminated is so fast you will not be able to trace it .

So nice little tool from System Internals is ProcMon, it has so many benefits and i believe if you want to be a successful System Admin you should always carry System Internal tools along with you and google on how these tools help!!! very very useful tools :)


So coming back to the point, i needed to trace what is calling taskkill,exe ,so ran procmon.. let it run for 10 mints and saved the logs...Just did ctrl +F to open Find tool and typed in taskkill,exe, boom it took me right there

Hopefully this helps someone!!! until next one you all have a good day!!!

VMSS2Core and Windbg saved the day!!!!

Good day All,

Welcome back!!! As part of root cause analysis we investigated 2 VM's issues and fixed them so thought of sharing this to all so that little steps from us help in fixing lot of problems.

We had 2 VM's ,one Windows 2008 and the other one is Windows 2012 both was in hung state.

When the Server was hung took a snapshot of the Server and using the vmss2core.exe generated the dump for it.
If you need more on vmss2core.exe check my other post

Windows 2008:
So using Windbg started to analyze the dump, the command i gave this time was !memusage

So out of 4 GB memory only 21 MB free space and only 448KB in standby causing the Server to run out of memory.


Windows 2012:

In Windbg i used this time !vm and this showed me like 500+ process of wmiprvse.exe.. So we know the issue what caused Server to hung.

Please bookmark this link that gives you a list of commnon Windbg command

https://blogs.msdn.microsoft.com/willy-peter_schaub/2009/11/27/common-windbg-commands-reference/

Hopefully this helps someone!!! and until next one you all have a good day!!!!

IP Changing on C7000 for ILO/OA/VC

Good day All,

Welcome back!!!
As part of some IP change we did IP changes for C7000 frame and below are the steps we followed....

Changing the IP address.

1.     Login to the Virtual Connect Manager assigned to the enclosure being worked on.

2.     Click on the ‘IP Address’ link under section Domain Settings
3.     Uncheck the box ‘Use Virtual Connect Domain IPv4 Address’ Click Apply. Do not proceed without doing this. It’s very important.

Note: Now if you ping VC ip that will not work.

4.     Login to the Active OA (On-Board Administrator) module for the schedule c7000 or c3000 enclosure to be changed.

5.     Click on the IPv4 link under the Enclosure Bay IP Addressing section.  Click on the ‘Device Bays’ tab.  Change the IP address for each slot along with the subnet mask and gateway.  New IP is in column G of the excel sheet. Following remains same for all ILO/VC and OA

a.      Subnet Mask: x.x.x.x

b.      Gateway: x.x.x.x

c.      DNS: x.x.x, x.x.x.

6.     Make sure that you check the enabled box rounded in red below, across each of the box where you are updating the IP/MASk/Gateway

7.     Click on the ‘Apply’ button towards the bottom of the screen to save the changes.

8.     Once applied validate the current address field boxed in yellow above is updated with the new IP address. It may take some time to update.

9.     Click on the ‘Interconnect Bays’ tab and change/add the assigned IP for each slot with New IP  Following remains same for all ILO/VC and OA

a.      Subnet Mask: x.x.x.x

b.      Gateway: x.x.x.

c.      DNS: x.x.x.x, x.x.x.

10.  Check the enabled box.

11.  There is no change in the NTP server IP.

12.  Click on the ‘Apply’ button towards the bottom of the screen to save the changes.

13.  The IP should be updated in the “current address” for the first two rows. It may not update for the remaining two rows, which are for CISCO Switches for SAN. Those two needs configuration change at switch end also.

So, if minimum two rows (mostly top most two rows) should be updated.

14.  Next go to “Enclosure Settings” and click on the Enclosure TCP/IP Settings link.

15.   Uncheck the box ‘Enclosure IP Mode’ and click apply. – Very important. Do not proceed without completing this step, else you may lose connectivity to OA and onsite support may be needed.

16.  When enclosure IP mode is selected both primary and secondary OA are accessible using the IP of the active or OA1, dynamically switching between the OA at the back ground when failover happens. Our purpose here is to access both OAs with their individual IPs during failover/fail back. Do the following to test the same after step 15.

a.     Login to the stand by OA. You will see nothing in that screen except an option for failover.

b.     Do the active to stand by failover.

c.      Now you would be logged out and you should be able to log into the active OA using the IP of the previous stand by OA.

d.     Once this verified proceed to next step.

17.  Log into active OA. Under the static IP Setting section change the first IP address, subnet mask, gateway of Standby OA to the new assign IP then click apply. 

***IMPORTANT***  DO NOT change the IP for active OA until you have validated you have connectivity for passive OA.

Note: Standby OA will be always at the right hand side (even after a failover) as underlined in red below.

18.  Verify that new IPs are applied to the OA. Log into active OA through putty and type in the below command and verify that stand by OA IP/Mask/Gateway is applied.

Show OA network standby

Once OA IP is changed, get in touch with Hawley, Jeremy of network team to update the VLAN at switch side for the OA you have worked. Provide him the MAC address of the stand by OA shown below

19.  Once standby OA is flipped to the new VLAN by network team, try pinging the stand by OA. If pinging log into the new IP of the stand by OA; and then switch it into an active OA. Now change the IP of the second OA, which is now standby. (Repeat of steps 30 and 31). Make sure you give the MAC of the next/remaining OA IP to Network team now.

20.  Once network team completed the VLAN change for the second OA also, make sure that you can ping and log in to the new OA (which is standby now) using the new IP.

21.  Validate that you can ping the new ILO IPs and Interconnect bay IPs updated in step 19 – 26.

22.  If we cannot ping any of the ILO IPs do the following

a.     Log into the active OA through putty

b.     Connect to the Server to which new ILO IP is not pinging using command

connect server <bay number of the server>

c.      Most cases the new IP can be force applied by simply putting it into DHCP and then reverting (turning off the DHCP option. Insert the following commands one by one after step B.

set /map1/dhcpendpt1 EnabledState=yes

set /map1/dhcpendpt1 EnabledState=no

If the new IP is applied and pinging then you can skip steps d, e and f below. Else proceed to step d.

d.     Verify the IPs / mask and gate way assigned to the ILO of the server. Below commands will help you to identify the IP/Mask and gateway assigned to the server.

Log into the active OA using putty

connect server <bay number of the server> -

Connects to the server where ILO IP need to be verified.

show /map1/enetport1/lanendpt1/ipendpt1  -

Shows IP & mask applied to the ILO of server highlighted in yellow above

show /map1/gateway1                                       

Shows the gateway applied to the ILO server highlighted in yellow above

e.     If any of the assigned value is not as per the new IP/mask/gateway then you can use the below commands to key in the same. Before running the command ‘connect’ to the server as in step highlighted in yellow above

set /map1/enetport1/lanendpt1/ipendpt1 SubnetMask=x.x.x.x

set /map1/enetport1/lanendpt1/ipendpt1 IPv4Address=<your IP address goes here>

set /map1/gateway1 AccessInfo=x.x.x.x

f.       Repeat the step C and confirm that correct values are applied.

23.  If we cannot ping the interconnect IPs do the following

a.     Connect to the active OA through putty

b.     Verify the IPs / mask and gate way assigned to the inter connect bay of the server using the below command (only first two values are of important)

show EBIPA interconnect

c.      If the new IP, Mask or gateway is not applied use the below commands to set the same.

set ebipa interconnect x.x.x.x x.x.x.x 1     (Applies IP and mask for bay 1)

set ebipa interconnect gateway x.x.x.x 1      (Applies gateway for bay 1)

Repeat the above step for interconnect bay 2 as well.

d.     Repeat step B and make sure that new IP/mask/gateway are applied to first two interconnect bay.

e.     If inter connect bay is still not ping-able get in touch with network team to test the firewall settings.

24.  Once both OAs has been assigned IPs ‘check the box ‘Enclosure IP Mode’ and click Apply

(Reversal of what is done in step 14 & 15)

25.  Login to the first virtual connect module IP that has the newly assigned IP located in interconnect bay 1 in order to login to the Virtual Connect Manager.

New IP Assigned at Step 9

26.  Once in the Virtual Connect Manager click on the ‘IP Address’ link under the Domain Settings section.  Click the box ‘Use Virtual Connect Domain IPv4 Address’ 

(Reversal of what is done in step 3)

27.  Enter the newly assigned IP address, subnet mask and gateway for the Virtual Connect Domain name.  New IP is therein column G, against Virtual Connect Mgr. Mask/Gateway remains the same.

28.  Click Apply.

29.   Click the ‘Configuration’ link under the Domain Settings section.

30.   Type in the new Virtual Connect Domain Name in the ‘Name of the Virtual Connect Domain Name:’ field. Just type in the NETBIOS name alone excluding domain name.

 

31.   Click Apply

ILO Configuration

32.  List out all the blades on the enclosure.

33.  Log into the ILO of each of the server.

34.  Go to Network èILO Dedicated network port è IPv4 Tab

35.  Uncheck “enable DHCPV4”

36.  Check “enable DNS server registration”

37.  Click submit. And it will prompt for ILO rest, do not reset now.

38.  Next click on the IPv6 tab

39.  Uncheck all the boxes.

40.  Click Submit. It will save the configuration. Do not rest the ILO yet.

41.  Next go to the general tab. Update the hostname and domain name. Details that need to go into these fields is given in the excel sheet in column H across each server name.

42.  Once done click on submit button. It will give a warning to Rest the ILO. Rest the ILO now by clicking button marked in red.

Kindly note that the above steps needs to be done for each of the ILOs of each individual blades on the frame that you are going to work.

Hope this helps someone!!!! and i got to pass on special thanks to Prasanth my buddy for capturing the screenshot and document it..
Until next one you all have a good day!!!!