SSL on 2 Node Storefront 3.6 Load Balancing Servers

Good day All,

Welcome back!!!

We got a request to apply SSL to Storefront Servers recently even for internal users and also add 1 more Server for Load balancing .The current Setup we had was 1 Node Storefront Server with 2.x version running on it

Below are the steps we followed to fulfill the request

Step 1. We add a new Storefront Server with same 2.x version and added the Server to Load balancer. The steps are pretty straight forward..

a.Installed Storefront on the new node
b.Go to primary node, Click Add Server under Server Group in Citrix Storefront MMC and it will show a Authorization code as below

c. Login to new Secondary Storefront Server and click join to Storefront farm in welcome screen when you open Citrix Storefront MMC and then type in the Primary Node Server name and Authorization code , OLA the Server is added to Load Balancer
c. Requested a new Virtual IP(VIP) from F5 Load Balancer team and it was configured to Load Balance traffic on port 80 between both Storefront Servers.
d. Last step was to update the DNS record as it was earlier 1 node and it was pointing to the Primary Storefront Server IP. So we changed the DNS record to point to F5 Load Balancer VIP


Step 2: We wanted to upgrade the Storefront Servers from 2.x to 3.6 before we apply SSL. If you have a huge user base and can't afford to have users downtime for very long time then you probably will have to make sure you involve Load Balancer team during the upgrade process

a. Request Load Balancing team to remove Primary Server from Load Balancing.
b. Download the Setup and run the upgrade, simple straight forward upgrade.
c.After testing upgrade , add the upgraded Server to Load Balancing and remove the old Server from it
d.Upgrade the Secondary node
e.Request Load Balancing team add the other Server

As we had required downtime we didn't involve Load Balancer team and Servers was upgraded 1 Node at a time.

Step 3: Final step was to apply SSL on both the Storefront Servers.
As our requirement was to apply SSL for internal users we wanted to have the SSL traffic get terminated at Storefront Servers and not at F5 Load Balancer. Also we wanted if any users types in url then it should get auto redirected to 443 traffic and send to Storefront Servers.

a. A certificate was requested .Generating certificate etc are pretty straight forward process and there are so many articles out there so will not be covering it.
b.Requested the F5 Load Balancer team to reconfigure the Virtual IP(VIP) so that HTTP to HTTPS redirection works and HTTPS traffic is sent to both Storefront Servers.
c. Certificate as uploaded to Certificate.MMC store and also root and Intermediate Certificate was added to both Storefront Servers.
c.On both Citrix Storefront Servers under IIS, new binding was added for 443 as below under Default website and under the tab which says SSL Certificate , the certificate we processed earlier was pointed and applied.

d.Same steps as above needs to be done on the other node as well.
e. last Step was under Primary  Citrix Storefront Server MMC, right click Server Group and click change Base URL and change the record from HTTP to HTTPS
f. During the testing we started to see the below error when we browsed the URL.

Troubleshooting Steps performed:

1. We know that before we applied SSL storefront was working fine so to identify if this error is on both nodes or 1 single node , so i went ahead and shutdown Secondary Server.
2. When tested, Storefront was working fine and we able to browse the Apps.
3. So this time i powered off Primary and brought online Secondary, now we Started to see the same error. Now i know the issue is with Secondary node. As we had load balancer sending traffic to both SF Servers so we seeing the above error when we hinting the Secondary node.
4. To fix the issue i brought the primary node online and went to Server group and started to check around.
5. Anyone any guess? what would be the fix? well guess what when i checked the Last Synchronization time it was showing couple of days ago, so clicked under Actions to propagate changes to all the other Nodes, and ola the issued got fixed.

To make sure we do a through testing , following things was tested

1. Primary Node was shutdown and Secondary Node was tested with Node IP, Load Balancer IP and HTTPS and application was tested.
2. Vise versa was tested
3. Last step was both Servers was brought online and tested both Servers with IP, VIP IP,HTTPS.

So this is how we completed this request, hopefully this helps someone...........

Until next one you all have great day!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

RDP-TCP recreation on Windows 2012/R2

Good day All,

Welcome back!!!

Recently on a Windows 2012 R2 domain controller unable to RDP. We tried almost everything and eventually rebooted it still we had the same issue unable to RDP Server and using the  KVM we able to see everything was healthy.
So decided that we should try deleting RDP-TCP connection and see if this helps.
Well i remember in old Windows 2008 days you go into Remote desktop Session Host configuration (tsconfig.msc) and delete it and recreate it .. simple right well that is gone in Windows 2012,R2 .. as MS moved to a improved version of RDSH they incorporated all this to GPO Settings

Windows 2008:

Windows 2012:

After searching for a while there is absolutely no way we could recreate the RDP-TCP using gui so came across this excellent article which talks about how to re-create it by deleting and recreating the registry Key and it worked like a charm and we able to RDP back.

Note: If any one would like to thank , then follow the link and convey it, i take no credit for this one.

Recreate the default RDP Listener

How to recreate the RDP listener.

1. Export the following registry key:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp
2. Delete the following registry key:
   HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp
3. Copy and paste the below text into notepad, and save the file as RDP-Tcp.reg. Additionally, if the operating system is 2012 R2, another file will be required with the contents of the second box.
   
   

   Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp] "fInheritMaxIdleTime"=dword:00000001 "fPromptForPassword"=dword:00000000 "fResetBroken"=dword:00000000 "PdClass"=dword:00000002 "LoadableProtocol_Object"="{5828227c-20cf-4408-b73f-73ab70b8849f}" "UserAuthentication"=dword:00000001 "fDisableCam"=dword:00000000 "fInheritAutoLogon"=dword:00000001 "InteractiveDelay"=dword:00000032 "Domain"="" "fInheritReconnectSame"=dword:00000001 "SelectTransport"=dword:00000000 "MinEncryptionLevel"=dword:00000002 "fInheritShadow"=dword:00000001 "WFProfilePath"="" "fReconnectSame"=dword:00000000 "PdDLL"="tdtcp" "PortNumber"=dword:00000d3d "PdFlag1"=dword:00000000 "WdName"="Microsoft RDP 8.0" "fInheritMaxSessionTime"=dword:00000001 "WdFlag"=dword:00000036 "SelectNetworkDetect"=dword:00000000 "fLogonDisabled"=dword:00000000 "MaxDisconnectionTime"=dword:00000000 "Callback"=dword:00000000 "PdDLL1"="tssecsrv" "NWLogonServer"="" "MaxIdleTime"=dword:00000000 "fDisableEncryption"=dword:00000001 "fInheritCallback"=dword:00000000 "fDisableCcm"=dword:00000000 "ColorDepth"=dword:00000003 "PdName"="tcp" "fEnableWinStation"=dword:00000001 "OutBufLength"=dword:00000212 "PdFlag"=dword:0000004e "CallbackNumber"="" "CdClass"=dword:00000000 "Shadow"=dword:00000001 "fDisableCdm"=dword:00000000 "PdName1"="tssecsrv" "fInheritSecurity"=dword:00000000 "CdDLL"="" "LanAdapter"=dword:00000000 "fInheritResetBroken"=dword:00000001 "CfgDll"="RDPCFGEX.DLL" "InitialProgram"="" "fDisableClip"=dword:00000000 "InputBufferLength"=dword:00000800 "fAllowSecProtocolNegotiation"=dword:00000001 "fDisableAudioCapture"=dword:00000000 "Password"="" "CdName"="" "fDisableLPT"=dword:00000000 "CdFlag"=dword:00000000 "PdClass1"=dword:0000000b "fAutoClientLpts"=dword:00000001 "fAutoClientDrives"=dword:00000001 "fInheritCallbackNumber"=dword:00000001 "OutBufCount"=dword:00000006 "fInheritMaxDisconnectionTime"=dword:00000001 "MaxInstanceCount"=dword:ffffffff "KeyboardLayout"=dword:00000000 "fDisableExe"=dword:00000000 "AudioEnumeratorDll"="rdpendp.dll" "Username"="" "KeepAliveTimeout"=dword:00000000 "fUseDefaultGina"=dword:00000000 "fHomeDirectoryMapRoot"=dword:00000000 "fInheritColorDepth"=dword:00000000 "fForceClientLptDef"=dword:00000001 "WorkDirectory"="" "SecurityLayer"=dword:00000001 "DrawGdiplusSupportLevel"=dword:00000001 "WdPrefix"="RDP" "fInheritAutoClient"=dword:00000001 "fDisableCpm"=dword:00000000 "Comment"="" "OutBufDelay"=dword:00000064 "fInheritInitialProgram"=dword:00000001 "MaxConnectionTime"=dword:00000000 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp\VideoRemotingWindowNames] "AGFullScreenWinClass"="*" "MacromediaFlashPlayerActiveX"="*" "EVRVideoHandler"="*" "MicrosoftSilverlight"="*" "ShockwaveFlashFullScreen"="*"

   
   Additional 2012 R2 values:

   Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Terminal Server\WinStations\RDP-Tcp] "UserAuthenticationBackup"=dword:00000000 "MaxMonitors"=dword:00000004 "MaxXResolution"=dword:00000a00 "MaxYResolution"=dword:00000640

4. Double-click the RDP-Tcp.reg file and click Yes at the prompt.

So later after sometime started to poke around to see if there is any other way rather than deleting and creating the registry setting , well none in Windows 2012.

Well then i went back to Windows 2008 Server opened tsconfig.msc and tried to connect to a Windows 2012 R2 Server and see if it connects and sure enough i was able to connect and it showed the RDP-TCP settings and tried deleting and recreating it and it worked like a charm.

In Windows 2016 TP4 i don't see any option to delete it from MMC, tested it from Windows 2008, was able to connect and recreate the Listener as well..

Windows 2016 TP4:

                                                                                                                                                               

Let's hope final release of Windows 2016 probably this option comes back which i highly doubt it .
As long as Windows 2008 around it works ,if not we have to work with registry, no much choice :)

Hopefully this helps someone, until next one you all have a great day ahead!!!!!!!!!!!!!!!!!!!!!

SSL Offloading Where to do it? in Citrix Storefront 7.6

Good day All,

Welcome back!!!

We are trying to setup SSL for our new Citrix 7.6 farm and we had a  question from our Network guy asking how is password been sent when user types the user name and password on the Citrix URL.
Well  Citrix support was called they kept saying it was clear text so to double confirm it i setup a lab and installed Netmon on the Storefront server.

My Source IP was 192.168.1.5 and my destination Storefront IP was 192.168.1.72 and by delivery controller was 192.168.1.73.

So opened the URL , typed it username and password before i hit enter logged on to the Store front Server and installed the Netmon and started the capture.

Logged into client machine 192.168.1.5 and at Citrix URL login screen hit entered and i was logged in to Citrix and my published app showed up.So i quickly jumped to my storefront and stopped the Netmon.

Microsoft Netmon  is very simple and powerful tool , all you have to do is Click All Traffic you will see it beautifully segregates traffic between 2 hosts..

Well i high-lighted in yellow, now you know my username and password for my Citrix login.

So i was curious and wanted to check how is password been sent from Storefront to Delivery Controller and my delivery controller IP was 192.168.1.73 and for every one knowledge its is been set at port 80 for communication

if you see the screen shot Storefront is sending a xml query to delivery controller on port 80 and good thing is password is not been sent as clear text but its been de-crypted .
Well there are tools out there which can help in de-crypting so at least it not clear-text.

So the big question becomes how far we should go to encrypt the traffic?????

1. It depends on how the client is connecting? if external then SSL is a must on for Citrix URL
2.If all Client communication is internal probably we can get away with no SSL
3.Is SSL needed between storefront and delivery controller, it would depend on company to company how far we need to go and how secured you want.. understand there will always be over head associated to it.
4.Most of the companies i have seen is they offload SSL on load balancer either on F5 or Net scalar to avoid over head on the Storefront, that means traffic from Client to F5 or Net Scalar will be 443 and from there it will be port 80 to Storefront.


So testing,testing and more testing how secure and how how fast you need the apps to users will determine how much secure you need it.


before i conclude i added SSL for storefront so now see the communication from user desktop to Storefront.. its all been encrypted on port 443 and Secure..

Let me know how secure you have implemented in your environment , so until next one you all have good day!!!!!!!!!!!!!!!!!!!!

Windows Performance Analysis pdf for a grab!!!!

Good day All,

Welcome back!!!
 I came across this link i see ebook of Client Huffman Windows Performance Analysis is up for a grab

As of this posting this link is available

http://www.rccsonline.com/library/backend/books/Syngress%20Publishing%20Windows%20Performance%20Analysis%20Field%20Guide%20(2015).pdf

 Don't miss to download a copy, it is must read if you are a Windows Admin :)

 Untill next one all have a good day!!!!!

Steps replacing Failed Virtual Connect Module on a C Class Frame

Good day All,

Welcome back!!!! recently we had a failed Virtual Connect Module and we had to replace it. Its hot swap-able but there are certain things you need to follow before you replace it..

Important Step is to identify what the current Firmware version the new VC Module is? only if its matches to the Firmware Version of the VC Module which is failed then only you can replace.
If the Firmware Version doesn't match then you will have to either upgrade Current Firmware or downgrade the Firmware depending on your scenario.In our case the VC module we got was 4.10 Version and we had to upgrade to 4.20 Version

Another Important note is we need a IP for the new VC when inserting into Spare bay.You can use the failed bay IP make sure to un-check it or else you will get duplicate IP error.

Steps:

1. So ask the FE to insert the VC in spare bay
2. Login to OA and under inter connect bays you will see the new VC, expand and click on Information tab you will see the current Firmware Version, in our case it was 4.10 .So we had to upgrade the VC.
3.Now go to Enclosure Settings\Enclosure Bay IP Addressing\IP4\Interconnect Bays and then enable checkbox on the bay the new VC is inserted, type in the EBIPA IP that is IP address and other details in the column and click Apply

4.We have seen sometime that it doesn't show ip in the Interconnect Bays information tab under Management IP Address ,so try resetting the module and check,

5.if you still don't see the IP then you will have to login to Primary OA using putty and do the following commands

show EBIPA interconnect 

set ebipa interconnect x.x.x.x x.x.x.x baynumber     (Applies IP and mask for bay 1)

set ebipa interconnect gateway x.x.x.x baynumer      (Applies gateway for bay 1)

6. Login to Windows machine were you installed the VCSU Utility and run the command in interactive mode to check healthcheck under start Program files

Please enter action ("help" for list): healthcheck
Please enter Onboard Administrator IP Address: x.x.x.x
Please enter Onboard Administrator Username: *************
Please enter Onboard Administrator Password: *************

7.Again start the VCSU Utility in interactive mode and it will ask series of questions..

Please enter action ("help" for list): update
Please enter Onboard Administrator IP Address: x.x.x.x
Please enter Onboard Administrator Username: *************
Please enter Onboard Administrator Password: *************
Please enter firmware package location: C:\vc\vcfwall420.bin
Please enter Configuration backup password (Optional):
Please enter Force Update options if any (eg: version,health): health
Please enter VC-Enet module activation order if any (eg: parallel or odd-even
or serial or manual. Default: odd-even):
Please enter VC-FC module activation order if any (eg: parallel or odd-even or
serial or manual. Default: serial):
Please enter the time (in minutes) to wait between activating or rebooting
VC-Enet modules (max 60 mins. Default: 0 mins):
Please enter the time (in minutes) to wait between activating or rebooting
VC-FC modules (max 60 mins. Default: 0 mins):
The target configuration is integrated into a Virtual Connect Domain. Please
enter the Virtual Connect Domain administrative user credentials to continue.
User Name: ************
Password: *************

Note: All the steps remain the same if you are trying to downgrade or upgrade Firmware for a VC just that in the steps highlighted in RED above if you upgrading as in my case you need to put as health.If downgrading you need to type in there as version.

It takes about 30-40 mints depending on how many VC modules present and it will show at the end updated version

8.After confirming that Version is at the same level as the failed VC Module now ask the FE to replace the failed VC Module.

9.Wait couple of mints and you will see that new VC Module settles down and will show green if you see on the OA screen.

So this was easy, any questions free to ask!!!!!

If you don't have links to VCSU utility or Version Please find below:

Note: You can use SPP as well but VC are critical so best bet is to use VCSU utility which also takes a back up of VC domain before upgrade how cool is that.

Virtual Connect Support Utility User Guide:

https://h20566.www2.hpe.com/hpsc/doc/public/display?sp4ts.oid=4144084&docId=emr_na-c04567803&docLocale=en_US

Virtual connect Support Utility Version 1.11

http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_5e16cbb76d9e46e891ca04048d

 Download the Bin file you needed, In our case it was Version 4.20

     Download Virtual Connect firmware level of your choosing. (in your case version 4.20)

http://h20564.www2.hpe.com/hpsc/swd/public/detail?swItemId=MTX_3adcc3c4275f460c8d97cad17e

Excellent article which goes over if the module you received is having higher version and want to downgrade to replace failed module
http://h20564.www2.hpe.com/hpsc/doc/public/display?docId=mmr_sf-EN_US000005572


Hopefully this helps someone, until next one you all have a good day!!!!

How to trace a process which gets created and disappears!!!!

Good day All,

Welcome back!!!

We had little issue on a Server where we had to find out which process taskkill,exe is calling it.
Any ideas?????
If you saying Process Explorer, yell yes you can but if the new Process getting created and getting terminated is so fast you will not be able to trace it .

So nice little tool from System Internals is ProcMon, it has so many benefits and i believe if you want to be a successful System Admin you should always carry System Internal tools along with you and google on how these tools help!!! very very useful tools :)


So coming back to the point, i needed to trace what is calling taskkill,exe ,so ran procmon.. let it run for 10 mints and saved the logs...Just did ctrl +F to open Find tool and typed in taskkill,exe, boom it took me right there

Hopefully this helps someone!!! until next one you all have a good day!!!

VMSS2Core and Windbg saved the day!!!!

Good day All,

Welcome back!!! As part of root cause analysis we investigated 2 VM's issues and fixed them so thought of sharing this to all so that little steps from us help in fixing lot of problems.

We had 2 VM's ,one Windows 2008 and the other one is Windows 2012 both was in hung state.

When the Server was hung took a snapshot of the Server and using the vmss2core.exe generated the dump for it.
If you need more on vmss2core.exe check my other post

Windows 2008:
So using Windbg started to analyze the dump, the command i gave this time was !memusage

So out of 4 GB memory only 21 MB free space and only 448KB in standby causing the Server to run out of memory.


Windows 2012:

In Windbg i used this time !vm and this showed me like 500+ process of wmiprvse.exe.. So we know the issue what caused Server to hung.

Please bookmark this link that gives you a list of commnon Windbg command

https://blogs.msdn.microsoft.com/willy-peter_schaub/2009/11/27/common-windbg-commands-reference/

Hopefully this helps someone!!! and until next one you all have a good day!!!!