Moving from one Domain to another for Windows 2003 Cluster - Lessons Learned

Good day All,

Welcome back!!!

Today i will go over challenges we had moving a 2003 Cluster from one domain to another.I know I know you guys would be wondering you still have Unsupported OS, well i guess we do and i may not be the only one in the world i guess :)
We are making progress, right now these Servers was in some other domain and we started doing consolidation and going to Windows 2012 AD, and eventually we will be moving to 2012 or may be 2016 Cluster who knows... but for now will share the lessons learned during the domain change.

Microsoft has a pretty good article our there and here is the Link.So please make to go over and do all the Policy changes believe me i tested my self in lab they Policy changes play a very key role.

Well reading the article you must be thing, hmm this is simple and pretty straight forward why do need another article, well remember Production Servers always never do smooth ride and we had our challenges which i wanted to share to you all

So let me list out..

1. Un-joining from old domain and rejoin to new Domin as per article shutting on passive nodes down and worked on first Active Node. Well unjoin went well, we updated the DNS for new domain in TCP/IP Properties and joined it, guess what we started to get this below error

The computer failed to join the domain. Please contact your domain
administrator and indicate that the computer failed to update the
dnshostname and/or servicePrincipalName (SPN) attritbute in its Active
directory computer account. Once the problem is resolved, you may join the
computer to the domain.

Not sure how how many of you know but when you try to join a domain a log file is setup in the Server c:\windows\Debug\ntsetup.log.
After review we found that Server was trying to join in some other Server in different location in domain but not to the nearest domain controller as specified in DNS1, DNS2 in TCP/IP Properties.
More strange was Computer object was getting created and just disappearing...
It was time to Pull in AD team and after checking couple of things, it was identified that when Unjoining the Server from domain, even though Server is getting unjoined witthout any errors but the Computer object in the trusted OLD domain was not getting deleted and that caused all the issue.
If anyone about to shout saying ID doesn't have domain Admin rights, we do :) but some thing to be investigated by AD team.

So the solution was pretty simple, unjoin from domain, delete the computer object from old domain wait for like 5-10 mints to replicated and then tried it and this time it was all ok

2. Moving on to 2nd issue,...  As i said above we are working on the Primary Active Node as per the Microsoft article..
After adding to new domain, we made all the Local Policy changes as per the Link above and changed the Cluster Service domain name and password and we started the cluster Service..
Well guess what cluster Service just timed out with System event log error 7031.

Now we got struck and started to reapply the Local Security Policy just in-case if anything got missed, nope that didn't help.
So i started to review the cluster logs i was seeing that when we start the cluster Service, Q(Quorum) drive was trying to come online and then going offline and shutting down all the cluster group resources and terminating the Cluster Service.

So first clue we identified was disk/LUN Issue. So to double check i ran the command

net start clussvc /fixquorum

guess what Cluster Service came online and when we checked Cluster Admin MMC, Cluster Name and Cluster IP was online and Q drive was in failed state.. and when check for other drives too those was failed... that was not good.

So i went to disk management and when observed i found that there was active LUN's with drive letters assigned and also same duplicate  LUN's was showing online as below... hmm then i thought that Multi-path issue and disk are showing twice in disk management.

So we reached out to Storage team and they told that for Multipath issue there is updated drivers and after applying and rebooting presto!!! Cluster Service came online with no issues.


3. So 3rd lesson we learned.. I don't think i mentioned before but its our windows 2003 SQL Active/Passive cluster and SQL was brought down and Services of SQL was put to manual too before starting this activity.So we asked the SQL team to change the SQL Service account domain name,account  and password and then started to bring the Service online, all SQL Service came online except the SQL Service account and it failed with this error

SQLServerAgent could not be started (reason: SQLServerAgent must be able to connect to SQLServer as SysAdmin, but '(Unknown)' is not a member of the SysAdmin role).

SQL Service in cluster started so it cant be permission issue so doing some search in one of blog they suggested to make sure the SQL Service account part of " Lock in memory" in local Security policy and reboot the Server.
Well after doing that SQL Service account came back online too with no issues.

Sorry my laptop crashed and couldn't go back to the same blog where it was suggested and thank them but i would like to thank them for the blog and also people like them is helping the community to fix issues.

After all ok on the first Node tested , we moved to 2nd Node and replicated every thing and all went fine with no issues.
Hopefully my lessons learned will help someone too!!!!

Special thanks to my buddy Prasanth who sticked around the issues along with me.

Till next one all have a good day!!!!!!!!!!!!!!!

Symantec Endpoint Protection creating Multiple instances on Citrix and Terminal Servers

Welcome Back!!!
We had issue where on Citrix Servers for every user when he opens a Application a instance of Symantec would open eating up lot of Memory on the background, something like this below

Colleague of mine had similar issue and they worked with vendor Symantec Endpoint and they suggested a registry key settings to fix the issue.

Prevent the process from starting by changing the registry value:

1. Click Start, Run and type “regedit” then click OK
2. Browse to the SMC key. In version of SEP older than 12.1 RU5, this is the same
location on 32- or 64-bit systems:
HKLM\SOFTWARE\Symantec\Symantec Endpoint Protection\SMC

In SEP 12.1.5 (12.1 RU5) and newer on 64-bit systems, LaunchSmcGui and most
other SMC keys and values have moved to Wow6432Node:
HKLM\SOFTWARE\Wow6432Node\Symantec\Symantec Endpoint Protection\SMC

3. Find the entry LaunchSmcGui and change it from DWORD 1 to DWORD 0 (add it if it
is not already present)

If you are unable to modify the listed registry keys, you can temporarily disable Tamper
Protection.

More info check this vendor website link

Hope this helps someone!!!

FAN making loud noise on DL 380p G8

Good day All,

Welcome back!!! We have a HP server in remote location which started to make loud noise..even though the Server was stable.

We opened a case with Vendor following troubleshooting steps was done

1. Asked us to update the Firmware drivers and we used June HP SPP
2. Still the same issue, so replace System Board same issue still no luck,
3.Suggested to upgrade to Oct HP SPP still same issue
4. We decided to come with 2 plans , replace again System Board,Fan if still issue then start pulling out the PCI cards each one at a time and test the Fan.

Well guess what after replacing the System Board the issue got fixed, now the big question is we did the same in Step 2 why it didn't fix the issue that time..
Vendor didn't have any answers but our thoughts is that usually System Board sit in warehouse for quite sometime and when then come they come with no update Firmware or drivers or it was not been tested before..

Hope this helps someone!!!!

Robocopy to copy 20 TB of data from NETAPP to Windows 2012R2 File Server cluster

Good day All,

Welcome back!!! one of my friend called me up the other day and he was saying that he needed help on Robocopy and guidance to move 30 TB of data from NETAPP to EMC Storgae,

Well i thought why are not do a blog on my experience and probably it will be helpful for someone.
Even though i was moving the data from NetApp to Windows 2012R2 File Server cluster the process should be the same.
Before i proceed the usually Cut over process and steps, lets dig little bit on what all attributes i used in order to achieve this.. this is my Robocopy.bat file.


ROBOCOPY /e /xj /ZB /r:2 /w:5 /LOG+:"C:\robo\Log.txt" /it /purge /copyall "\\Source IP\Source_Share" "\\Destiantion Server IP\destination_share"

@Echo Copying Complete
Pause


/E :: copy subdirectories, including Empty ones
/XJ :: eXclude Junction points. (normally included by default)
/ZB :: use restartable mode; if access denied use Backup mode
/R:n :: number of Retries on failed copies
/W:n :: Wait time between retries
/LOG+:file :: output status to LOG file
/IT :: Include Tweaked files
/PURGE :: delete dest files/dirs that no longer exist in source

/COPYALL :: COPY ALL file info

Important ones to look at the attribute and you should be aware
1. /ZB  is very important if source files\folders you don't have permission .. basically we telling that if no pemission too just treat as Backup mode and copy over to the destination location
2./PURGE : this one confuses lot of people , remember to make sure destination folders are empty.. if you have data it will be gone.So what this basically doing is replicating Source and Destination folders and anything not in Source will be removed .
3. /COPYALL : i will copy all the files and folders will the permissions in tact.

so this is how we did

1. Requested permission as Administrators on the Netapp filler
2.Work with business to identify how much of data we will move on weekly basis because this is huge data we looking and there is no way you can do 1 time cut-over
So we identified the root share folders and started setting up Robocopy.bat files with above line marked in yellow on the destination folder or any intermediate servers.
3. Now how do we move datacopy, if you have a dedicated link its better to use it.
4. Usually our change weekends starts on Saturday so we set up initial Sync to run over weekend ends before business starts on the Monday.
5. During the week we will do one or more sync so that during the final cut over we will have less data to cut over and less time.We will rename the log file so that it creates a new log file and not append the existing log file that way we don't have check though big log text file for skipped or errors
6. Finally on the day of cut over we  request all the users not to access the shares,do the final sync and do some testing to make sure all permission and everything are intact.

It took about 3-4 weeks to get through all the 20 TB data successfully with Zero loss of data.

Note: Windows 2012R2 File share cluster creation there are lot of articles on the internet if anyone is looking.

Hope this helps someone!!!

WINDBG saved my day one more time!!!!

Good day All,

Sorry its been a very busy couple of Months so started to get sometime so will catch up on couple of topics..

Last couple of days my laptop was acting up and when ever i tried to reboot the Laptop it will through a blue dump.. i thought it was one of thing and when retired again then same error so decided to fix it.

As Laptop was set to configure only Minidump i said to myself lets see if i can find anything in Mini dump if not then will set it for Kernal or Full dump.

Note; if any one wants to know what is difference in Mini,kernal and Full dump kindly google around there are tons of article about it.

So just loaded the windbg and loaded the Mini dump and ran the usual !analyze- v command.

As it couldn't load the Symbol to Module it showing the Module name as ntkrnlmp. If you see the Faulting Module it showing a driver called jpnrna6.

So either you click on that Faulting Module or type lmvm jpnrna6, it will show you more details on the driver.As this is third party driver there was no symbol associated to it was not loading the symbols and correct faulting drivers was not listing in the above !analyze command.

Now that we know the drivers location path, i checked the properties and found that its related to Juniper VPN client.
Searched on the vendor website found a updated drivers, uninstalled the existing VPN client and re-installed it and vola !!! issue got fixed.

So one more reason why little knowledge on windbg will help us do the initial troubleshooting.

Hope this helps someone!!!!

Printer Settings could not be saved, Operation could not be completed!!!!!!!!!!!!!!!!!!!

Good day All,

Welcome Back!!!
Today i will share with all a incident which happened.. We had a request to change Printer settings like new IP and Name for a printer on a Windows 2008 Print Server.

So as usually we right clicked on the Printer, changed the name and went on to Add new Port, new IP and when we clicked Apply we started to see this error as below

After some troubleshooting we identified that it needs Windows Firewall to be enabled and running as by default we have Windows Firewall disabled on all Windows Servers. As soon as we enabled Windows Firewall we where able to Apply the settings.Why we need this to be enabled i will let you guys do some research....

Hope this helps someone, until next time you guys have a good day!!!!!!!!!!!!!!!!!!!!!!!!!!!

VMHardware Version for various flavours of VMware vSphere

Good day All,

Welcome come back!!!

We started to perform Virtual Machine Hardware Version in our environment , having this chart little handy helped me, hopefully it may help someone!!!!

Windows 2008 booting failed with illegal Opcode

Good day All,

Welcome back!!!

We had patching this weekend and after we patched a Windows 2008 Server, the Server didn't come online and it started rebooting with the error as illegal opcode and with some flag errors as below

As part of troubleshooting we booted the Server with a Windows 2008 Bootable ISO and ran Check Disk..
After check disk was run, rebooted the Server and guess what ola!!! Server booted with no errors and came back online. What caused the error is still unknown, if anyone has any thoughts welcome to write back to me..


Hope this helps someone!!! till next time.. Have a good day!!!!!!!!!!!!!!!!

How to identify which Server Schedule job running from a Email it generates!!!!!!

Good day All!!!

We had some Schedule job setup for VMware capacity to be reported via Email every week for like 5 Virtual center and we needed some modification to be done on the job so team started to search for it.

Team started to search on lot of Servers including our Virtual Centers to check where is the Schedule job setup and they had no clue to identify it as this was setup long time ago.. and finally this came to me.. Does anyone guessed? how to trace back the Server which is generating that mail?

If anyone guessed Internet Header then yes you are absolutely right!!!!!!!!!

We had outlook 2013, in case you don't know how to check the Internet Header , Open the message, click on that little arrow in the Tags and that will pull the properties page with Internet headers .

Now copy the text to a notepad and look for something called "Received: From " that will show you the host name and IP Address of the Server which was sending the mail out... Ola! resolved!!!!

Hope this helps someone!!!!!!!!!!

DNS Resolving to Wrong Production IP internally on the Server

Good day All,

The other day we a had a little issue when client reported that when Pinging from outside its resolving to correct Production IP but when we run Ping from Internally on the Server it showing some other IP, so why is it ?

Couple of things to rule out

1. DNS is resolving properly so its not external issue
2. Is it local host file, so i checked if anyone added the IP address in local host file, but its not.
3. Checked other IP's on the Server and Server was resolving to MGMT IP internally from the Server. So i checked in TCP/IP properties if DNS registration is enabled but its not..

So any ideas to anyone what is the issue? well if anyone guess Binding Order for NIC's then yup that is the right answer

Binding Order place a important role when any application is trying to resolve DNS internally.. it will always PICK the 1st NIC on the top and resolve without going to localhost or DNS Server.

So always make sure that Production NIC is on the top of list

Hope this helps someone!!!!

Income Tax Filing if your Employer already deducted the Taxes

Good day All,

Well this is not technical topic but wanted to help guys as it’s this time of Year and we usually need to file our returns.

Note I am not a Charted Accountant, this is useful for guys who already paid their Taxes through your Employer and just have to file the returns.

Prerequisites before you start:

1.       PAN Number

2.       Form 16 and Form16A of Company

3.       Registered and Login details of Income-tax e-filing website.

https://incometaxindiaefiling.gov.in/e-Filing/UserLogin/LoginHome.html

• Now let’s start Step-by-Step Process to file your returns
• 1.       We have 2 ways to file returns either by downloading an Excel sheet or On-line Filing. I have done both but I think offline mode of Excel sheet is pretty simple, so this article I will go over Offline Mode.
• 2.       Click on the below link and download ITR 1and extract the Excel sheet to a folder.
• https://incometaxindiaefiling.gov.in/e-Filing/Services/DownloadItrLink.html

• 3.       You will see a file called 2015_ITR1_PR1.xls, open it. Please make sure to read the Read Me.txt file to enable Macros, this is very important

• 4.       A pop will show-up, read it… it tells you that anything Red Labels are Mandatory.. see below and say ok

• 5.       You start filling in the details in Income Details Sheet

• 6.       As said Above anything Marked Red are Mandatory so start filling all the details as possible till Row 32

• 7.       Row 41

1.       INCOME FROM SALARY: Open your Form 16 1^st line would be your Gross Salary, look couple of rows below after your HRA and Standard Deductions You will see something called “Income chargeable under the head SALARIES(3-5)” , make a note so that will be your INCOME FROM SALARY

2.       Row 42 Income from One House Property: If  you have Let out then mention it if not just leave it blank.

3.       Row 44 will be Auto Filled

4.       Row 46, 80c Maximum value is 1, 50,000.Open your Form16 and add all the amounts under Section 80C and fill in the field.

5.       Row 47 to 63 if you have anything mentioned in your Form 16 then fill the respective Fields.

6.       Row 65 Taxable Total Income is Auto Filled as below

7.       Row 66 to 78 you don’t have to fill anything just leave as it is

•          8. Now save your Excel sheet and Click on Validate as below

•          9. Sheet Income Details OK.If you see any error, a description will be present asking to correct   it.  
  
   
•          10. Click on TDS Tab Sheet and Fill Row which shows 19 TDS1, as seen below. TAN Number and   Name of the Employer can be found in your Form 16A, Income under Salary would be same Salary you mentioned in Income Details Sheet ROW 41 and Tax Deducted would be amount mentioned under  Tax payable in Form 16

Note: If you have mulitple Form 16, you need to put both the employers 

•         11. Click Save the sheet and Click Validate to make sure Sheet is ok.

•          12. Now go to Taxes Paid and Verification tab and Fill Row 27. Make sure you mention the number of Savings Bank Account you hold and all his details as below

•         13. Now save your sheet and click Validate

•          14. Now Click on Sheet Income Details and Click Calculate Tax ,
  
• 15. It will take a moment to calculate Tax and now if you go at Row 25 Under Refund of Taxes paid and Verification Sheet you will see that it Auto calculated and Value show as Zero(0).

 Note if your Refund is not Zero then either you paid more Tax or Less Tax and need to re-verify everything.

 This was easy.. Click Generate XML and you will see  xml generated in the same location as Excel sheet was opened.

The below is just to remind in case Calculate Tax was not done 

Now login to Income tax e-filing webiste

Click the Submit button a confirmation mail with IRT-1 will be sent to your registered Email ID.

Open the IRT-1 PDF, just double check to make sure everything is ok and take a print and send to Banglore Address given in the same PDF.

Well this was easy, all you need to do is spend like 5 bucks for Stamp and like 30mints of your time :)

Income tax Slab:

Hope this helps someone... till next time have a good day!!!

Smart Link on a c3000 HP Enclosure - Shared uplink set down caused half my ESX host unreachable

Good day All,

Welcome back!!! before i start sharing on the outage and how a small configuration would have avoided the outage we had, just want to share something on how to make sure you are redundant if you ever have to work on c 3000 as all embedded NIC's use only Interconnect module 1 to go out.
Confused see below


c3000:

c7000:

So you guys where you able to differentiate.. if not let me explain... On a Blade lets assume you have 2 Embedded Flex NIC's(LOM ) , so if you need to have redundancy you need 2 Interconnect Modules so that NIC 1 will have all his traffic go through Bay 1 and NIC 2 will have all this traffic go through Bay 2.. well this on a c7000.

Same Scenario on c3000 , both NIC 1 and NIC 2 will use Interconnect Module 1 to go out, so there is no redundancy for Interconnect Modules and also for the NIC's.. So how do you achieve that..
Well you take the help of Mezzanine Cards.. So what you have to do is add in a additional Mezzanine card which will be your additional 2 port NICs and its traffic will go through the Interconnect Module 2. So when you team them you need to make that you are teaming 1 on-board Embedded NIC along with 1 Mezzanine card NIC so that you get NIC's redundancy and also Interconnect Module redundancy.

Question would be then why not go buy a c7000, well cost and also if its small location you can buy 2 c3000 and have them redundancy for say like ESXi farm rather than 1 c7000 right?


Now let me go over the outage we had: We have 2 c3000 Frames and we got 4 ESXi blades on each frame and we have configured or created the Cluster in such way that 2 ESXi host from each Frame are part of each Cluster for Production and NON-Production.We also have created 4 uplink Sets 2 from each Interconnect Modules configured as Active/Active with same VLAN's going through both uplink sets.

The issue we had was Uplink from Interconnect Bay 2 got unlinked on the Switch end  and any VM's which was using that uplink was not communicating to Network ,VMotion link was down too and we unable to VMotion any VM's.

As part of Initial troubleshooting to bring VM's to Network we enabled Smart Link in Virtual Connect , basically what it does is it keeps checking if any UPLink Set is down, if any Link goes down it moves the traffic to other working uplink Set and we didn't had this configured on the Frames..

Steps to enable Smart Link:
1. Log into VC Manager
2. Edit the shared uplink sets, one at a time
3. Edit each of the Associated Networks within the shared uplink set and check the "Smart Link" box and click Apply, do not click Apply on the main shared uplink edit screen until all Associated Networks have been edited
4.After all has been edited click Apply on the main shared
5. Repeat for all shared uplink sets on both enclosures.

the reason for uplink down was some Network changes done and it was reverted back and issue got fixed.. but with this outage we atleast came to realize that Smart link could have save the day and i hope anyone reading this article will go back and check there settings..
I have lot more c3000 and started to do the same..


Hope this helps someone , till next time all have a good day!!!!

Task Scheduler - Logon failure when you try to create a TASK

Good day All,

I posted this fix sometime long back but still i see people getting struck and do all kind of things when they see the below error on Windows 2008 and Window 2012 when you try to create a Schedule Task

Windows 2008:

 Windows 2012:

The reason is because more security feature was added not to allow storage of passwords and credentials for network authentication .. how is it related to creating a schedule ..i will let you guys check it on Technet site..

So to fix this issue you need to Disable the setting in Local Security Policy as by default its Enabled. Please note changing the setting doesn't need any reboot or outage.

Hope this helps someone.. till next time all have a good day!!!

WINDBG for WINDOWS 2012/2012R2

Good day All,

I promised myself sometime back that will try to do lot of Post, but lately so much swamped never get sometime to draft it.. well always a excuse right :)

We Started to added lot of Windows 2012 R2 in our environment and one of the Server crashed with blue dump.. SoI  just made my Windbg files to carry around on my memory stick so i thought will share the steps you should do

1. Search for Windows 8.1 SDK or click here
2. You need to click Install and download

3.Click on sdksetup,exe which got downloaded. Please note you will need Internet access, this is not a standalone version of 81. SDK

4. Click on Download the Windows Software Development Kit for Windows 8.1 for installation on a separate computer and make a note Download Path and Click Next

5. Click No and Click Next
6. Un-check everything except Debugging Tools For Windows and click download button

Note: If anyone wants Windows Performance Analyzer/Recorder you can have it checked.

7.Browse to the download location , in my case C:\MY\8.1\Installers, in-case you  missed to make a note of download path then here is the default download path location
C:\Users\vadivelu\Downloads\Windows Kits\8.1\StandaloneSDK

8. Browse to the location and copy the file  "X64 Debuggers And Tools-x64_en-us.msi" on any OS version machine listed below.Just make sure that to uninstall any OLD debugging tools if present.

Note: this version of windbg can be used for Windows 7/8/81./2008/2008R2/2012/2012R2

9. Now Just double click the MSI file and it will just show progress bar and in 2 mints it will disappear.. don't worry nothing for us to configure, it just installed the debugger in the machine..

10. Browse to the below location and you will see something like this C:\Program Files\Windows Kits\8.1\Debuggers

11. Well that's it the standalone version of Windbg is ready.. just copy the x64 Folder and you can carry along on your memory stick or copy on to any Server and start debugging

12. Just a note don't forget to set your Symbol search path before debugging and also you should make sure internet is working or else Symbol's will not load

SRV*http://msdl.microsoft.com/download/symbols

13. If you are one of those guys like me, don't want to go internet or in corporate environment you have no access to internet then just carry a copy of symbols by clicking here

14. Download and extract to some folder and just point your Symbols Search path to local drive something like this
SRV*C:\Symbols_x64*http://msdl.microsoft.com/download/symbols

15. If you are one of those guys you have a dedicated Server for debugging and don't want to type the Symbol Search path all the time then you can do something like this, open a command prompt and type as below for one time and you are done..After now anytime you open Windbg or even Process Explorer Symbol search path is all Set

setx /M _NT_SYMBOL_PATH SRV*C:\Symbols_x64*http://msdl.microsoft.com/download/symbols

16. If you have dedicated Symbols Server where you download and share it , then you can set the Symbol search path as below something like this ...

setx /M _NT_SYMBOL_PATH SRV*\\Servername\foler*http://msdl.microsoft.com/download/symbols

We come to end now so till next time all have good day!!!

Different OS Versions upgrade Options

Good day All,

We started to plan upgrading lot of our Windows 2008\2012 Servers to Windows 2012 R2 and was wondering what can be upgraded.. so i made a little chart so that it can be handy for me and to all...

Hope this helps someone!!! 

VIRTUAL CONNECT DOMAIN BACKUP AND OA BACKUP FOR c7000

Good day All,

Today i will cover how to backup Virtual Connect Manager domain and OA if ever have to do it.

1. VC domain - couple of mouse clicks and you are done and the picture below explains it all..

2. On-board Administrator Backup/Restore:

Backup:
1. Login to Active OA
2.Click on Enclosure Settings,Configuration Scripts..
3.Click SHOW CONFIG and it will run the script and ask you save it

4.Always run SHOW ALL too that will give all the Firmware and IP details handy

Restore:
1. Login to Active OA
2.Click on the Enclosure Settings, Configuration Scripts..
3.Click File and Browse to the Configuration Script location you saved and click Upload.

Hope this helps someone!!!!