Friday, November 21, 2014

WINDOWS 2003 ,Insufficient System Resource error while taking TSM Backup

Good day All,


We have Windows 2003 which has close to like 1.5 TB of 1 disk and like 600 GB of couple of disk was converted to a Virtual Machine and after that we started to see TSM backup getting failure and Server would just hang with saying Insufficient System Resources and becomes unresponsive..

We have seen this error in past TSM when taking backup consumes all the Page Pool Memory and backups will fail.. so we basically follow the steps in the article, tweak the registry settings and maximize the Paged and NON-Paged pool..
http://support.microsoft.com/kb/304101

After tweaking Backup went fine for couple of weeks and we started to see the same error.. it go so frustrating that we started to have the issue every other day and lot of backup failures started to be reported..

We escalated this to client saying the Backup disk are too big and we should start to migrate the data to either a new Windows 2008 or 2012 OS .

At this stage i got involved and first question i had was any Backup Failure before this Server was refreshed to Virtual Machines and the answer we got was No.

So i was not really buying it because when it was Physical it was working fine with no backup Failure what migrating to Virtual Machine it was not.. So i started to big more.. enabled Poolmon and started to observe the trend.. After checking for some time i saw that the Paged pool was not growing beyond 230 MB and non-paged pool no beyond 100 MB, i said what we had tweaked the registry but still it not set growing.. So we started to recheck the registry setting but that looked all ok...


Well i opened by Windbg, used the kernel debug mode connected to the Server and when i run
 !VM command it clearly showed the Max values of Paged and Non-paged and that was too low even after registry changes..

Further digging i stumble across this article on /3 GB switch which clearly says that only half of Paged /Non-Paged pool will be used..

http://blogs.technet.com/b/askperf/archive/2007/03/23/memory-management-demystifying-3gb.aspx

Guess what when i checked my boot.ini file yes we had that /3GB switch.. so i felt having /3GB switch for a 4 GB memory Server with having IIS Application didn't really made sense to me so i went ahead and removed the /3GB Switch...
After that close to like 4 months now not a single backup failure..  In-fact the article which we used to tweak registry settings to maximize the Paged/Non-Paged Pool clearly asked to check if the boot.ini had that 3GB switch..

Well on the happy note issue got resolved but client decided to postpone the new OS build for now...


No comments:

Post a Comment