Wednesday, October 1, 2008

What a week (or two) . . .

Ever since I got back from taking vacation I have been covered up at work. The biggest thing I have been working on is trying to figure out why 7 of 16 domain controllers decided to start using 95%-100% of their CPUs while I was away. Since I have been back I have spent about 40 hours working on that problem. I was about to breakdown and call Microsoft ($250) but finally today I had a breakthrough.

About 6 weeks ago Ty (my faithful network admin) pushed out a group policy that ran a script. The scripts purpose was to get everything in the domain to do a "gpupdate /force". Well the DCs didn't like it one bit. While I was gone the DCs were either logged into or rebooted and then started trying to run the script but they would hang. When they hung the system process would max out the CPU on the server and any clock cycles left over were being eaten up by cmd.exe. Yikes.

I noticed the cmd.exe using a lot of the CPU but even when it was killed the system process still continued to have high CPU utilization so I dismissed it as a cause. Mistake.

The two main tools I used were Process Explorer and KrView (aka KernRate) but in the end just a good old fashioned guess solved the case.

So, we killed the script and the call for it from GPO, rebooted the servers, and all seems well.

1 comment:

Ken said...

Glad you found me. I'm on facebook as strobelight.audio, which would be easier to exchange emails. I recognize your mug. Now what made you think to do a search for 12stoneaudio?

Ken