Tag: DB2
SOS – Power tool that helps to identify lock contention issues in .NET applications
by admin on Mar.19, 2009, under Storage
SOS – Power tool that helps to identify lock contention issues in .NET applications
In my last few blogs, I have written about tools such as perfmon, Vtune and how you can use these for .NET based applications. Today, I am going to talk about another tool called SOS. This is a very powerful tool, that I have used it many times to identify lock contention related issues. This is actually a DLL which has managed debugging extensions for debugging and fixing functionality issues in .NET applications. It is shipped as sos.dll and is in the same directory as CLR. Multi core processors have become norm and we have upto 24 cores in a machine (called Dunnington). So lock contentions are almost at top of list of issues that are affecting scalability and performance. I have seen this in many ISV applications.
It is not that easy to use SOS. Since it is a DLL, you need to load it in a debugger. You can load SOS in Visual Studio or in other debuggers such as Windbg or NTSD. I have been using SOS inside windbg and so will continue to write this article with windbg in mind. I will also cover some information (commands etc of windbg for folks who have not used windbg before).
Important Notes: Note that you need pdb file for mscorwks.dll in order for SOS to work. Also the version of SOS.dll should match that of mscorwks.dll. For those who don’t know, mscorwks.dll is the main CLR dll which is shipped with .NET Framework redistributable package.
Installing windbg:
Install windbg from http://www.microsoft.com/whdc/devtools/debugging/default.mspx to a directory (let us call c:\debuggers).
Where to get mscorwks.pdb? – When you install .NET SDK, you will get mscorwks.pdb installed in your machine. But if you have not installed, just follow these steps to automatically download it (assuming you have internet connection).
c:\Windows\Microsoft.NET\Framework64\v2.0.50727>dir mscorwks.dll
09/12/2007 11:11 AM 10,451,968 mscorwks.dll
c:\Windows\Microsoft.NET\Framework64\v2.0.50727>c:\Debuggers\symchk.exe /if .\mscorwks.dll /s SRV*c:\websymbols*http://msdl.microsoft.com/download/symbols
SYMCHK: FAILED files = 0
SYMCHK: PASSED + IGNORED files = 1
c:\Windows\Microsoft.NET\Framework64\v2.0.50727>dir c:\websymbols\mscorwks.pdb
Directory of c:\websymbols\mscorwks.pdb
<![CDATA[09/14/2007 10:56 AM .]]>
<![CDATA[09/14/2007 10:56 AM ..]]>
<![CDATA[09/14/2007 10:57 AM E7B72F32EB6D45A5952A94564F939FD43]]>
0 File(s) 0 bytes
3 Dir(s) 78,916,845,568 bytes free
c:\Windows\Microsoft.NET\Framework64\v2.0.50727>dir c:\websymbols\mscorwks.pdb\E7B72F32EB6D45A5952A94564F939FD43
mscorwks.pdb
So now you will get mscorwks.pdb downloaded to “c:\websymbols\mscorwks.pdb\E7B72F32EB6D45A5952A94564F939FD43” directory. Copy this dll to CLR directory i.e c:\Windows\Microsoft.NET\Framework64\v2.0.50727>
Ok. Now you have installed windbg and PDB file which are necessary for using SOS. After that you have to follow these steps for setting the path etc
1) Start a new command window by running cmd.exe
2) type set path=C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;%path%
3) type set _NT_SYMBOL_PATH= C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727;
_NT_SYMBOL_PATH tells the debugger to search symbols in the directory or directories listed. Note that the above steps (2 and 3) can be done at a global level also.
Now you have setup everything and ready to use SOS. You can start windbg in couple of ways
<![CDATA[C:\debuggers\windbg.exe –p (this command will attach itself to already running application)]]>
C:\debuggers\windbg.exe workload.exe (if you want to debug workload.exe. This will launch this application)
C:\debuggers\windbg –pn “name of the process to attach to (this is same as 1st one but instead of giving PID, you give the process name).
bp<![CDATA[
bl = list all the breakpoints
<![CDATA[be ]]> = Enable the breakpoint # given
<![CDATA[bd ]]> = Disable breakpoint #
g = go continue the execution of the thread
k = Call stack for the current thread
kp = Call stack for the current thread with function parameter information
~ = Display all threads
~*kp = Display call stack of all the threads
sxe eh = Enable exception handing
sxd eh = disable exception handling. Debugger won’t stop when exception occurs
sxe dz = Enable divide by zero exception
sxd dz = Disable divide by zero exception. You can do this for other exceptions such as av etc.
<![CDATA[.sympath ]]> ex: .sympath C:\windows\symbols; = sets the symbol path inside the debugger
dv = display local variable information. For unmanaged code.
The chm file for windbg will have complete list of commands. So now you have all environment set and ready to start using SOS. Let me write in 2 examples how you would be using. 1 for web based applications (ASP.NET) and other one for console based applications (you have an exe. Ex: windows form applications).
Web Based Applications: You typically would want to use SOS on web server / app server where your code is running. The main process you want to look @ is W3Wp.exe. Follow these steps in order to attach debugger to this process and load SOS.
<![CDATA[1) c:\debuggers\windbg.exe –p . You will be prompted if you want to save the work space and you can decide, yes or no. ]]>
2) .loadby SOS mscorwks -> this is how you load SOS.dll. Note that it is important to match version of SOS.dll and mscorwks.dll. Both these will be in CLR directory
3) Type !Help -> If you see following screen, then SOS.dll is successfully loaded.
0:000> .loadby SOS mscorwks //loading SOS
0:000>! Help // List all the commands
——————————————————————————-
SOS is a debugger extension DLL designed to aid in the debugging of managed
programs. Functions are listed by category, then roughly in order of
importance. Shortcut names for popular functions are listed in parenthesis.
<![CDATA[Type "!help " for detailed info on that function. ]]>
Object Inspection Examining code and stacks
—————————– —————————–
DumpObj (do) Threads
DumpArray (da) CLRStack
DumpStackObjects (dso) IP2MD
DumpHeap U
DumpVC DumpStack
GCRoot EEStack
ObjSize GCInfo
FinalizeQueue EHInfo
PrintException (pe) COMState
TraverseHeap BPMD
Examining CLR data structures Diagnostic Utilities
—————————– —————————–
DumpDomain VerifyHeap
EEHeap DumpLog
Name2EE FindAppDomain
SyncBlk SaveModule
DumpMT GCHandles
DumpClass GCHandleLeaks
DumpMD VMMap
Token2EE VMStat
EEVersion ProcInfo
DumpModule StopOnException (soe)
ThreadPool MinidumpMode
DumpAssembly
DumpMethodSig Other
DumpRuntimeTypes —————————–
DumpSig FAQ
RCWCleanupList
DumpIL
For Console based applications (launching exe and not attaching): Note that all the usage of commands etc are exactly similar. But loading SOS.dll needs mscorwks.dll to be already loaded. If you are going to launch the exe under the debugger such as C:\debuggers\windbg.exe workload.exe then you will have to first make sure mscorwks.dll is loaded and so you have to do couple of extra steps. They are,
C:\debuggers\windbg.exe workload.exe and break in to the debugger. At the command, put a break point @ mscorwks!DllMain you can do that by,
0:000> bp mscorwks!DllMain
Bp expression 'mscorwks!DllMain' could not be resolved, adding deferred bp
<![CDATA[Type g to continue the execution of the program. You will hit the break point.]]>
0:000> g
ModLoad: 00800000 00817000 C:\Program Files\Common Files\Logishrd\LVMVFM\LVPrcInj.dll
ModLoad: 7e410000 7e4a1000 C:\WINDOWS\system32\USER32.dll
ModLoad: 77f10000 77f59000 C:\WINDOWS\system32\GDI32.dll
ModLoad: 76390000 763ad000 C:\WINDOWS\system32\IMM32.DLL
ModLoad: 77dd0000 77e6b000 C:\WINDOWS\system32\ADVAPI32.dll
ModLoad: 77e70000 77f02000 C:\WINDOWS\system32\RPCRT4.dll
ModLoad: 77fe0000 77ff1000 C:\WINDOWS\system32\Secur32.dll
ModLoad: 629c0000 629c9000 C:\WINDOWS\system32\LPK.DLL
ModLoad: 74d90000 74dfb000 C:\WINDOWS\system32\USP10.dll
ModLoad: 48000000 4801f000 C:\PROGRA~1\Google\GOOGLE~3\GOEC62~1.DLL
ModLoad: 71ab0000 71ac7000 C:\WINDOWS\system32\WS2_32.dll
ModLoad: 77c10000 77c68000 C:\WINDOWS\system32\msvcrt.dll
ModLoad: 71aa0000 71aa8000 C:\WINDOWS\system32\WS2HELP.dll
ModLoad: 77f60000 77fd6000 C:\WINDOWS\system32\SHLWAPI.dll
ModLoad: 77c10000 77c68000 C:\WINDOWS\system32\msvcrt.dll
ModLoad: 79e70000 7a3d6000 C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\mscorwks.dll
ModLoad: 78130000 781cb000 C:\WINDOWS\WinSxS\x86_Microsoft.VC80.CRT_1fc8b3b9a1e18e3b_8.0.50727.762_x-ww_6b128700\MSVCR80.dll
Breakpoint 0 hit
eax=00000001 ebx=79e70000 ecx=7a390dd4 edx=00000000 esi=00000001 edi=00000000
eip=79e9ee51 esp=0012f158 ebp=0012f198 iopl=0 nv up ei pl nz na po nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000202
mscorwks!DllMain:
79e9ee51 55 push ebp
Now that mscorwks.dll is loaded, you can load SOS using the same command as explained above. You can then disable break point using command bd 0 . Then all the commands that you use will exactly be the same.
Note that if you type !Help <command> you will get lot more information in windbg. Now, I am going to give you an example, how you can get lock contention information from SOS and how you want to correlate this information using thread call stacks etc. I have taken some sample program for this. I have 2 threads (1 producer thread and another consumer thread). Both these threads have shared variable. 1 thread is trying to increment while other is trying to call decrement thus creating contention on this. Now, I will tell you how you can get more information on this using the commands. Note that this is not a web based application but a console application. But the commands and procedure to get this information is exactly the same for web based or console based .Net applications
0:005> !Help SyncBlk
——————————————————————————-
<![CDATA[!SyncBlk [-all | ]]]>
A SyncBlock is a holder for extra information that doesn't need to be created
for every object. It can hold COM Interop data, HashCodes, and locking
<![CDATA[information for thread-safe operations. ]]>
SyncBlk is a very interesting and important command by which you can detect “dead lock” situations or lock contention issues. You need to look @ out put of SyncBlk along with call stack (~*kp) command. For ex:
0:005> !SyncBlk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
5 001a00bc 3 1 0019da80 1f5c 4 00b259b4 SharpSync2.LockTest
—————————–
What this is telling that thread a80 owns the syncblock 00b259b4 which is SharpSync2.LockTest object. This is good information for you to look @ as the hot lock. You can also then type !Threads and you get following output
0:005> !Threads
ThreadCount: 6
UnstartedThread: 0
BackgroundThread: 1
PendingThread: 0
DeadThread: 2
Hosted Runtime: no
PreEmptive GC Alloc Lock
ID OSID ThreadOBJ State GC Context Domain Count APT Exception
0 1 1df8 00181758 200a020 Enabled 00000000:00000000 0014c8d0 0 MTA
2 2 1db8 00182198 b220 Enabled 00000000:00000000 0014c8d0 0 MTA (Finalizer)
XXXX 3 0 0019c228 9820 Enabled 00000000:00000000 0014c8d0 0 MTA
XXXX 4 0 0019ca38 9820 Enabled 00000000:00000000 0014c8d0 0 MTA
3 5 1d8c 0019d248 200b020 Enabled 00000000:00000000 0014c8d0 0 MTA
4 6 1f5c 0019da80 200b020 Enabled 00000000:00000000 0014c8d0 1 MTA
As you can see from !Threads that thread 0019da80 is holding a lock (Lock count column). And in !SyncBlk you will which lock (object or syncblock) it is holding. Cool, now you know for your application, which is the hot lock and it is very good information to have. Now you will have to see which thread is waiting for this lock. So what you can do is to use “call stack” command and list call stack of all the threads that are running. That you can do by, ~*kp command
0:005> ~*kp
//This is the main thread which calls run thread function and waiting for that to finish.
0 Id: 1f98.1df8 Suspend: 1 Teb: 7ffdf000 Unfrozen
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
0012f164 79f46ef1 ntdll!KiFastSystemCallRet
0012f1cc 79f46e7d mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
0012f1ec 79f46ddb mscorwks!Thread::DoAppropriateAptStateWait+0x3c
0012f270 79f46c7b mscorwks!Thread::DoAppropriateWaitWorker+0×144
0012f2c0 79f2c570 mscorwks!Thread::DoAppropriateWait+0×40
0012f310 7a079a6c mscorwks!Thread::JoinEx+0×86
0012f364 7a07a0d7 mscorwks!ThreadNative::DoJoin+0xb1
0012f40c 007f0d44 mscorwks!ThreadNative::Join+0xc3
*** WARNING: Unable to verify checksum for SyncApp2.exe
<![CDATA[0012f43c 007f02e2 SyncApp2!CSharpSync2.Sync2.RunWorkload()+0x114 [c:\Delete\LockTest\Sync2.cs @ 75]]]>
<![CDATA[0012f480 79e7be1b SyncApp2!CSharpSync2.SyncApp2.Main()+0x21a]]> [c:\Delete\LockTest\SyncApp2.cs @ 158]
0012f490 79e7bd9b mscorwks!CallDescrWorker+0×33
0012f510 79e7bce8 mscorwks!CallDescrWorkerWithHandler+0xa3
0012f650 79e7bbd0 mscorwks!MethodDesc::CallDescr+0x19c
0012f668 79e802f4 mscorwks!MethodDesc::CallTargetWorker+0×20
0012f67c 79edb56e mscorwks!MethodDescCallSite::CallWithValueTypes_RetArgSlot+0×18
0012f7e0 79edb367 mscorwks!ClassLoader::RunMain+0×220
0012fa48 79edb23c mscorwks!Assembly::ExecuteMainMethod+0xa6
0012ff18 79edae8b mscorwks!SystemDomain::ExecuteMainMethod+0×398
0012ff68 79edadf3 mscorwks!ExecuteEXE+0×59
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\WINDOWS\system32\mscoree.dll –
0012ffb0 79004044 mscorwks!_CorExeMain+0x11b
1 Id: 1f98.cd8 Suspend: 1 Teb: 7ffde000 Unfrozen
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
007efed8 7c80a105 ntdll!KiFastSystemCallRet
007efef4 79f0cf96 KERNEL32!WaitForMultipleObjects+0×18
007eff54 79f0cef3 mscorwks!DebuggerRCThread::MainLoop+0xcf
007eff84 79f0ce36 mscorwks!DebuggerRCThread::ThreadProc+0xca
007effb4 7c80b713 mscorwks!DebuggerRCThread::ThreadProcStatic+0×82
007effec 00000000 KERNEL32!GetModuleFileNameA+0x1b4
2 Id: 1f98.1db8 Suspend: 1 Teb: 7ffdd000 Unfrozen
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
02cbfd78 7c80a105 ntdll!KiFastSystemCallRet
02cbfd94 79f12341 KERNEL32!WaitForMultipleObjects+0×18
02cbfdb4 79ee8d96 mscorwks!WKS::WaitForFinalizerEvent+0x7a
02cbfdc8 79ed8a2c mscorwks!WKS::GCHeap::FinalizerThreadWorker+0×75
02cbfdd8 79ed89ca mscorwks!Thread::UserResumeThread+0xfb
02cbfe6c 79ed88f1 mscorwks!Thread::DoADCallBack+0×355
02cbfea8 79ed988c mscorwks!Thread::DoADCallBack+0×541
02cbfed0 79ed9857 mscorwks!ManagedThreadBase_NoADTransition+0×32
02cbfedc 79ed96b5 mscorwks!ManagedThreadBase::FinalizerBase+0xb
02cbff14 79ed8e36 mscorwks!WKS::GCHeap::FinalizerThreadStart+0xbb
02cbffb4 7c80b713 mscorwks!Thread::intermediateThreadProc+0×49
02cbffec 00000000 KERNEL32!GetModuleFileNameA+0x1b4
//This is producer thread. As you can see from the call stack that it is waiting for some //resource. JITutil_MonContention and Awarelock.Enter are 2 key function that you can take a //look
3 Id: 1f98.1d8c Suspend: 1 Teb: 7ffdc000 Unfrozen
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
02eaf404 79f46ef1 ntdll!KiFastSystemCallRet
02eaf46c 79f46e7d mscorwks!WaitForMultipleObjectsEx_SO_TOLERANT+0x6f
02eaf48c 79f46ddb mscorwks!Thread::DoAppropriateAptStateWait+0x3c
02eaf510 79f46c7b mscorwks!Thread::DoAppropriateWaitWorker+0×144
02eaf560 79f46c23 mscorwks!Thread::DoAppropriateWait+0×40
02eaf5bc 79e7187c mscorwks!CLREvent::WaitEx+0xf7
02eaf5cc 7a100b32 mscorwks!CLREvent::Wait+0×17
02eaf658 7a100d2f mscorwks!AwareLock::EnterEpilog+0×94
02eaf674 7a100e9a mscorwks!AwareLock::Enter+0×61
02eaf6d8 7a090276 mscorwks!AwareLock::Contention+0x16c
02eaf778 007f0ec8 mscorwks!JITutil_MonContention+0xa3
*** WARNING: Unable to verify checksum for C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\mscorlib\45edd9c5830503e45cb2f01176434a19\mscorlib.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for C:\WINDOWS\assembly\NativeImages_v2.0.50727_32\mscorlib\45edd9c5830503e45cb2f01176434a19\mscorlib.ni.dll
02eaf7ac 793d70fb SyncApp2!CSharpSync2.LockTest.Producer(<HRESULT 0×80004001>)+0×40 [c:\Delete\LockTest\LockTest.cs @ 73]
<![CDATA[00b25a9c 793608fd mscorlib_ni!System.Threading.ThreadHelper.ThreadStart_Context()+0x3b]]>
<![CDATA[00b25a9c 793d71dc mscorlib_ni!System.Threading.ExecutionContext.Run()+0x81]]>
<![CDATA[02eaf7e0 79e7be1b mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x40]]>
02eaf7f0 79e7bd9b mscorwks!CallDescrWorker+0×33
02eaf870 79e7bce8 mscorwks!CallDescrWorkerWithHandler+0xa3
02eaf9ac 79e7bbd0 mscorwks!MethodDesc::CallDescr+0x19c
02eaf9c4 79e802f4 mscorwks!MethodDesc::CallTargetWorker+0×20
02eaf9d8 7a0786cb mscorwks!MethodDescCallSite::CallWithValueTypes_RetArgSlot+0×18
//This is consumer thread. If you look @ blue color text which is the thread id //(1f5c), which matches with !Threads command above which is holding the lock. So //we know that consumer thread has lock which producer thread is waiting for //creating contention. This should be enough for developers to fix the performance //issue.
4 Id: 1f98.1f5c Suspend: 1 Teb: 7ffdb000 Unfrozen
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
02faf5cc 79f56e05 ntdll!KiFastSystemCallRet
02faf600 7a04cd47 mscorwks!EESleepEx+0xa3
02faf60c 7a12f1c1 mscorwks!CExecutionEngine::ClrSleepEx+0xd
02faf61c 7a0e44c1 mscorwks!ClrSleepEx+0×13
02faf658 7a079218 mscorwks!Thread::UserSleep+0×63
02faf6f8 007f0f85 mscorwks!ThreadNative::Sleep+0xce
<![CDATA[02faf72c 793d70fb SyncApp2!CSharpSync2.LockTest.Consumer()+0x4d [c:\Delete\LockTest\LockTest.cs @ 110]]]>
00b25b68 793608fd mscorlib_ni!System.Threading.ThreadHelper.ThreadStart_Context(<HRESULT 0×80004001>)+0x3b
<![CDATA[00b25b68 793d71dc mscorlib_ni!System.Threading.ExecutionContext.Run()+0x81]]>
<![CDATA[02faf760 79e7be1b mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+0x40]]>
02faf770 79e7bd9b mscorwks!CallDescrWorker+0×33
02faf7f0 79e7bce8 mscorwks!CallDescrWorkerWithHandler+0xa3
02faf92c 79e7bbd0 mscorwks!MethodDesc::CallDescr+0x19c
02faf944 79e802f4 mscorwks!MethodDesc::CallTargetWorker+0×20
02faf958 7a0786cb mscorwks!MethodDescCallSite::CallWithValueTypes_RetArgSlot+0×18
02fafb5c 79ed8a2c mscorwks!ThreadNative::KickOffThread_Worker+0x15d
02fafb6c 79ed89ca mscorwks!Thread::UserResumeThread+0xfb
02fafc00 79ed88f1 mscorwks!Thread::DoADCallBack+0×355
02fafc3c 7a0e4d96 mscorwks!Thread::DoADCallBack+0×541
02fafc64 7a0e4dad mscorwks!Thread::DoADCallBack+0×575
# 5 Id: 1f98.1ff8 Suspend: 1 Teb: 7ffda000 Unfrozen
ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be wrong.
02fbfff4 00000000 ntdll!DbgBreakPoint
So with help of !Threads, !SyncBlk and call stack, we can come to know about lock contention issues. Note that you will also be able to come to know dead lock issues. Please to !Help SyncBlk in the debugger and you should be able to see the example given.
As I mentioned, SOS is a very powerful utility. Another frequent problems we have are GC related (OutOfMemoryException etc). You can also look into how many objects you have allocated and what is the approximate size of those objects. When ever I did this with my customers, almost always they were surprised. For ex: one of the ISV had ~21K instances of objects which are actually Array Lists. One might wonder, why so many. So you can use following command
0:005> !DumpHeap -stat
total 4435 objects
Statistics:
MT Count TotalSize Class Name
79129fd4 1 12 System.Collections.Generic.GenericEqualityComparer`1[[System.String, mscorlib]]
7910f948 1 12 System.Resources.FastResourceComparer
79106da8 1 12 System.RuntimeTypeHandle
79101be4 1 12 System.__Filters
79101b94 1 12 System.Reflection.Missing
<![CDATA[]]>
So for my example as you can see, you have 4435 objects (different). This command will give you count (how many instances) and what is the total size. I have only put partial results here. This information is on ascending order. So you can look at your top 5 or 6 object with huge size and start looking into how you can minimize.
Another interesting command is !EEHeap. Ex:
0:005> !EEHeap -GC
Number of GC Heaps: 1
generation 0 starts at 0x00b21018
generation 1 starts at 0x00b2100c
generation 2 starts at 0x00b21000
ephemeral segment allocation context: (0x00b38884, 0x00b39ff4)
segment begin allocated size
0018fa18 790d5588 790f4b38 0x0001f5b0(128432)
00b20000 00b21000 00b39ff4 0x00018ff4(102388)
Large object heap starts at 0x01b21000
segment begin allocated size
01b20000 01b21000 01b23250 0×00002250(8784)
Total Size 0x3a7f4(239604)
——————————
GC Heap Size 0x3a7f4(239604)
<![CDATA[This command tells you what is the GC generation heap size, LOH size etc. From where the gen0 starts and ends etc. So you can call !DumpHeap to dump all objects within that address range. How cool is that?]]>
For ex:
segment begin allocated size
0018fa18 790d5588 790f4b38 0x0001f5b0(128432)
Start address – 790d5588
Allocated till – 790f4b38 and the size is 128432.
You can even put break point and look at the assembly code generated. Since these are all jitted functions, you have to follow some steps to get to the virtual address where the code is stored. For ex: Let us say I want to put the break point in Consumer function and see the assembly code generated. This is what I will do.
0:005> !Name2EE SyncApp2.exe CSharpSync2.LockTest.Consumer
Module: 002b2c24 (SyncApp2.exe)
Token: 0x0600000c
MethodDesc: 002b3608
Name: CSharpSync2.LockTest.Consumer()
JITTED Code Address: 007f0f38
!Name2EE takes the exe name and the complete name of the function. This will let you know the virtual address where the code is located. Once you know the address you can use !U command to list the assembly.
0:005> !U 007f0f38
Normal JIT generated code
CSharpSync2.LockTest.Consumer()
Begin 007f0f38, size 9c
>>> 007f0f38 55 push ebp
007f0f39 8bec mov ebp,esp
007f0f3b 57 push edi
007f0f3c 56 push esi
007f0f3d 53 push ebx
007f0f3e 83ec20 sub esp,20h
007f0f41 33c0 xor eax,eax
007f0f43 8945d4 mov dword ptr [ebp-2Ch],eax
<![CDATA[]]>
You can also put breakpoint in this function using command (bp 007f0f38).
That is it guys. For some developers who have not windbg before it might be little overwhelming but trust me it will be very helpful. For ex: Many times, I work with ISV’s for analyzing performance without access to source code. So SOS is a very good way to get much information of the application, object, lock that you can tell the developers so they can work on to improve performance.
Let me know what you felt about this article. Always appreciate your feedback.
Thanks,
IBM Black Hole To Swallow Sun
by admin on Mar.18, 2009, under Storage
IBM Black Hole To Swallow Sun
 |
IBM Black Hole To Swallow Sun
IBM, the biggest monopoly holder in the technology business, half of who’s revenue comes straight from their competition free Mainframe business, is interested in buying Sun Microsystems. A company who’s mantra since the dot com crash has been about disrupting… |
|
A company who’s mantra since the dot com crash has been about disrupting the status quo of company’s such as IBM. Insert culture car crash here. Before I begin I have my suspicions about the timing of the WSJ leak. Those usually happen when the acquisition target is looking to fan the flames of a bidding war when another less interested party has come sniffing around. I already think it’s overpriced at the reported $6.5B and I’d guess that price will be higher by the time they close. The question is what would Sun bring to IBM’s table? The key assets I’d see for IBM in a Sun acquisition are: -The largest piece of the shrinking tape pie. This would turn the tape market into a two player game. Quantum on one side IBM/STK on the other. Tape is incredibly important to IBM, even used as primary storage by it’s Mainframe customers, but as a technology while it has a present tape does not have a future. It won’t die as a technology until five seconds after the Mainframe does but it’s an innovation dead end. -Control of Java. Something which does a couple of hundred million in revenue annually. For all their talk about opening Java we’ll see how that shoe fits when it’s an IBM asset. -A hammer lock on the server business. Combined IBM/Sun server share exceeds 40% and Sun’s server business is their fastest growing and probably most interesting division. -Control of MySQL The Oracle/DB2 war still rages so MySQL could be an interesting IBM attack point in that fight to that fight. -The Constellation switch family Cisco has parked their tanks on the server makers lawn. Sun have been making rumblings about going after the high margin switch business with commodity components. HP has ProCurve, IBM has…? Post Sun acquisition they’d have the upcoming Constellation family of switches. -It’s all about EDS HP’s aquisition of EDS puts IBM on the back foot. A reversal here would be to go around EDS entirely, one way of doing that is to leverage the fact that many large EDS run shops are Sun customers. Buy Sun, put yourself into that relationship with the customer and HP have a hard time locking you out. The key liabilities I’d see in such an acquisition are: -Solaris IBM’s cash rich pSeries division is an AIX stronghold with IBM’s numerous attempts to kill AIX and replace it with Linux having all failed. With Linux being pushed hard on xSeries hardware, on the Mainframe and even on pSeries where does Solaris settle? The answer is that it doesn’t. This would be Tru64 meets HP-UX all over again with noises made about integration but eventually the acquired UNIX going away. There will be massive uproar for the fanatical Solaris admin base (Of which I was a member) and as a sop to them eventually some crusty code would probably be thrown over the wall without Sun’s community source strings. -Sun Storage Sun’s server division had ambitions to chase the storage Dollar. Would an injection of Sun DNA into IBM awaken the same imperative in IBM’s xSeries division? If it did that would be bad news for IBM storage partners. On the other hand we can see the NetApp/Sun suit settling within moments of the ink drying. I don’t see IBM copping to any liability or paying NetApp any money. They might even have a patent cross licensing agreement in place. That doesn’t change the fact the Sun guys will still despise NetApp with the burning fury of a thousand stars. I also don’t worry about ZFS going away, maybe the AIX team might want it maybe they’d see it as too much effort to decouple it from Solaris but regardless it was always going to be worm food in the face of btrfs. Sun legal could never get to a place where widespread adoption was going to be easy. Sun’s storage division for all the hype is a blip in overall Sun revenues but it wouldn’t be if IBM decided they now had the technology to do it better than anyone they were rebranding gear from. As for SPARC, hand it over to Fujitsu. They built faster SPARC processors with better yields than Sun ever did and with POWER in their pocket I can’t see IBM being interested in anything besides the SPARC IP. Sun have a lot of talented people and EMC is hiring. If the whole IBM thing doesn’t work out, come and see us. I’ll be back later with more on this, today is a busy busy work day. |
|
| Update your feed preferences | |
Part 1
by admin on Mar.14, 2009, under Storage
EMC 2009 Strategic Forum — Paul Maritz and VMware – Part 1
 |
EMC 2009 Strategic Forum — Paul Maritz and VMware – Part 1
We’re continuing a sequence of posts recapping key themes and ideas from this event (targeted to investment and industry analysts) on March 10th. Many important highlights, one of which was the sheer pleasure of listening to Paul Maritz, CEO of… |
|
Many important highlights, one of which was the sheer pleasure of listening to Paul Maritz, CEO of VMware, present his vision and strategy to this important audience.
Paul is one of those must-see industry speakers. If you ever have a chance to hear him stretch out on his favorite topics, make the effort to go see him.
And this time, he did not disappoint — even going farther than the material he shared at the recent VMworld in Cannes.
Opening Thoughts
Paul oriented everyone by replaying a slide from earlier in the session, pointing out that he — and VMware — were responsible for the “virtual infrastructure” part of EMC’s two-part strategy. He did a good job setting historical context, by sharing how he had started out working for Intel in the very early days when Intel was considering its first microprocessor. He recounted how no one really saw the potential for what the microprocessor could do, or how it could change computing at the time, but — over time — by investing in the technology and extending its applicability and ecosystem, we have the IT world we live in today. He felt very strongly that virtualization was in a similar early stage — people were just beginning to realize how powerful virtualization technology could potentially be, and how it could change many aspects of the technology world we know today. And that the journey had just begun, in his opinion.
The current VDC-OS initiative (virtual data center operating system) was targeted towards data centers that wanted better efficiency and control of both their application workloads and their server environments. The vClient initiative was targeted at what users saw: creating virtualized desktop and rich application experiences that could intelligently target any device the user chose, and exploit its native capabilities — again, about efficiency and control for the IT department, as well as new choices for users. And the recently announced vCloud initiative was aimed at service providers to provide compatible infrastructure choices that could be controlled by enterprise IT if needed. All very logical, all very sensible.
He told a story that the very first environment targeted by the VMware team was the simple use case of running a guested windows OS on a Linux workstation — nothing more. The client hypervisor evolved into the server hypervisor over time — and more people saw the potential of this technology. A major step forward occurred when multiple hypervisors started cooperating with not only each other, but external management and orchestration frameworks. The result was that virtualization quickly evolved from a per-server model to a pooled-resource model, with DRS (distributed resource scheduling) as an obvious example. Paul took the opportunity that this sort of software is extremely difficult to do well — multiple, independent entities, all communicating and cooperating together, and doing so in a predictable and efficient manner. He flatly stated that the competing virtualization technologies were still perfecting individual server hypervisors, and hadn’t made the leap into multiple cooperating hypervisors, which VMware had been successfully doing for quite some time. This all turned out to be merely a warmup for what was to come. The Cloud As An Architecture
He then presented a fascinating view of cloud-as-architecture which I’m stealing and adding to my decks going forward. Sorry, you can’t see how the slide built using this medium, but try to imagine how the slide built as I describe it. He started with creating an idealized “cloud” container — whether it be inside a data center, in a service provider’s facility, or both. At the bottom, he saw that these architectures would be built from industry-standard building blocks that we were all familiar with. He took pains to point out that these wouldn’t necessarily be the cheapest components, just the best components for the job at hand. Above that, software helped these industry-standard components deliver scale and availability (think ESX, for example). And, to exploit this scale and availability, existing applications have to be “wrapped” in a virtualization layer — they couldn’t use a cloud architecture natively without being rewritten. Above the software was a key insight — that virtualization created an obvious and compelling place in the stack to inject policies regarding security, compliance and other governance concerns. The architectural attractiveness of this sort of approach shouldn’t be lost on too many people. Going farther up the stack, there will be a need for management that monitors service delivery and re-orchestrates resources as needed. Finally, this architectural stack would provide idealized support for newer application development frameworks (e.g. Ruby on Rails, et. al.), alongside supporting legacy applications. And this leads to the key point of the slide: virtualization is the only way to make this happen in an evolutionary way — a point lost in some of the other cloud discussions I’ve been following.
Considering The vSphere Architecture Paul then shared a high-level conceptual view of vSphere oriented around two primary functions: aggregation of resources (vCompute, vStorage, vNetwork) and a layer that handles services and policy. He also stated a repeating theme — even though VMware products such as vSphere would have base capabilities, he thought that any VMware product should be extensible with value add from other companies that have specific expertise that VMware did not have. Hence the “plug in” architecture shown here. vCompute Paul stated the obvious — if the proposition was that all data center applications would have to be virtualized to participate in a cloud (whether internal or external), the virtualization layer should be prepared to support the largest and most demanding applications. And he presented some fairly compelling evidence that this would be the case sooner than later.
The vast majority can be comfortably virtualized today (as we know), but there’s a long tail that must be considered. As you can see, Paul clearly committed that the next version of VMware’s flagship product would have the “beef” to handle very large applications: up to 8 virtual CPUs in a single virtual machine, 256 GB of memory per virtual machine, 40 Gb/sec network bandwidth and driving over 200,000+ IOPs. That’s a lot of crunch for a single application instance, isn’t it? And that’s just in 2009.
He showed how performance linearly scaled, and then compared the results using virtual machines against a native machine. Now, of course there was a tax for using virtualization on this particular test (if I remember correctly, TPC-C workloads drive very small I/O transactions, which probably isn’t the friendliest workload for a hypervisor). He also stated that the overhead associated with all forms of virtualization was dropping rapidly with each successive release. And then he put it all in perspective — we were talking nose-bleed levels of performance here to start with.
That’s impressive, if you think about it — for a single virtual machine. Sure, there are some apps out there that need to drive more from a single application image, but not a whole lot. And that’s what can be done today — without considering faster hardware (coming this year) or further optimizations in the hypervisor. I think Paul was having fun, because he wanted to put up a slide that showed all the hardware required to drive this sort of workload today.
As Chad Sakac would point out, we’d love the chance to rerun this particular test on a single rack of enterprise flash drives, but I guess we’ll have to wait for that. Now, what kind of applications do you have in your environment that drive enough I/O to keep 500 very fast spindles extremely busy? I think he needed to make a point — VMware wasn’t just for small applications any more — and hadn’t been for a while. vNetwork
The example he used was network state, and the problems created when virtual machines move from server to server, and specific network state characteristics might be lost. VMware had created a basic capability to solve this problem (the vNetwork distributed switch), but had also created the capability to do an entirely software-based version of their Nexus switch (the 1000V) that ran entirely as a virtual machine. From a network administrator’s point of view, this meant that not only was the network simpler to configure and monitor, but the virtual switch behaved exactly the same as a hardware switch would, except without the hardware part, of course.
If you think about it, there’s really no reason anymore to demand dedicated hardware server appliances to do most of IT’s housekeeping these days. I wonder how many other vendors will see the light, and start moving in that direction? Paul also showed another related example using vStorage. He put up an EMC-related example of of how EMC used the vStorage APIs to do a better job of delivering virtual (thin) provisioning.
Again, this approach to pluggable architecture not only works well for aggregation functions (storage, network, compute) but the service and policy layer — an area that EMC continues to exploit with our RSA portfolio (security) as well as our Smarts resource management portfolio. The fact that VMware recognized this need for a standardized way for other vendors to extend VMware’s functionality — largely from day one — reflects fairly progressive thinking in the commercial operating system world, in my opinion. Putting It All Together
He routinely speaks of the power of virtualization to create a “giant computer” or a “software mainframe” that takes industry-standard components and platforms, and makes them do amazing things. One eye-opening chart was just how big a VMware-based “giant computer” could be this year. Take a look at the “giant computer” specifications on the left. His use of the term “software mainframe” might be doing the architecture a bit of a disservice The energy efficiency example he gave was equally eye-opening.
He showed an example of a VMware cluster running a complex “VMark” benchmark — one where lots of tasks ramp up, run for a while, and then ramp down — sort of designed to simulate what a workday might look like. Using DPM, it’s now possible to dynamically move workloads to fewer servers, and power down the CPUs that aren’t being used. When the workload expands, these CPUs are re-powered on, workloads moved over, and so on. At a high level, this sort of approach saved 50% of energy consumed — and that’s after any energy saving that was achieved from virtualization in the first place. And then he put up a stunner.
It was possible for one virtual machine to emit an instruction-level trace that could be used to synchronize state on a second (or third, or fourth) virtual machine. If the first machine failed, the alternate machine would be in the exact same state as the first one, and could take over the workload without the need to reboot and reinitialize. Now, having spend time with true fault-tolerant architectures (Stratus, et. al.) the idea of doing this particular trick entirely in software was a tantalizing prospect, to say the least. And the ability to specify a given application as HA (high availability with reboot), FT (instruction and state synchronized for immediate failover), or even FT+ (always have a hot synchronized virtual machine, even after the first one had failed) — and to do so on virtually any legacy application, and using industry-standard components — well, that was certainly something to consider. Now, who wouldn’t want this in their private cloud? My head was starting to hurt by this time, but there was more to come … Management
He felt that customers expected a platform vendor to be the best at managing their platform, and couldn’t entirely leave it up to third parties. And VMware was no exception. Where he thought VMware had the opportunity to do better than the rest of the industry was to establish core capabilities that spoke in terms that the business wanted to see — performance, availability, security, cost, etc. — and still provide an extensible platform for other vendors to participate. More To Come Even though I’m now several longish posts into covering this event, there’s still more to come. A lot of important things with important implications were shared publicly, and I just want to make sure that as many people as possible have the opportunity to appreciate some of this material, and hopefully influence some of their thinking going forward. I know it’s seriously influenced my thinking … |
|
| Update your feed preferences | |
