|
A power tool to understand memory layout |
|
Core analyzer |
|
The simplest form of Core Analyzer is a stand-alone program, which adopts an easy-to-use command line interface. It is fast because it doesn’t read and resolve debug symbols. You can use it along side with a debugger which takes advantage of both tools. User invokes the utility program just like a normal debugger. The following command loads core dump file core.1234 that is generated by executable foo into Core Analyzer. $core_analyzer foo core.1234 Once the core dump file is loaded, Core Analyzer lists the following menu which are currently supported features. [0] Print General Core Information [1] Find References to an Object (horizontal search) [2] What Is This Address and Underlying Object Type (vertical search) [3] Objects Shared Between Threads [4] Memory Pattern Analysis [5] Query Heap Memory Block [6] Page Walk (check the integrity of surrounding memory blocks) [7] Heap Walk (check the whole heap for corruption and memory usage) [8] Biggest heap memory blocks [9] Biggest Heap Memory Owners (variables) [10] Heap Memory Leak Candidates [11] Quit Core file informationThe first function prints general information in the core file headers. It helps you to get an overall view of the problem. For example, executable name and its depending shared libraries indicate the software involved and its version; The signal received explains the reason of the core generation: segmentation fault, illegal instruction, misaligned data access, etc.; A big number of file handles may suggest resource leaks; Abnormal high CPU usage by some threads could mean load imbalance; A large number of memory regions or its total size that nears a system’s limit (like vm.max_map_count or ulimit on Linux) reveals the potential problem. The tool also sends a warning if the core file is truncated. The following is a typical output. NOTE [0] name: CORE type: PRSTATUS pid=30784 signal=6 user_time=3.450475sec sys_time=0.46992sec %rip=0x325610c34b %rsp=0x7fbfffdc08 [1] name: CORE type: PRPSINFO process_state=R filename=MSTRSvr [/home/myan/MSTRSvr -w /home/myan] [2] name: CORE type: PRXREG [prxregset struct] [3] name: CORE type: AUXV type=16 hardware_capability=395049983 type=6 pagesize=4096 type=17 frequency_of_times()=100 type=3 &phdr[0]=0x400040 type=4 sizeof(phdr[0])=56 type=5 #phdr_entries=8 type=7 ld.so_base_addr=0x0 type=8 proc_flags=0x0 type=9 entry_point=0x406840 type=11 real_uid=547 type=12 effective_uid=547 type=13 real_gid=507 type=14 effective_gid=507 type=23 was_exec_setuid-like? NO type=15 plat_name=0x7fbfffee48 [4] name: CORE type: FPREGSET [fpregset struct] floating point register set . . . [507] name: CORE type: PRSTATUS pid=30809 signal=6 user_time=3.486469sec sys_time=0.276957sec %rip=0x3256108d2f %rsp=0x401ff8e0 [508] name: CORE type: FPREGSET [fpregset struct] floating point register set No. vaddr memsz filesz perm name ========================================================== [0] 0x400000 143360 0 R-X [ unknown ] [1] 0x522000 8192 8192 RW- [ unknown ] [2] 0x524000 27492352 27492352 RWX [ heap ] [3] 0x40000000 4096 4096 --- [ guard page ] [4] 0x40001000 2097152 2097152 RWX [ stack ][tid=30809] . . . [1057] 0x2ab9989000 266240 266240 RW- [ heap ]
Search references to given data objectA couple of search functions are provided to find references. Horizontal search takes a target object represented by the object’s starting address and its size. Core Analyzer scans every byte of the core file which means the complete process image including all thread contexts and their stacks, all modules’ global data sections, and the whole heap memory. It also takes into consideration of the fact that a valid reference could point to anywhere inside the object other than its starting address, i.e. referring to a slice of the object which is common in high-level language like the base class in C++. In the following listing, an object starting at 0xb523c0 with 280 bytes of size has five direct references (two reference to its subobject located at 0xb523e0): four heap objects (all in-use) and one global variable in a loaded module’s .data section. User may choose up to 32 levels of indirect references. Searching all references to object starting at 0xb523c0 size 280 ------------------------- Level 1 ------------------------- [heap] block 0xb523c0 (size=280, inuse) +32: 0xb523e0 [heap] block 0xb523c0 (size=280, inuse) +40: 0xb523e0 [heap] block 0xc748b0 (size=48, inuse) +40: 0xb523c0 [heap] block 0xc8ba20 (size=32, inuse) +24: 0xb523c0 module /home/myan/lib/libMJCntMgr.so [data] 0x2a96e49e00: 0xb523c0 |—> searched object [0xb523c0, b524d8) Another search function is vertical search, which finds the data type of a given heap object. Objects allocated on heap are dynamic and, therefore, lack the debug symbol for their types. The idea is to find at least one variable with debug symbol (global variable, local variable or passed parameter) that references the target object either directly or indirectly. We can then infer the target object’s data type through the ownership chain. If the direct references are all from heap objects as well, their references (second level references relative to the target object) are searched. The process may be repeated recursively until a known variable with debug symbol is found. Walking back to the target builds a chain of reference from the known typed object to the target. Hence the name vertical search compared with the horizontal search shown previously. The following example shows the search result for an arbitrary address 0xee02e0. Two reference chains are found. Let’s take a look at the first one. It starts with a global variable named gModuleMainPtr residing in module libMJPrfMon.so which is revealed by gdb command “info symbol”. The variable’s data member at offset 16, mpModuleMain, consists of a std::list<int> named mCounterList. The second node of the list is the searched target. Now we are clear what address 0xee02e0 is. The second search result is very similar to the first one except the top reference is a local variable on thread 9. ------------------------- 1 ------------------------- [.data] module /lib/libMJPrfMon.so 0x2a9d2b9b20: 0xd5e370 |--> heap block [0xd5e370, 0xd5e7b8) size=1096 0xd5e470: 0xee35a0 |--> heap block [0xee35a0, 0xee35c0) size=32 0xee35a0: 0xee02e0 |--> heap block [0xee02e0, 0xee0300) size=32 ------------------------- 2 ------------------------- [stack] thread 9 frame#0 rsp+272 0x4160a050: 0xd5e3a8 |--> heap block [0xd5e370, 0xd5e7b8) size=1096 0xd5e470: 0xee35a0 |--> heap block [0xee35a0, 0xee35c0) size=32 0xee35a0: 0xee02e0 |--> heap block [0xee02e0, 0xee0300) size=32
(gdb) info symbol 0x2a9d2b9b20 gModuleMainPtr + 16 in section .data (gdb) p gModuleMainPtr $1 = { ... mpModuleMain = 0xd5e370 } (gdb) p *gModuleMainPtr.mpModuleMain $2 = { ... mCounterList = { <std::_List_base<int*,std::allocator<int*> >> = { _M_impl = { <std::allocator<std::_List_node<int*> >> = { <__gnu_cxx::new_allocator<std::_List_node<int*> >> = {<No data fields>} }, members of std::_List_base<int*,std::allocator<int*> >::_List_impl: _M_node = { _M_next = 0xee35a0, _M_prev = 0x16b2c80 } } }, <No data fields>}, ... } (gdb) p *$2.mCounterList._M_impl._M_node._M_next $3 = { _M_next = 0xee02e0, _M_prev = 0xd5e470 }
Objects shared between threadsA thread is an independent execution unit that can be scheduled to a processor. Multi-threaded programs take advantage of multi-core hardware and process jobs in parallel. As a result, performance may be improved dramatically and the code logic is more streamlined. However, data sharing and synchronization in a multi-threaded program is notoriously complex, subtle and subject to error. For example, race condition is easily one of the most challenging and nasty bugs. This command is to reveal all shared objects that are currently referenced by any two or more thread contexts. A thread context includes the thread’s registers and stack memory. The listed objects will provide an unique view of how involved threads synchronize and share data. If race condition is suspected, a full list of candidates is ready for verification. ------------------------ 1 ------------------------ shared object: [heap block] 0xd460b0--0xd460e0 size=48 [stack] thread 68 rsp+832 @0x584c0e80: 0xd460b0 [stack] thread 69 rsp+800 @0x582bfe60: 0xd460b0 [stack] thread 69 rsp+744 @0x582bfe28: 0xd460b0 [stack] thread 69 rsp+656 @0x582bfdd0: 0xd460b0 ------------------------ 2 ------------------------ shared object: [heap block] 0xdba430--0xdbac38 size=2056 (_vptr=0x32553159a8) [stack] thread 57 rsp+256 @0x59acb7a0: 0xdba430 [stack] thread 58 rsp+256 @0x598ca7a0: 0xdba430 [stack] thread 255 rsp+544 @0x40e05e40: 0xdba430 [stack] thread 258 rsp+1616 @0x401fff30: 0xdba430 . . .
Data pattern for given memory rangeLooking at a corrupted object, we often scratch our heads and wonder how did it happen. The answer may be right in front of us. The culprit of memory corruption often leaves signature data behind it, for example, a recognizable string, a pointer to certain type of object, etc. Therefore, it is very helpful to inspect the data around the corrupted memory for any clue. Memory pattern analysis discovers the data content for known types. In the following example, the given memory belongs to a stack frame. We can see strings, pointer to strings, function return address (instruction address), pointers to heap objects, etc. and how they are laid out on the stack memory. 0x7fbfffe020: 0x0000003200000003 0x7fbfffe028: 0x3255209ec3 => module [/lib64/ld-linux-x86-64.so.2] .text 0x7fbfffe0c8: 0x0000007fbfffe640 => thread 1 [frame 0] 0x7fbfffe630: 0x0000000000774200 => heap block [0x774200 inuse 200 bytes] 0x7fbfffe980: 0x0000000000524538 => heap block [0x524520 inuse 104 bytes] (wchar*) => [status-iserver.xml] . . . Another usage of data pattern analysis is to help debugging optimized code. As we know, optimization makes it difficult for the debugger to print local variables and passed parameters. Depending on the implementation of a specific debugger, it sometimes prints stale values, wrong values, or refuses to print a variable due to optimization. However, a variable’s value must be stored somewhere. We could use Core Analyzer’s data pattern function to analyze the stack memory associated with a function. With the type and size information of the function’s local variables (which is not affected by optimization), it is easy to guess a local variable’s stack location.
Heap walk and memory corruption checkMost functions of core analyzer depend on heap data structures managed by runtime memory manager. A couple of heap-walking functions are useful to determine the integrity of heaps or the state of an individual memory block. For example, for a given address 0xc41140, the following output shows it points to the middle of an in-use memory block. This implies that the input address points to a subset or a slice of a data object of size 1864 bytes. If the memory block was in free status, any active reference to it may be suspicious. Block address ? 0xc41140 [Main Arena] In-use [Start Addr] 0xc41110 [Block Size] 1864 [Offset] +0x30 Instead of one single memory block, we can check the heap data structures within a region for the memory layout or potential memory corruption. The following output lists all memory blocks reside in a dynamic arena (terminology used by PTMalloc, the system’s memory manager on RedHat Linux). Dynamic arena (0x2ab5200020): [0x2ab52008c0 - 0x2ab52fa000] [0x2ab52008d0 - 0x2ab5200908] 56 bytes inuse [0x2ab5200910 - 0x2ab5200928] 24 bytes inuse [0x2ab5200930 - 0x2ab5200978] 72 bytes inuse [0x2ab5200980 - 0x2ab5200998] 24 bytes inuse . . . [0x2ab52dff10 - 0x2ab52dff48] 56 bytes inuse [0x2ab52dff50 - 0x2ab52fa000] 106672 bytes free Total inuse 8425 blocks 823672 bytes Total free 93 blocks 129944 bytes If any memory is corrupted, Core Analyzer reports the instance. For example, the following output of heap walk indicates the heap data (the size tag of the next memory block) at address 0x5010d0 is corrupted with an abnormal value of 0x50505050. We should check the preceding memory block for possible overrun. Main arena (0x3255634640): [0x501000 - 0x522000] [0x501010 - 0x501048] 56 bytes inuse [0x501050 - 0x501068] 24 bytes inuse [0x501070 - 0x5010b8] 72 bytes inuse [0x5010c0 - 0x5010d8] 24 bytes inuse The chunk at 0x5010d0 may be corrupted. Its size tag is 0x50505050 Heap walk not only checks the whole heap for potential memory corruption but also reports the memory usage statistics. The following output shows the memory manager’s tuning parameters and the current usage statistics with no error found. Tuning params & stats: mmap_threshold=131072 pagesize=4096 n_mmaps=153 mmapped_mem=110485504 sbrk_base=0x524000 Main arena (0x3255634640): [0x524000 - 0x1f5c000] Dynamic arena (0x2abbd00020) owns regions: [0x2abbd008c0 - 0x2abbd21000] Dynamic arena (0x2ab5200020) owns regions: [0x2ab52008c0 - 0x2ab52fa000] mmap block: [0x2aad2ee000 - 0x2ab12ef000] 65540KB mmap block: [0x2ab134f000 - 0x2ab144f000] 1024KB . . . Total inuse 134890608 bytes Total free 2851368 bytes
As a by-product of heap walk, we can easily find out the top memory blocks in terms of size. This may shed some light on how memory is used by a program, which is not so obvious sometimes. The following example shows the top nine biggest memory blocks and the references to them. Top 9 biggest in-use heap memory blocks: addr=0x2a9673f010 size=219488240 (209MB) addr=0x2aa3891010 size=48951280 (46MB) addr=0x2a95743010 size=16760816 (15MB) addr=0x2aaef98010 size=8392688 (8MB) addr=0x2aaed7b010 size=2215920 (2MB) addr=0x2aafc0a010 size=2117616 (2MB) addr=0x2a9557d010 size=1642480 (1MB) addr=0x2aa6740010 size=802800 (783KB) addr=0x2aa68a2010 size=401392 (391KB) Search references to these memory blocks... ----------------- [0] references to heap block 0x2a9673f010 ----------------- [heap block] 0xe75390--0xe75ce8 size=2392 @+152: 0x2a9673f010 [unknown] 0x2a9570e430: 0x2a9950b3b0 ----------------- [1] references to heap block 0x2aa3891010 ----------------- [heap block] 0xe75390--0xe75ce8 size=2392 @+112: 0x2aa4f75eb0 [heap block] 0xe75390--0xe75ce8 size=2392 @+120: 0x2aa5375eb0 [heap block] 0xe75390--0xe75ce8 size=2392 @+144: 0x2aa3891010 ----------------- [2] references to heap block 0x2a95743010 ----------------- [.data/.bss] /bin/mysqld @0xe40e20: 0x2a95743010 [.data/.bss] /bin/mysqld @0xe40e28: 0x2a957432b8 [heap block] 0xe9ea10--0xea2a18 size=16392 @+8: 0x2a957434b8 [stack] thread 28 rsp+7312 @0x41c8dd20: 0x2a957436b8 ... |
|
Core Analyzer As Stand-alone Program |
