Core Analyzer Integrated with Debuggers

A power tool to understand memory layout

Core analyzer

Core analyzer may be used as a stand-alone program which displays and analyzes each individual memory object and their interrelationship. However, it is sometimes difficult for a user to correlate this information with source program without proper symbols. Fortunately, some mainstream debuggers, like windbg and gdb, have either comprehensive APIs for extension or source opened, which makes it easy to integrate core analyzer with them and produce more useful results. The current implementation is tested on Windows/x64 and RedHat EL/x86_64 platforms. However, it won't be difficult to port to other platforms if necessary. If you choose to build by yourself, for example, for a custom memory manager or a different platform, please refer to the steps in README file.

Windbg is the most powerful debugger on Windows with rich functions and friendly user interface. Developers doing serious debugging on Windows are certainly familiar with it. What they might not know well is that the underlying debug engine supports an extensive programming interface. User may develop powerful features based on the exported APIs which includes essentially all debug symbols, data access, various events, etc. You can literally rewrite the entire Windbg with the support of the debug engine API.

Core analyzer may be compiled into a Windbg DLL extension. To use the feature, you will need to load the extension DLL ref.dll (included in the source bundle) into Windbg with “.load” command. There are other ways to load it automatically. Please see Windbg help for more information.

gdb, the GNU debugger, is well accepted in the developer community with its rich and powerful features across many platforms. Though it doesn’t support plug-in as Windbg does, gdb is open sourced which makes it easy to embed core analyzer within it. Based upon the formal releases of gdb, additional commands are added and compiled into binary. User may use the custom built gdb executable (also included in the source bundle) without any configuration steps, which assumes that heap memory is managed by ptmalloc.

The following paragraphs list the supported commands and their usage within the debuggers. Examples are taken from mysql and Intelligence Server. The former is an open source project and the later a proprietary product that the author has been working with. Both are heavy duty server programs with intense memory usage and high-level of concurrency.

———————————————————————————————————————————–———————

COMMAND

(windbg)

!heap [/verbose or /v] [/leak or /l]

!heap [/block or /b] [/cluster or /c] <addr_exp>

!heap [/usage or /u] <var_exp>

!heap [/topblock or /tb] [/topuser or /tu] <num>

(gdb)

heap [/verbose or /v] [/leak or /l]

heap [/block or /b] [/cluster or /c] <addr_exp>

heap [/usage or /u] <var_exp>

heap [/topblock or /tb] [/topuser or /tu] <num>

DESCRIPTION

Scan and check target process's heap for possible memory corruption. If there is no error, the command reports a summary of the heaps.

Option [/verbose] displays more detail while walking each heap, including memory histogram for both in-use and free memory.

Option [/leak] reports a list of memory blocks that are potentially leaked. The algorithm is based on the concept that a heap memory, which is not referenced by any local or global variable directly or indirectly, is not reachable by any code and therefore is leaked. The tool may report false positive if a module’s section is not recognized by debugger.

Option [/block] queries the memory block that consists of the input address. It shows the memory block's address range, its size, and whether it is free or in use.

Option [/cluster] displays a cluster of memory blocks surrounding the given address, in other words, the memory layout around the interested spot.

Option [/usage] calculates heap memory consumption by a variable or an object. If you wonder how your memory is consumed by a program, this command, and the following options [/topblock] and [/topuser], could shed some light on it. Unlike regular memory tools, the feature doesn’t need to setup and instrument the program and is, therefore, very convenient to use.

Option [/topblock] lists biggest heap memory blocks in terms of size. By traversing the whole heap memory, the command displays the top n memory blocks with biggest size. It also tries to reveal which other objects have references to these big memory blocks.

Option [/topuser] lists local or global variables that consume the most heap memory in terms of aggregated size, or the total heap memory reachable through a variable. This is equivalent to query every local and global variable with “heap /usage”, and find the top list.

Example 1: Scan the whole heap

0:093> !heap

heap 0x0x150000 ...

segment 0x150cc0 - 0x1fa000

uncommitted 0x1fa000 - 0x250000

heap 0x0x250000 ...

segment 0x250cc0 - 0x254000

uncommitted 0x254000 - 0x260000

heap 0x0x710000 ...

segment 0x710cc0 - 0x716000

uncommitted 0x716000 - 0x790000

heap 0x0x790000 ...

segment 0x790cc0 - 0x7a0000

segment 0x7a0070 - 0x8a0000

segment 0x2f20070 - 0x3120000

segment 0x9330070 - 0x9723000

uncommitted 0x9723000 - 0x9730000

...

heap 0x0xf090000 ...

segment 0xf090cc0 - 0xf095000

uncommitted 0xf095000 - 0xf110000

Total busy: 10698222 bytes

Total free: 1245078 bytes

(gdb) heap

Tuning params & stats:

mmap_threshold=131072

pagesize=4096

n_mmaps=168

mmapped_mem=113397760

sbrk_base=0x524000

Main arena (0x3255634640): [0x524000 - 0x2234000]

Dynamic arena (0x2ab3100020) owns regions:

[0x2ab8800020 - 0x2ab8821000]

[0x2ab8700020 - 0x2ab87fd000]

[0x2ab8600020 - 0x2ab86fd000]

[0x2ab8500020 - 0x2ab85fd000]

[0x2ab7b00020 - 0x2ab7c00000]

[0x2ab7a00020 - 0x2ab7b00000]

[0x2ab4b00020 - 0x2ab4bfd000]

[0x2ab31008c0 - 0x2ab31e1000]

There are 168 mmap blocks

Total inuse 138449632 bytes

Total free 11353640 bytes

Example 2: Display heaps with memory histogram. In this example, 97% of in-use memory blocks are between 64 to 128 bytes while big blocks, greater than 512KB, dominate memory usage, 95% in terns of bytes. The free memory blocks, on the other hand, are concentrated in the range of 32 to 64 bytes.

(gdb) heap /v

Tuning params & stats:

mmap_threshold=19927040

pagesize=4096

n_mmaps=15

mmapped_mem=300949504

sbrk_base=0x7bb0000

Main arena (0x3f085539e0) owns regions:

[0x7bb0010 - 0x871c000] Total 11MB in-use 64562(9MB) free 96(1MB)

Dynamic arena (0x2aaadc000020) owns regions:

[0x2aaadc0008c0 - 0x2aaadc021000] Total 129KB in-use 28(58KB) free 6(71KB)

...

========== In-use Memory Histogram ==========

Size-Range Count Total-Bytes

16 - 32 111(0%) 2KB(0%)

32 - 64 92(0%) 4KB(0%)

64 - 128 63304(97%) 7MB(2%)

128 - 256 141(0%) 26KB(0%)

256 - 512 563(0%) 219KB(0%)

512 - 1024 292(0%) 254KB(0%)

1024 - 2KB 155(0%) 234KB(0%)

2KB - 4KB 112(0%) 298KB(0%)

4KB - 8KB 103(0%) 647KB(0%)

8KB - 16KB 39(0%) 388KB(0%)

16KB - 32KB 42(0%) 714KB(0%)

32KB - 64KB 3(0%) 128KB(0%)

64KB - 128KB 3(0%) 304KB(0%)

128KB - 256KB 2(0%) 271KB(0%)

256KB - 512KB 6(0%) 1MB(0%)

512KB - 7(0%) 284MB(95%)

Total 64975 297MB

========== Free Memory Histogram ==========

Size-Range Count Total-Bytes

16 - 32 13(8%) 312(0%)

32 - 64 89(59%) 4KB(0%)

64 - 128 11(7%) 856(0%)

128 - 256 7(4%) 1KB(0%)

256 - 512 6(4%) 1KB(0%)

512 - 1024 1(0%) 712(0%)

1024 - 2KB 6(4%) 6KB(0%)

2KB - 4KB 1(0%) 3KB(0%)

4KB - 8KB 7(4%) 47KB(2%)

8KB - 16KB 3(2%) 47KB(2%)

16KB - 32KB 1(0%) 16KB(0%)

32KB - 64KB 4(2%) 200KB(10%)

512KB - 1(0%) 1MB(82%)

Total 150 1MB

Example 3: Report memory leaks base on reference. Two blocks are listed below as potential leaks. A close look at their contents may reveal clues as who originally own them.

(gdb) heap /leak

Potentially leaked heap memory blocks:

[1] addr=0x7bb39d0 size=200

[2] addr=0x7bb3bd0 size=56

Total 2 (256) leak candidates out of 64975 (297MB) in-use memory blocks

Example 4: Display heap memory layout of the segment containing an input address. In Windows, memory tagged with “HEAP ENTRY” is heap data while “USER SPACE” are memory blocks allocated to application.

0:093> !heap /c 0x790cc0

0x790cc0 - 0x790cd0 [struct HEAP_ENTRY]

0x790cd0 - 0x792cd0 size=8192 busy [USER SPACE]

0x792cd0 - 0x792ce0 [struct HEAP_ENTRY]

0x792ce0 - 0x796cd0 size=16368 busy [USER SPACE]

0x796cd0 - 0x796ce0 [struct HEAP_ENTRY]

0x796ce0 - 0x797cd0 size=4080 busy [USER SPACE]

0x797cd0 - 0x797ce0 [struct HEAP_ENTRY]

0x797ce0 - 0x797cf0 size=16 free [USER SPACE]

0x797cf0 - 0x797d00 [struct HEAP_ENTRY]

0x797d00 - 0x79bcf0 size=16368 busy [USER SPACE]

0x79bcf0 - 0x79bd00 [struct HEAP_ENTRY]

0x79bd00 - 0x79fcf0 size=16368 busy [USER SPACE]

0x79fcf0 - 0x79fd00 [struct HEAP_ENTRY]

0x79fd00 - 0x7a0010 size=784 free [USER SPACE]

Total inuse 61376 bytes

Total free: 800 bytes

(gdb) heap /c 0x2ab31008c0

Dynamic arena (0x2ab3100020): [0x2ab31008c0 - 0x2ab31e1000]

[0x2ab31008d0 - 0x2ab3100948] 120 bytes inuse

[0x2ab3100950 - 0x2ab31009c8] 120 bytes inuse

[0x2ab31009d0 - 0x2ab3100a48] 120 bytes inuse

[0x2ab3100a50 - 0x2ab3100ab8] 104 bytes inuse

[0x2ab3100ac0 - 0x2ab3100b28] 104 bytes inuse

[0x2ab3100b30 - 0x2ab3100b68] 56 bytes inuse

[0x2ab3100b70 - 0x2ab3100be8] 120 bytes inuse

[0x2ab3100bf0 - 0x2ab3100c48] 88 bytes inuse

. . .

[0x2ab31e0f70 - 0x2ab31e0fc8] 88 bytes inuse

[0x2ab31e0fd0 - 0x2ab31e1000] 48 bytes free

Total inuse 9361 blocks 844408 bytes

Total free 1 blocks 56 bytes

Example 5: Display memory block that includes address 0x932ad58

0:093> !heap /b 0x932ad58

In-use

[Start Addr] 0x932ad50

[Block Size] 64

[Offset] +8

Example 6: The following command lists the top eight heap memory blocks, which are referenced by various global variables, local variables, and heap objects.

(gdb) heap /tb 8

Top 8 biggest in-use heap memory blocks:

addr=0x2a9673f010 size=219488240 (209MB)

addr=0x2aa3891010 size=48951280 (46MB)

addr=0x2a95743010 size=16760816 (15MB)

addr=0x2aaef98010 size=8392688 (8MB)

addr=0x2aaed7b010 size=2215920 (2MB)

addr=0x2aafc0a010 size=2117616 (2MB)

addr=0x2a9557d010 size=1642480 (1MB)

addr=0x2aa6740010 size=802800 (783KB)

Example 7: Instead of top heap memory blocks, option “/topusage” lists the top global/local variables or heap memory objects that use the most heap memory, which is calculated by the sum of their referenced heap memory directly.

(gdb) heap /tu 8

[1] [heap block] 0x7bb67e0--0x7bb7138 size=2392

|--> 256MB (3 blocks)

[2] [heap block] 0x2aaaabab8010--0x2aaab8c0a000 size=219488240

|--> 209MB (1 blocks)

[3] [heap block] 0x2aaab8c0a010--0x2aaabbab9000 size=48951280

|--> 46MB (1 blocks)

[4] [stack] thread 18 frame 4 buffer @0x42f37e40

|--> 16MB (10 blocks)

[5] [.data/.bss] /usr/local/mysql/bin/mysqld query_cache @0xe40d60

|--> 16MB (3 blocks)

[6] [heap block] 0x7be0050--0x7be4058 size=16392

|--> 15MB (2 blocks)

[7] [heap block] 0x7be4060--0x7be8068 size=16392

|--> 15MB (2 blocks)

[8] [stack] thread 24 frame 14 rsp+3424 @0x403fb750: 0x2aaaaaabc538

|--> 15MB (1 blocks)

Example 8: If we are interested in a particular global, local variable or a heap memory object, “/usage” option will report the statistics in two ways. As shown below, the first number (290MB) includes all heap memory reachable by variable “buffer” through any level of indirection, while the second number (16MB) includes only those referenced directly by the variable.

(gdb) heap /u buffer

Heap memory consumed by [stack] buffer @0x42f37e40

All reachable:

|--> 290MB (64694 blocks)

Directly referenced:

|--> 16MB (10 blocks)

—————————————————————————————————————————–—————————

COMMAND

(windbg)

!ref <addr_exp>

!ref [/thread or /t] <addr_exp> <size> [level]

(gdb)

ref <addr_exp>

ref [/thread or /t] <addr_exp> <size> [level]

DESCRIPTION

Search the target process for references to the input address. If only one argument, an address, is provided, the command tries to find the memory object associated with the input address as well as the object’s type and symbol. It searches the target space to find a variable with known type, such as a global/local variable or a heap object with RTTI, which references the input address directly or indirectly. If such a variable is found, we could deduce the correct data type associated with the input address. If an address and/or its size are given, the “ref” command searches all references to the data object starting at address "address" and as long as "size" bytes. The references may be up to “level” of indirection which is one by default.

Option [/thread] limits the search to thread contexts only, i.e. threads’ stack memory and registers, which is much faster.

Example 1: Search references to the input address until its type may be deduced. Here input address belongs to a heap. The command finds a local variable that references this heap data object indirectly.

0:000> !ref 0x00000000`0035a0f0

Address 0x35a0f0 belongs to heap block [0x35a0f0, 0x35a108] size=24

------------------------- 1 -------------------------

[stack] thread 0 frame 0 alist (type="class list<int>" size=48)._Myhead @0x12fe88: 0x359ff0

|--> [heap block] 0x359ff0--0x35a008 size=24 (type="_Node" size=24)._Next @+0: 0x35a070

|--> [heap block] 0x35a070--0x35a088 size=24 (type="_Node" size=24)._Next @+0: 0x35a0f0

|--> [heap block] 0x35a0f0--0x35a108 size=24 (type="_Node" size=24)

Example 2: Search all references to the given object defined by starting address and size. The following example shows all references to the given object (starting at address 0x35a0f0 and ending at 0x35a108) up to 2 levels of indirection.

0:000> !ref 0x00000000`0035a0f0 24 2

------------------------- Level 2 -------------------------

[heap block] 0x35a1f0--0x35a208 size=24 @+8: 0x35a170

|--> [heap block] 0x35a170--0x35a188 size=24

[heap block] 0x35a0f0--0x35a108 size=24 @+8: 0x35a070

[heap block] 0x359ff0--0x35a008 size=24 @+0: 0x35a070

|--> [heap block] 0x35a070--0x35a088 size=24

------------------------- Level 1 -------------------------

[heap block] 0x35a170--0x35a188 size=24 @+8: 0x35a0f0

[heap block] 0x35a070--0x35a088 size=24 @+0: 0x35a0f0

|--> searched target [0x35a0f0, 0x35a114)

Example 3: Search references in thread contexts. The following example shows two references are found in thread 5 on frame 6 and 7 respectively.

0:092> !ref /t 0x00000000`08afa760

------------------------ 1 ------------------------

[stack] thread 5 frame 6 this (type="class ConsoleAdminTask *" size=8) @0x940fb50: 0x8afa760

------------------------ 2 ------------------------

[stack] thread 5 frame 7 lTask (type="class MSITask *" size=8) @0x940fb90: 0x8afa760

———————————————————————————————————————————————————

COMMAND

(windbg) !obj <expression>

(gdb) obj <expression>

DESCRIPTION

Given the C++ class type of the input expression, the “obj” command searches for all objects of the specified type in target process’s address space as well as references to them. The virtual table of the C++ class is used to find the instances of the class.

Example: In the following example, the current frame is a method of class MSICommandQTask. Command “!obj this” reveals that the target process has two objects of this type and they are referenced by two threads 37 and 24 respectively.

0:009> ?? this

class MSICommandQTask * 0x00000000`0823db80

+0x000 __VFN_table : 0x00000000`06eef5b0

+0x008 mTaskImp : 0x00000000`081d39e0 MSITaskImp

+0x010 mQ : 0x00000000`08225980 MSIBasicQ<IMSIDSSCommand,MSIDSSCommandInfo>

+0x018 mOwnerId : 6357106

+0x020 mpStartEvent

+0x030 mThreadPool : 0x00000000`08e001c0 MSIBasicThreadPool

0:009> !obj this

Searching objects of type="MSICommandQTask*" size=8 (vtable 0x6eef5b0--0x6eef5b1)

[heap block] 0x82325c0--0x8232600 size=64

[heap block] 0x8236b80--0x8236bc0 size=64

Searching references to above objects

[stack] thread 37 frame 6 this (type="class MSICommandQTask *") @0xd74f2a0: 0x82325c0

[stack] thread 24 frame 6 this (type="class MSICommandQTask *") @0xbd4f2a0: 0x8236b80

———————————————————————————————————————————————————

COMMAND

(windbg) !shrobj [thread id] [thread id] . . .

(gdb) shrobj [thread id] [thread id] . . .

DESCRIPTION

A thread is an independent execution unit that can be scheduled to a processor. Multi-threaded programs take advantage of multi-core hardware and process jobs in parallel. As a result, performance may be improved dramatically and the code logic is more streamlined. However, data sharing and synchronization in a multi-threaded program is notoriously complex, subtle and subject to error. For example, race condition is easily one of the most challenging and nasty bugs. This command is to reveal all shared objects that are currently referenced by the selected threads. If no thread id is given, the command would output all shared objects that are referenced by any two or more thread contexts. A thread context includes the thread’s registers and stack memory. The listed objects will provide an unique view of how involved threads synchronize and share data. If race condition is suspected, a full list of candidates is ready for verification.

Example: The following output shows that thread 88, 89 and 90 share a “MSynch::CriticalSectionImpl” object which is likely their synchronization object. Thread 49 and 50 share another object “MSIDSSCommandInfo” which is referenced by multiple local variables in a couple of function frames in each thread.

(gdb) shrobj

------------------------ 1 ------------------------

shared object: [heap block] 0x2405de0--0x2405e40 size=96 (type="M8Synch4_64!MSynch::CriticalSectionImpl" size=88)

[stack] thread 90 frame 5 @0x1475f140: 0x2405de8

[stack] thread 90 frame 5 l_listLock.mrCriticalSection(type="MSynch::CriticalSection*") @0x1475f118: 0x2405de8

[stack] thread 89 frame 5 @0x1455f140: 0x2405de8

[stack] thread 89 frame 5 l_listLock.mrCriticalSection(type="MSynch::CriticalSection*") @0x1455f118: 0x2405de8

[stack] thread 88 frame 5 @0x1435f140: 0x2405de8

[stack] thread 88 frame 5 l_listLock.mrCriticalSection(type="MSynch::CriticalSection*") @0x1435f118: 0x2405de8

------------------------ 2 ------------------------

shared object: [heap block] 0x8225d80--0x8225e00 size=128 (type="MSIDSSCommandInfo" size=128)

[stack] thread 50 frame 8 lQ(type="MSIDSSCommandInfo*") @0xf14f320: 0x8225d80

[stack] thread 50 frame 6 iSubQ(type="MSIDSSCommandInfo*") @0xf14f2b8: 0x8225d80

[stack] thread 50 frame 6 lBasicQ(type="MSIDSSCommandInfo*") @0xf14f200: 0x8225d80

[stack] thread 50 frame 5 this(type="MSIDSSCommandInfo*") @0xf14f1e0: 0x8225d80

[register] thread 50 r9d=0x8225d80

[register] thread 50 r9=0x8225d80

[stack] thread 49 frame 8 lQ(type="MSIDSSCommandInfo*") @0xef4f320: 0x8225d80

[stack] thread 49 frame 6 iSubQ(type="MSIDSSCommandInfo*") @0xef4f2b8: 0x8225d80

[stack] thread 49 frame 6 lBasicQ(type="MSIDSSCommandInfo*") @0xef4f200: 0x8225d80

[stack] thread 49 frame 5 this(type="MSIDSSCommandInfo*") @0xef4f1e0: 0x8225d80

. . .

—————————————————————————————————————————–—————————

COMMAND

(windbg) !decode [/v] [frame=n1-n2] [from=address] [to=address] [reg=value] […]

(gdb) decode [/v] [frame=n1-n2] [from=address] [to=address] [%reg=value] […]

DESCRIPTION

Display disassembled instructions annotated with data object context. Assembly code is difficult to read for most developers working with high-level source code. But it is necessary sometimes to debug tough issues. With this command, the user may understand what happens at machine instruction level with ease even for highly optimized code. The object context links the obscure registers and transient stack memory with source-level variables and symbols.

By default, the command disassembles the currently selected frame starting with the beginning of the function and ends at the instruction currently being executed. It updates the register values (debug context) according to the instruction executed. The stack pointer and program counter are always known at the function entry. It also assumes the register values through which function parameters are passed in as well as some registers preserved across function calls. User may set the initial value of any register through the command option [%reg=value], where reg is the name of a register, e.g. rdi. User may also choose the starting and/or the ending addresses to disassemble through option [from=address] and [to=address] as long as the input address is valid to the selected function. Multiple functions may be disassembled together through [frame=n1-n2] option. The command will use the updated register values of the caller as the initial context of the called function. option [/v] turns on verbose mode to reveal more detail of the function context.

This command uses initial register context and stack memory to show how data objects are propagated as the function is executed. But it has inherent limitation since it tries to show history deduced from current state. For example, a loop variable may be incremented to a value that fails the loop condition. The command reads its value from the stack memory and may display its final value at the beginning of the loop. Another example is jump instruction. If the program takes a particular jump path, this command couldn’t detect it and might follow the wrong execution path and therefore get false result. However, the user should be able to tell the false value with small effort.

Example The following lists the instruction of function mysql_parse. Annotation is added after the comment sign “##”. The function has four parameters among which three are known at function entry and one is optimized out. As a matter of fact, the user can find out the value of the unknown parameter (length) by disassembling the caller function. These parameters are passed in through registers rdi, rsi, edx and rcx respectively. After the function prologue, two functions are called, lex_start and mysql_reset_thd_for_next_command, where parameter thd with value 0x2aaf95e840 is passed through register rbp. The user may find more detail information through the annotation.

(gdb) decode

Parameters: thd(%rdi)=0x2aaf95e840, rawbuf(%rsi)=0x17c4c50, length(%edx)=<optimized out>, parser_state(%rcx)=0x41fdbe30

The following registers are assumed at the beginning: rcx=0x41fdbe30, rbx=0x2aaf95e8a0, rsp=0x41fdaef8, rbp=0x2aaf95e840, rsi=0x17c4c50, rdi=0x2aaf95e840, r12=0x41fdbf08, r13=0x2aaf95ea30, r14=0x41fdbe30, r15=0x2aaf961f21, rip=0x56f5e0

Saved registers: rbx(0x2aaf95e8a0), rbp(0x2aaf95e840), r12(0x41fdbf08), r13(0x2aaf95ea30), rip(0x570590)

Dump of assembler code for function mysql_parse(THD*, char*, unsigned int, Parser_state*):

0x000000000056f5e0 <+0>: mov %rbx,-0x20(%rsp) ## [%rsp-0x20]=0x2aaf95e8a0 [heap block] 0x2aaf95e840--0x2aaf9615d8 size=11672 thd::Statement.query_string (type="struct CSET_STRING") @+96

0x000000000056f5e5 <+5>: mov %rbp,-0x18(%rsp) ## [%rsp-0x18]=0x2aaf95e840 [heap block] 0x2aaf95e840--0x2aaf9615d8 size=11672 thd

0x000000000056f5ea <+10>: mov %rdi,%rbp ## %rbp=0x2aaf95e840 [heap block] 0x2aaf95e840--0x2aaf9615d8 size=11672 thd

0x000000000056f5ed <+13>: mov %r12,-0x10(%rsp) ## [%rsp-0x10]=0x41fdbf08 [stack] frame 14 rsp+4104 @0x41fdbf08

0x000000000056f5f2 <+18>: mov %r13,-0x8(%rsp) ## [%rsp-0x8]=0x2aaf95ea30 [heap block] 0x2aaf95e840--0x2aaf9615d8 size=11672 thd.net (type="struct st_net") @+496

0x000000000056f5f7 <+23>: sub $0x48,%rsp ## %rsp=0x41fdaeb0 End of function prologue

0x000000000056f5fb <+27>: mov %rsi,%r12 ## %r12=0x17c4c50 [heap block] 0x17c4c40--0x17c6c58 size=8216 rawbuf @+16

0x000000000056f5fe <+30>: mov %edx,%ebx ## %ebx?

0x000000000056f600 <+32>: mov %rcx,%r13 ## %r13=0x41fdbe30 [stack] frame 14 rsp+3888 @0x41fdbe30

0x000000000056f603 <+35>: callq 0x563990 <lex_start(THD*) at /mysql-5.5.28/sql/sql_lex.cc:362> ##

0x000000000056f608 <+40>: mov %rbp,%rdi ## %rdi=0x2aaf95e840 [heap block] 0x2aaf95e840--0x2aaf9615d8 size=11672 thd

0x000000000056f60b <+43>: callq 0x566b00 <mysql_reset_thd_for_next_command(THD*) at /mysql-5.5.28/sql/sql_parse.cc:5322>

. . .

—————————————————————————————————————————–—————————

COMMAND

(windbg) !pattern <start> <end>

(gdb) pattern <start> <end>

DESCRIPTION

Analyze the memory content from address “start” to “end”. It recognizes global variables, stack variables, heap blocks, strings, etc.

Example: The data in the following memory area consists of heap blocks, local variables, function addresses, strings, etc.

0:091> !pattern 0x1935f1a0 0x1935f220

memory pattern [0x1935f1a0, 0x1935f220]:

0x1935f1a0: 0x1935f1f0 => [stack] thread 91 frame 5 iMilliseconds (type="unsigned int" size=4)

0x1935f1a8: 0x080f0d40 => [heap block] 0x80eef10--0x80f2f00 size=16368 @+7728

0x1935f1b0: 0

0x1935f1b8: 0

0x1935f1c0: 0xfffffffe

0x1935f1c8: 0x06cf7900 => [.text/.rodata] MJThread_64!`MSIThread::GetSelfDestruct

0x1935f1d0: 0x1935f1f0 => [stack] thread 91 frame 5 iMilliseconds (type="unsigned int" size=4)

0x1935f1d8: 0x06eaf154 => [.text/.rodata] MJJobExc_64!MSICommandQTask::GetNextCommand

0x1935f1e0: 0x08153970 => [heap block] 0x8153310--0x8155300 size=8176 @+1632

0x1935f1e8: 0x1935f308 => [stack] thread 91 frame 8 lpCommand (type="class IMSIDSSCommand *" size=8)

0x1935f1f0: 0x00002710

0x1935f1f8: 0x003b0000

0x1935f200: 0x08153970 => [heap block] 0x8153310--0x8155300 size=8176 @+1632

0x1935f208: 0x00760500 => [heap block] 0x75ff90--0x761f80 size=8176 @+1392

(wchar_t*) => L"UserA"

0x1935f210: 0x07fc5f30 => [heap block] 0x7fc0080--0x7fc8070 size=32752 @+24240

0x1935f218: 0x06ea6d00 => [.text/.rodata] MJJobExc_64!MSICmdPUThreadInfo

—————————————————————————————————————————–—————————

COMMAND

(windbg) !segment [address]

(gdb) segment [address]

DESCRIPTION

Kernel manages a user process’s address space in terms of memory segments. This command lists all memory segments and their attributes of the target process. It is similar to the utility Linux “pmap” with more information of heap segments. If an address is given, it displays the segment that contains the address.

Example 1: List all segments.

0:091> !segment

vaddr size perm name

=====================================================

[ 0] [0x10000 - 0x12000] 8K rw-

[ 1] [0x20000 - 0x21000] 4K rw-

[ 2] [0x30000 - 0x123000] 972K ---

[ 3] [0x123000 - 0x125000] 8K rw-

[ 4] [0x125000 - 0x130000] 44K rw- [stack] [tid=0]

[ 5] [0x130000 - 0x135000] 20K r--

[ 6] [0x140000 - 0x141000] 4K r--

[ 7] [0x150000 - 0x1f1000] 644K rw- [heap]

[ 8] [0x1f1000 - 0x250000] 380K ---

[ 9] [0x250000 - 0x254000] 16K rw- [heap]

[ 10] [0x254000 - 0x260000] 48K ---

[ 11] [0x260000 - 0x276000] 88K r--

[ 12] [0x280000 - 0x2c1000] 260K r--

[ 13] [0x2d0000 - 0x311000] 260K r--

[ 14] [0x320000 - 0x326000] 24K r--

[ 15] [0x330000 - 0x331000] 4K r--

[ 16] [0x340000 - 0x341000] 4K r-- [.text/.rodata] [Z:\BIN\x64\M8Exec2_64.dll]

[ 17] [0x341000 - 0x37d000] 240K r-x [.text/.rodata] [Z:\BIN\x64\M8Exec2_64.dll]

[ 18] [0x37d000 - 0x39d000] 128K r-- [.data/.bss] [Z:\BIN\x64\M8Exec2_64.dll]

[ 19] [0x39d000 - 0x3a2000] 20K rw- [.data/.bss] [Z:\BIN\x64\M8Exec2_64.dll]

[ 20] [0x3a2000 - 0x3b3000] 68K r-- [.data/.bss] [Z:\BIN\x64\M8Exec2_64.dll]

[ 21] [0x3c0000 - 0x3c1000] 4K r--

[ 22] [0x3d0000 - 0x3d1000] 4K r-- [.text/.rodata] [Z:\BIN\x64\M8DatTy4_64.dll]

. . .

(gdb) segment

vaddr size perm name

=====================================================

[ 0] [0x400000 - 0x403000] 12K r-x [.text] [a.out]

[ 1] [0x502000 - 0x503000] 4K rw- [.data] [a.out]

[ 2] [0x503000 - 0x524000] 132K rwx [heap]

[ 3] [0x2a95556000 - 0x2a95557000] 4K rw-

[ 4] [0x2a95579000 - 0x2a9557c000] 12K rw-

[ 5] [0x3255200000 - 0x3255215000] 84K r-x [.text] [/lib64/ld-linux-x86-64.so.2]

[ 6] [0x3255314000 - 0x3255315000] 4K r--

[ 7] [0x3255315000 - 0x3255316000] 4K rw- [.data] [/lib64/ld-linux-x86-64.so.2]

[ 8] [0x3255400000 - 0x325552f000] 1212K r-x [.text] [/lib64/tls/libc.so.6]

[ 9] [0x325552f000 - 0x325562f000] 1024K ---

[ 10] [0x325562f000 - 0x3255632000] 12K r--

[ 11] [0x3255632000 - 0x3255634000] 8K rw- [.data] [/lib64/tls/libc.so.6]

[ 12] [0x3255634000 - 0x3255639000] 20K rw-

[ 13] [0x3255700000 - 0x3255785000] 532K r-x [.text] [/lib64/tls/libm.so.6]

[ 14] [0x3255785000 - 0x3255884000] 1020K ---

[ 15] [0x3255884000 - 0x3255885000] 4K r--

[ 16] [0x3255885000 - 0x3255886000] 4K rw- [.data] [/lib64/tls/libm.so.6]

[ 17] [0x325a500000 - 0x325a50d000] 52K r-x [.text] [/lib64/libgcc_s.so.1]

[ 18] [0x325a50d000 - 0x325a60c000] 1020K ---

[ 19] [0x325a60c000 - 0x325a60d000] 4K rw- [.data] [/lib64/libgcc_s.so.1]

[ 20] [0x325a700000 - 0x325a7d6000] 856K r-x [.text] [/usr/lib64/libstdc++.so.6]

[ 21] [0x325a7d6000 - 0x325a8d5000] 1020K ---

[ 22] [0x325a8d5000 - 0x325a8de000] 36K rw- [.data] [/usr/lib64/libstdc++.so.6]

[ 23] [0x325a8de000 - 0x325a8f0000] 72K rw-

[ 24] [0x7fbfffe000 - 0x7fc0000000] 8K rw- [stack] [pid=13139 tid=1]

[ 25] [0xffffffffff600000 - 0xffffffffff601000] 4K r-x

Example 2: The segment including address 0x3255632080 is C runtime’s data segment.

(gdb) segment 0x3255632080

Address 0x3255632080 belongs to segment:

[0x3255632000 - 0x3255634000] 8K rw- [.data] [/lib64/tls/libc.so.6]

—————————————————————————————————————————–—————————

COMMAND

(windbg) !set [address] [value]

(windbg) !unset <address>

(gdb) assign [address] [value]

(gdb) unassign <address>

DESCRIPTION

When heap data is damaged, we may not be able to layout the memory blocks around the corruption spot. As a result, affected data objects can’t be identified or searched for reference. If the user has the knowledge of heap data structure, the “set” command instructs core analyzer to replace the invalid data at the given address and repair the heap. However, it doesn’t actually overwrite the memory in the target’s process space. The “unset” command undoes the replacement.

Since gdb already has “set” command built-in, “assign” and “unassign” commands are used correspondingly.

Example: The heap shown below is corrupted at 0x33bf30, which is heap data structure HEAP_ENTRY. Therefore, heap walk is halted at the corruption spot. The memory objects after it are unknown. By manually fix this data structure using “set” command, it reveals the memory layout and likely helps us get insight of the memory corruption.

0:000> !heap 0x00000000`0033bf00

. . .

0x33bef0 - 0x33bf00 [struct HEAP_ENTRY]

0x33bf00 - 0x33bf04 size=4 busy [USER SPACE]

0x33bf04 - 0x33bf30 [Unused Bytes]

[Error] HEAP_ENTRY at 0x33bf30 has an invalid/corrupted size value 65535

Total inuse 38350 bytes

Total free: 0 bytes

[Error] Failed to show the related arena 0x33bf00

0:000> dq 0x00000000`0033bf30 l2

00000000`0033bf30 ffffffff`ffffffff ffffffff`ffffffff

0:000> !set 0x00000000`0033bf30 0xfeeefeee`feeefeee

0:000> !set 0x00000000`0033bf38 0x003b0793`00040004

0:000> !heap 0x00000000`0033bf00

. . .

0x33bef0 - 0x33bf00 [struct HEAP_ENTRY]

0x33bf00 - 0x33bf04 size=4 busy [USER SPACE]

0x33bf04 - 0x33bf30 [Unused Bytes]

0x33bf30 - 0x33bf40 [struct HEAP_ENTRY]

0x33bf40 - 0x33bf45 size=5 busy [USER SPACE]

0x33bf45 - 0x33bf70 [Unused Bytes]

0x33bf70 - 0x33bf80 [struct HEAP_ENTRY]

0x33bf80 - 0x33bf86 size=6 busy [USER SPACE]

0x33bf86 - 0x33bfb0 [Unused Bytes]

0x33bfb0 - 0x33bfc0 [struct HEAP_ENTRY]

0x33bfc0 - 0x33bfc7 size=7 busy [USER SPACE]

0x33bfc7 - 0x33c000 [Unused Bytes]

0x33c000 - 0x340000 [Uncommitted Range]

Total inuse 38368 bytes

Total free: 16384 bytes

—————————————————————————————————————————–—————————

MISCELLANEOUS COMMANDS

(windbg) !max_ref_level <n>

(gdb) max_ref_level <n>

DESCRIPTION

By default, reference search is limited to 16 levels of indirection. This should be enough for most applications. If the object relationship is deeper than this, for example, a long link list, user may set the indirection even higher. Without input number, this command simply shows the current setting.

(windbg) !shrobj_level <n>

(gdb) shrobj_level <n>

DESCRIPTION

By default, the “shrobj” command shows objects shared among threads of up to 1 level of indirection. This command sets the maximum indirection level to n, which should be between 1 to 16. If no argument is given, it shows the current setting of indirection level.

(windbg) !include_free / !ignore_free

(gdb) include_free / ignore_free

DESCRIPTION

By default, core analyzer ignores references in heap memory that is free or thread stack memory that does not belong to any active function because they are stale references. However, some rare cases, like heap corruption due to double-free or access after free, may benefit from these references residing in “free” memory blocks. User can toggle the option for core analyzer to enable or disable searching for free memory.

(gdb) dt <expression>

DESCRIPTION

This is a gdb only command which display an expression’s type information like the output of Windbg’s dt command. The gdb “ptype” command doesn’t show the offset of each data member which is convenient sometimes, for example when debugging optimized or assembly code.

Example

(gdb) dt this

type=MSICommandQTask * size=8

{

+0 (base) struct MSITask size=16

{

+0 size=8

+8 struct MSITaskImp* mTaskImp size=8

}

+16 struct MSIBasicQ<IMSIDSSCommand, MSIDSSCommandInfo>* mQ size=8

+24 int mOwnerId size=4

+32 struct SmartPtr<Event, DeleteOperatorGeneric<Event> > mpStartEvent size=16

+48 struct MSIBasicThreadPool* mThreadPool size=8

}*;

Core Analyzer Integrated with Debuggers