Debug Tool

    pprof (opens new window): from gperftools, it is used to transform the content generated by gperftools into a format that is easy for people to read, such as PDF, SVG, text, etc.

    : in the absence of this library, pprof can only be converted to text format, but this way is not easy to view. After the library is installed, pprof can be converted to SVG, PDF and other formats, and the call relationship is clearer.

    perf (opens new window): Linux kernel comes with performance analysis tool. there are some examples of perf usage.

    flamegraph (opens new window): a visualization tool used to show the output of perf in the form of flame graph.

    Debugging memory is generally divided into two aspects. One is whether the total amount of memory use is reasonable. On the one hand, the excessive amount of memory use may be due to memory leak in the system, on the other hand, it may be due to improper use of program memory. The second is whether there is a problem of memory overrun and illegal access, such as program access to memory with an illegal address, use of uninitialized memory, etc. For the debugging of memory, we usually use the following ways to track the problems.

    When we find that the memory usage is too large, we can first check the be.out log to see if there is a large memory application. Because of the TCMalloc currently used by Doris to manage memory, when a large memory application is encountered, the stack of the application will be printed to the be.out file. The general form is as follows:

    This indicates that Doris be is trying to apply memory of ‘1396277248 bytes’ on this stack. We can use the ‘addr2line’ command to restore the stack to a letter that we can understand. The specific example is shown below.

    3. /home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1335
    4. /home/ssd0/zc/palo/doris/core/thirdparty/src/gperftools-gperftools-2.7/src/tcmalloc.cc:1357
    5. /home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.cpp:267
    6. /home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_table.hpp:86
    7. /home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:239
    8. /home/disk0/baidu-doris/baidu/bdg/doris-baidu/core/be/src/exec/hash_join_node.cpp:213
    9. thread.cpp:?

    HEAP PROFILE

    Sometimes the application of memory is not caused by the application of large memory, but by the continuous accumulation of small memory. Then there is no way to locate the specific application information by viewing the log, so you need to get the information through other ways.

    At this time, we can take advantage of TCMalloc’s heapprofileDebug Tool - 图6 (opens new window). If the heapprofile function is set, we can get the overall memory application usage of the process. The usage is to set the ‘heapprofile’ environment variable before starting Doris be. For example:

    1. export HEAPPROFILE=/tmp/doris_be.hprof
    2. ./bin/start_be.sh --daemon

    In this way, when the dump condition of the heapprofile is met, the overall memory usage will be written to the file in the specified path. Later, we can use the ‘pprof’ tool to analyze the output content.

    1. $ pprof --text lib/palo_be /tmp/doris_be.hprof.0012.heap | head -30
    3. Using local file lib/palo_be.
    4. Using local file /tmp/doris_be.hprof.0012.heap.
    5.    610.6  91.3%  91.3%    610.6  91.3% doris::SystemAllocator::allocate_via_malloc (inline)
    6.     18.1   2.7%  94.0%     18.1   2.7% _objalloc_alloc
    7.      5.6   0.8%  94.9%     63.4   9.5% doris::RowBatch::RowBatch
    8.      5.1   0.8%  95.6%      7.1   1.1% butil::ResourcePool::add_block (inline)
    9.      3.7   0.5%  96.2%      3.7   0.5% butil::iobuf::create_block (inline)
    10.      3.4   0.5%  96.7%      3.4   0.5% butil::FlatMap::init
    12.      2.6   0.4%  97.6%      2.6   0.4% __gnu_cxx::new_allocator::allocate (inline)
    13.      2.0   0.3%  97.9%      2.0   0.3% butil::ObjectPool::add_block_group (inline)
    14.      2.0   0.3%  98.2%      2.0   0.3% butil::ResourcePool::add_block_group (inline)
    15.      1.7   0.3%  98.4%      1.7   0.3% doris::SegmentReader::_load_index

    Contents of each column of the above documents:

    • Column 1: the memory size directly applied by the function, in MB
    • Column 4: the total memory size of the function and all the functions it calls.
    • The second column and the fifth column are the proportion values of the first column and the fourth column respectively.
    • The third column is the cumulative value of the second column.

    Of course, it can also generate call relation pictures, which is more convenient for analysis. For example, the following command can generate a call graph in SVG format.

    1. pprof --svg lib/palo_be /tmp/doris_be.hprof.0012.heap > heap.svg

    NOTE: turning on this option will affect the execution performance of the program. Please be careful to turn on the online instance.

    For Doris be, you can also use the way of opening and closing the heap profile dynamically to analyze the memory application of the process. Doris supports the . Then you can use ‘pprof’ to directly perform dynamic head profile on the remote running Doris be. For example, we can check the memory usage increment of Doris through the following command

    The output of this command is the same as the output and view mode of heap profile, which will not be described in detail here. Statistics will be enabled only during execution of this command, which has a limited impact on process performance compared with heap profile.

    LSAN

    is an address checking tool, GCC has been integrated. When we compile the code, we can enable this function by turning on the corresponding compilation options. When the program has a determinable memory leak, it prints the leak stack. Doris be has integrated this tool, only need to compile with the following command to generate be binary with memory leak detection version.

    1. BUILD_TYPE=LSAN ./build.sh

    When the system detects a memory leak, it will output the corresponding information in be. Out. For the following demonstration, we intentionally insert a memory leak code into the code. We insert the following code into the open function of StorageEngine.

    1.     char* leak_buf = new char[1024];
    2.     strcpy(leak_buf, "hello world");
    3.     LOG(INFO) << leak_buf;

    We get the following output in be.out

    1. =================================================================
    2. ==24732==ERROR: LeakSanitizer: detected memory leaks
    4. Direct leak of 1024 byte(s) in 1 object(s) allocated from:
    5.     #0 0xd10586 in operator new[](unsigned long) ../../../../gcc-7.3.0/libsanitizer/lsan/lsan_interceptors.cc:164
    6.     #2 0xd3cc96 in main /home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159
    7.     #3 0x7f573b5eebd4 in __libc_start_main (/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4)
    9. SUMMARY: LeakSanitizer: 1024 byte(s) leaked in 1 allocation(s).

    From the above output, we can see that 1024 bytes have been leaked, and the stack information of memory application has been printed out.

    NOTE: turning on this option will affect the execution performance of the program. Please be careful to turn on the online instance.

    NOTE: if the LSAN switch is turned on, the TCMalloc will be automatically turned off

    Except for the unreasonable use and leakage of memory. Sometimes there will be memory access illegal address and other errors. At this time, we can use ASAN (opens new window) to help us find the cause of the problem. Like LSAN, ASAN is integrated into GCC. Doris can open this function by compiling as follows

    Execute the binary generated by compilation. When the detection tool finds any abnormal access, it will immediately exit and output the stack illegally accessed in be.out. The output of ASAN is the same as that of LSAN. Here we also actively inject an address access error to show the specific content output. We still inject an illegal memory access into the ‘open’ function of ‘storageengine’. The specific error code is as follows

    We get the following output in be.out

    1. =================================================================
    2. ==23284==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x61900008bf80 at pc 0x00000129f56a bp 0x7fff546eed90 sp 0x7fff546eed88
    3. WRITE of size 1 at 0x61900008bf80 thread T0
    4.     #0 0x129f569 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:106
    5.     #1 0xe2c1e3 in main /home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159
    6.     #2 0x7fa5580fbbd4 in __libc_start_main (/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4)
    7.     #3 0xd30794  (/home/ssd0/zc/palo/doris/core/output3/be/lib/palo_be+0xd30794)
    9. 0x61900008bf80 is located 0 bytes to the right of 1024-byte region [0x61900008bb80,0x61900008bf80)
    10. allocated by thread T0 here:
    11.     #0 0xdeb040 in operator new[](unsigned long) ../../../../gcc-7.3.0/libsanitizer/asan/asan_new_delete.cc:82
    12.     #1 0x129f50d in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**) /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:104
    13.     #2 0xe2c1e3 in main /home/ssd0/zc/palo/doris/core/be/src/service/doris_main.cpp:159
    14.     #3 0x7fa5580fbbd4 in __libc_start_main (/opt/compiler/gcc-4.8.2/lib64/libc.so.6+0x21bd4)
    16. SUMMARY: AddressSanitizer: heap-buffer-overflow /home/ssd0/zc/palo/doris/core/be/src/olap/storage_engine.cpp:106 in doris::StorageEngine::open(doris::EngineOptions const&, doris::StorageEngine**)

    From this message, we can see that at the address of 0x61900008bf80, we tried to write a byte, but this address is illegal. We can also see the application stack of the address [0x61900008bb80, 0x61900008bf80].

    NOTE: turning on this option will affect the execution performance of the program. Please be careful to turn on the online instance.

    In addition, if stack information is output in be.out, but there is no function symbol, then we need to handle it manually to get readable stack information. The specific processing method needs a script to parse the output of ASAN. At this time, we need to use to help with parsing. The specific usage is as follows:

    1. cat be.out | python asan_symbolize.py | c++filt

    With the above command, we can get readable stack information.

    When the CPU idle of the system is very low, it means that the CPU of the system has become the main bottleneck. At this time, it is necessary to analyze the current CPU usage. For the be of Doris, there are two ways to analyze the CPU bottleneck of Doris.

    pprof

    Because Doris has integrated and compatible with GPERF rest interface, users can analyze remote Doris be through the ‘pprof’ tool. The specific usage is as follows:

    1. pprof --svg --seconds=60 http://be_host:be_webport/pprof/profile > be.svg

    In this way, a CPU consumption graph of be execution can be generated.

    This is a quite common CPU analysis method. Compared with pprof, this method must be able to log in to the physical machine of the analysis object. However, compared with pprof, which can only collect points on time, perf can collect stack information through different events. The specific usage is as follows:

    This command counts the CPU operation of be for 60 seconds and generates perf.data. For the analysis of perf.data, the command of perf can be used for analysis.

    The analysis results in the following pictures

    Perf Report

    To analyze the generated content. Of course, you can also use flash graph to complete the visual display.

    1. perf script | ./FlameGraph/stackcollapse-perf.pl | ./FlameGraph/flamegraph.pl > be.svg

    This will also generate a graph of CPU consumption at that time.