There are several techniques which we can use to try and help get to the bottom of these:

Interpreting hs_err_pid files

If the JVM crashes whilst executing our native C++ code via JNI, then it will typically write an to a file, which on Linux may be named like /tmp/jvm-8666/hs_error.log, or on a Mac may be named likehs_err_pid76448.log in the same location that the java process was launched from.

Such a error report file might look like (Mac):

The most interesting part of the trace is likely the stack frames. For example consider this frame:

  1. C  [librocksdbjni-osx.jnilib+0x1c38a]  rocksdb::InternalKeyComparator::InternalKeyComparator(rocksdb::Comparator const*)+0x4a

We can see that something went wrong from a function (in this case a constructor) in rocksdb::InternalKeyComparator, however how do we relate these back to file and line-numbers in our source code?

We have to translate the offsets provided in the trace:

On a Mac this would look like:

  1. $ atos -o java/target/librocksdbjni-osx.jnilib 0x1c38a
  2. ava_org_rocksdb_Logger_setInfoLogLevel (in librocksdbjni-osx.jnilib) (loggerjnicallback.cc:152)

Linux

On a Linux system this would look like:

  1. $ addr2line -e java/target/librocksjni-linux64.so 0x1c38a

TODO

ASAN

Mac (Apple LLVM 7.3.0)

  • Set JDK 7 as required by RocksJava
  1. export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home
  • Ensure a clean start:
  1. make clean jclean
  • Compile the Java test suite with ASAN compiled in:
  1. DEBUG_LEVEL=2 COMPILE_WITH_ASAN=true make jtest_compile
  • Execute the entire Java Test Suite:
  1. make jtest_run

or for a single test (e.g. ComparatorTest), execute:

  1. cd java
  2. java -ea -Xcheck:jni -Djava.library.path=target -cp "target/classes:target/test-classes:test-libs/junit-4.12.jar:test-libs/hamcrest-core-1.3.jar:test-libs/mockito-all-1.10.19.jar:test-libs/cglib-2.2.2.jar:test-libs/assertj-core-1.7.1.jar:target/*" org.rocksdb.test.RocksJunitRunner org.rocksdb.ComparatorTest

NOTE: if you see an error like:

  2. DYLD_INSERT_LIBRARIES=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.1.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib
  3. "interceptors not installed" && 0

Then you need to fist execute:

  1. $ export DYLD_INSERT_LIBRARIES=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.1.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib

NOTE: Also:

If ASAN detects an issue, you will see output similar to the following:

  1. Run: org.rocksdb.BackupableDBOptionsTest testing now -> destroyOldData 
  2. Run: org.rocksdb.BackupEngineTest testing now -> deleteBackup 
  3. =================================================================
  4. ==80632==ERROR: AddressSanitizer: unknown-crash on address 0x7fd93940d6e8 at pc 0x00011cebe075 bp 0x70000020ffe0 sp 0x70000020ffd8
  5. WRITE of size 8 at 0x7fd93940d6e8 thread T0
  6.     #0 0x11cebe074 in rocksdb::PosixLogger::PosixLogger(__sFILE*, unsigned long long (*)(), rocksdb::Env*, rocksdb::InfoLogLevel) posix_logger.h:47
  7.     #1 0x11cebc847 in rocksdb::PosixLogger::PosixLogger(__sFILE*, unsigned long long (*)(), rocksdb::Env*, rocksdb::InfoLogLevel) posix_logger.h:53
  8.     #2 0x11ce9888c in rocksdb::(anonymous namespace)::PosixEnv::NewLogger(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<rocksdb::Logger>*) env_posix.cc:574
  9.     #3 0x11c09a3e3 in rocksdb::CreateLoggerFromOptions(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DBOptions const&, std::__1::shared_ptr<rocksdb::Logger>*) auto_roll_logger.cc:166
  10.     #4 0x11c3a8a55 in rocksdb::SanitizeOptions(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DBOptions const&) db_impl.cc:143
  11.     #5 0x11c3ac2f3 in rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) db_impl.cc:307
  12.     #6 0x11c3b38b4 in rocksdb::DBImpl::DBImpl(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) db_impl.cc:350
  13.     #7 0x11c4497bc in rocksdb::DB::Open(rocksdb::DBOptions const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<rocksdb::ColumnFamilyDescriptor, std::__1::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::__1::vector<rocksdb::ColumnFamilyHandle*, std::__1::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**) db_impl.cc:5665
  14.     #8 0x11c447b74 in rocksdb::DB::Open(rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**) db_impl.cc:5633
  15.     #9 0x11bff8ca4 in rocksdb::Status std::__1::__invoke_void_return_wrapper<rocksdb::Status>::__call<rocksdb::Status (*&)(rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**), rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**>(rocksdb::Status (*&&&)(rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**), rocksdb::Options const&&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&&&, rocksdb::DB**&&) __functional_base:437
  16.     #10 0x11bff89ff in std::__1::__function::__func<rocksdb::Status (*)(rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**), std::__1::allocator<rocksdb::Status (*)(rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**)>, rocksdb::Status (rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**)>::operator()(rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**&&) functional:1437
  17.     #11 0x11bff269b in std::__1::function<rocksdb::Status (rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**)>::operator()(rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**) const functional:1817
  18.     #12 0x11bfd6edb in rocksdb_open_helper(JNIEnv_*, long, _jstring*, std::__1::function<rocksdb::Status (rocksdb::Options const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, rocksdb::DB**)>) rocksjni.cc:37
  19.     #13 0x11bfd723e in Java_org_rocksdb_RocksDB_open__JLjava_lang_String_2 rocksjni.cc:55
  20.     #14 0x10be77757  (<unknown module>)
  21.     #15 0x10be6b174  (<unknown module>)
  22.     #16 0x10be6b232  (<unknown module>)
  23.     #17 0x10be654e6  (<unknown module>)
  24.     #18 0x10b6dc897 in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) (libjvm.dylib+0x2dc897)
  25.     #19 0x10b6dc667 in JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*) (libjvm.dylib+0x2dc667)
  26.     #20 0x10b868427 in Reflection::invoke(instanceKlassHandle, methodHandle, Handle, bool, objArrayHandle, BasicType, objArrayHandle, bool, Thread*) (libjvm.dylib+0x468427)
  27.     #21 0x10b86888d in Reflection::invoke_method(oopDesc*, Handle, objArrayHandle, Thread*) (libjvm.dylib+0x46888d)
  28.     #22 0x10b729246 in JVM_InvokeMethod (libjvm.dylib+0x329246)
  29.     #23 0x10be77757  (<unknown module>)
  30.     #24 0x10be6b232  (<unknown module>)
  31.     #25 0x10be6b232  (<unknown module>)
  32.     #26 0x10be6b8e0  (<unknown module>)
  33.     #27 0x10be6b232  (<unknown module>)
  34.     #28 0x10be6b232  (<unknown module>)
  35.     #29 0x10be6b232  (<unknown module>)
  36.     #30 0x10be6b232  (<unknown module>)
  37.     #31 0x10be6b057  (<unknown module>)
  38.     #32 0x10be6b057  (<unknown module>)
  39.     #33 0x10be6b057  (<unknown module>)
  40.     #34 0x10be6b057  (<unknown module>)
  41.     #35 0x10be6b057  (<unknown module>)
  42.     #37 0x10be6b057  (<unknown module>)
  43.     #38 0x10be6b705  (<unknown module>)
  44.     #39 0x10be6b705  (<unknown module>)
  45.     #40 0x10be6b057  (<unknown module>)
  46.     #41 0x10be6b057  (<unknown module>)
  47.     #42 0x10be6b057  (<unknown module>)
  48.     #43 0x10be6b057  (<unknown module>)
  49.     #44 0x10be6b057  (<unknown module>)
  50.     #45 0x10be6b057  (<unknown module>)
  51.     #46 0x10be6b057  (<unknown module>)
  52.     #47 0x10be6b057  (<unknown module>)
  53.     #48 0x10be6b705  (<unknown module>)
  54.     #49 0x10be6b705  (<unknown module>)
  55.     #50 0x10be6b057  (<unknown module>)
  56.     #51 0x10be6b057  (<unknown module>)
  57.     #52 0x10be6b057  (<unknown module>)
  58.     #53 0x10be6b057  (<unknown module>)
  59.     #54 0x10be6b232  (<unknown module>)
  60.     #55 0x10be6b232  (<unknown module>)
  61.     #56 0x10be6b232  (<unknown module>)
  62.     #57 0x10be6b232  (<unknown module>)
  63.     #58 0x10be654e6  (<unknown module>)
  64.     #59 0x10b6dc897 in JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*) (libjvm.dylib+0x2dc897)
  65.     #60 0x10b6dc667 in JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*) (libjvm.dylib+0x2dc667)
  66.     #61 0x10b71004d in jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, Thread*) (libjvm.dylib+0x31004d)
  67.     #62 0x10b7092d4 in jni_CallStaticVoidMethodV (libjvm.dylib+0x3092d4)
  68.     #63 0x10b71c28d in checked_jni_CallStaticVoidMethod (libjvm.dylib+0x31c28d)
  69.     #64 0x109fdd0fd in JavaMain (java+0x1000030fd)
  70.     #65 0x7fff8df9c99c in _pthread_body (libsystem_pthread.dylib+0x399c)
  71.     #66 0x7fff8df9c919 in _pthread_start (libsystem_pthread.dylib+0x3919)
  72.     #67 0x7fff8df9a350 in thread_start (libsystem_pthread.dylib+0x1350)
  73.  
  74. AddressSanitizer can not describe address in more detail (wild memory access suspected).
  75. SUMMARY: AddressSanitizer: unknown-crash posix_logger.h:47 in rocksdb::PosixLogger::PosixLogger(__sFILE*, unsigned long long (*)(), rocksdb::Env*, rocksdb::InfoLogLevel)
  76. Shadow bytes around the buggy address:
  77.   0x1ffb27281a80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  78.   0x1ffb27281a90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  79.   0x1ffb27281aa0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  80.   0x1ffb27281ab0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  81.   0x1ffb27281ac0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  82. =>0x1ffb27281ad0: 00 00 00 00 00 00 00 00 00 04 00 00 00[04]00 00
  83.   0x1ffb27281ae0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  84.   0x1ffb27281af0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  85.   0x1ffb27281b00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  86.   0x1ffb27281b10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  87.   0x1ffb27281b20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  88. Shadow byte legend (one shadow byte represents 8 application bytes):
  89.   Addressable:           00
  90.   Partially addressable: 01 02 03 04 05 06 07 
  91.   Heap left redzone:       fa
  92.   Heap right redzone:      fb
  93.   Freed heap region:       fd
  94.   Stack left redzone:      f1
  95.   Stack mid redzone:       f2
  96.   Stack right redzone:     f3
  97.   Stack partial redzone:   f4
  98.   Stack after return:      f5
  99.   Stack use after scope:   f8
  100.   Global redzone:          f9
  101.   Global init order:       f6
  102.   Poisoned by user:        f7
  103.   Container overflow:      fc
  104.   Array cookie:            ac
  105.   Intra object redzone:    bb
  106.   ASan internal:           fe
  107.   Left alloca redzone:     ca
  108.   Right alloca redzone:    cb
  109. ==80632==ABORTING
  110. make[1]: *** [run_test] Abort trap: 6
  111. make: *** [jtest_run] Error 2

The output from ASAN shows a stack-trace with file names and line numbers of our C++ code that led to the issue, hopefully this helps shed some light on where the issue is occurring and perhaps why.

Unfortunately all of those (<unknown module>) are execution paths inside the JVM, ASAN cannot discover them because the JVM we are using was not itself build with support for ASAN. We could attempt to build our own JVM from the OpenJDK project and include ASAN, but at the moment that process for Mac OS X seems to be broken: https://github.com/hgomez/obuildfactory/issues/51.

TODO Note the path of the DSO for libasan on Mac OS X: /Library/Developer/CommandLineTools/usr/lib/clang/7.3.0/lib/darwin/libclang_rt.asan_osx_dynamic.dylib

  • Set JDK 7 as required by RocksJava

You might also need to run sudo alternatives —config java and select OpenJDK 7.

  • Ensure a clean start:
    • Compile the Java test suite with ASAN compiled in:
    1. DEBUG_LEVEL=2 COMPILE_WITH_ASAN=true make jtest_compile
    • Execute the entire Java Test Suite:
    1. LD_PRELOAD=/usr/lib64/libasan.so.0 make jtest_run
    1. cd java
    2. LD_PRELOAD=/usr/lib64/libasan.so.0 java -ea -Xcheck:jni -Djava.library.path=target -cp "target/classes:target/test-classes:test-libs/junit-4.12.jar:test-libs/hamcrest-core-1.3.jar:test-libs/mockito-all-1.10.19.jar:test-libs/cglib-2.2.2.jar:test-libs/assertj-core-1.7.1.jar:target/*" org.rocksdb.test.RocksJunitRunner org.rocksdb.ComparatorTest

    If ASAN detects an issue, you will see output similar to the following:

    1. Run: org.rocksdb.util.BytewiseComparatorTest testing now -> java_vs_java_directBytewiseComparator 
    2. ASAN:SIGSEGV
    3. =================================================================
    4. ==4665== ERROR: AddressSanitizer: SEGV on unknown address 0x0000fffffff0 (pc 0x7fd481f913e5 sp 0x7fd48599e308 bp 0x7fd48599e340 T1)
    5. AddressSanitizer can not provide additional info.
    6.     #0 0x7fd481f913e4 (/usr/lib64/libc-2.17.so+0x1633e4)
    7.     #1 0x7fd48282da65 (/usr/lib64/libasan.so.0.0.0+0xfa65)
    8.     #2 0x7fd481be5944 (/usr/lib64/libstdc++.so.6.0.19+0xbf944)
    9.     #3 0x7fd3c57bcfc2 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x714fc2)
    10.     #4 0x7fd3c57edb07 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x745b07)
    11.     #5 0x7fd3c57f215d (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x74a15d)
    12.     #6 0x7fd3c59f3774 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x94b774)
    13.     #7 0x7fd3c59eb598 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x943598)
    14.     #8 0x7fd3c58a2c11 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x7fac11)
    15.     #9 0x7fd3c58c64bf (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x81e4bf)
    16.     #10 0x7fd3c58c5bc8 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x81dbc8)
    17.     #11 0x7fd3c57a7bc4 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x6ffbc4)
    18.     #12 0x7fd3c57a5fc5 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x6fdfc5)
    19.     #13 0x7fd3c579bd80 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x6f3d80)
    20.     #14 0x7fd3c579bef7 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x6f3ef7)
    21.     #15 0x7fd47c86ae97 (+0x14e97)
    22. Thread T1 created by T0 here:
    23.     #0 0x7fd482828c3a (/usr/lib64/libasan.so.0.0.0+0xac3a)
    24.     #1 0x7fd4823fd7cf (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101-2.6.6.1.el7_2.x86_64/jre/lib/amd64/jli/libjli.so+0x97cf)
    25.     #2 0x7fd4823f8386 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101-2.6.6.1.el7_2.x86_64/jre/lib/amd64/jli/libjli.so+0x4386)
    26.     #3 0x7fd4823f8e38 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101-2.6.6.1.el7_2.x86_64/jre/lib/amd64/jli/libjli.so+0x4e38)
    27.     #4 0x400774 (/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101-2.6.6.1.el7_2.x86_64/jre-abrt/bin/java+0x400774)
    28.     #5 0x7fd481e4fb14 (/usr/lib64/libc-2.17.so+0x21b14)
    29. ==4665== ABORTING
    30. make[1]: *** [run_test] Error 1
    31. make[1]: Leaving directory `/home/aretter/rocksdb/java'
    32. make: *** [jtest_run] Error 2

    The addresses presented in the stack-trace from GCC ASAN on Linux, can be translated into file and line-numbers by using addr2line, for example:

    Given the stack frame (from above):

    1. #3 0x7fd3c57bcfc2 (/home/aretter/rocksdb/java/target/librocksdbjni-linux64.so+0x714fc2)

    We can translate it with the command:

    1. $ addr2line -e java/target/librocksdbjni-linux64.so 0x714fc2
    2. /home/aretter/rocksdb/./db/dbformat.h:126

    Linux (Ubuntu 16.04) (GCC 5.4.0)

    • Set JDK 7 as required by RocksJava
    1. export JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64"
    2. export PATH="${PATH}:${JAVA_HOME}/bin"

    You might also need to run sudo alternatives —config java and select OpenJDK 7.

    • Ensure a clean start:
    1. make clean jclean
    • Compile the Java test suite with ASAN compiled in:
    1. DEBUG_LEVEL=2 COMPILE_WITH_ASAN=true make jtest_compile
    • Execute the entire Java Test Suite:
    1. LD_PRELOAD=/usr/lib/gcc/x86_64-linux-gnu/5.4.0/libasan.so make jtest_run

    or for a single test (e.g. ComparatorTest), execute:

    C++ Debugger

    When things get desperate you can also run your RocksJava tests through the C++ debugger, to trace the C++ JNI code in RocksJava.

    • Set JDK 7 as required by RocksJava
    1. export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home"
    2. export PATH="${PATH}:${JAVA_HOME}/bin"
    • Ensure a clean start:
    1. make clean jclean
    • Compile the RocksJava statically:
    1. DEBUG_LEVEL=2 make rocksdbjavastatic
    • Start LLDB with a single RocksJava test:
    1.     lldb -- /Library/Java/JavaVirtualMachines/jdk1.7.0_80.jdk/Contents/Home/bin/java -ea -Xcheck:jni -Djava.library.path=target -cp "target/classes:target/test-classes:test-libs/junit-4.12.jar:test-libs/hamcrest-core-1.3.jar:test-libs/mockito-all-1.10.19.jar:test-libs/cglib-2.2.2.jar:test-libs/assertj-core-1.7.1.jar:target/*" org.rocksdb.test.RocksJunitRunner org.rocksdb.ComparatorTest
    • Using LLDB with RocksJava:You can then start the RocksJava test under lldb:
    1. (lldb) run

    You will likely need to instruct gdb not to stop on internal SIGSEGV and SIGBUS signals generated by the JVM:

    1. (lldb) pro hand -p true -s false SIGSEGV
    2. (lldb) pro hand -p true -s false SIGBUS

    gdb (Linux)

    • Set JDK 7 as required by RocksJava
    1. export JAVA_HOME="/usr/lib/jvm/java-7-openjdk-amd64"
    2. export PATH="${PATH}:${JAVA_HOME}/bin"

    You might also need to run and select OpenJDK 7.

    • Ensure a clean start:
    1. make clean jclean
    • Compile the RocksJava statically:
    1. DEBUG_LEVEL=2 make rocksdbjavastatic
    • Start GDB with a single RocksJava test:
    1.     gdb --args java -ea -Xcheck:jni -Djava.library.path=target -cp "target/classes:target/test-classes:test-libs/junit-4.12.jar:test-libs/hamcrest-core-1.3.jar:test-libs/mockito-all-1.10.19.jar:test-libs/cglib-2.2.2.jar:test-libs/assertj-core-1.7.1.jar:target/*" org.rocksdb.test.RocksJunitRunner org.rocksdb.ComparatorTest
    1. gdb> handle SIGSEGV pass noprint nostop