gdb debugging tips

    The object will be displayed in the julia session, not in the gdb session. This is a useful way to discover the types and values of objects being manipulated by Julia’s C code.

    Similarly, if you’re debugging some of Julia’s internals (e.g., compiler.jl), you can print obj using

    1. ccall(:jl_, Cvoid, (Any,), obj)

    This is a good way to circumvent problems that arise from the order in which julia’s output streams are initialized.

    Julia’s flisp interpreter uses value_t objects; these can be displayed with call fl_print(fl_ctx, ios_stdout, obj).

    Useful Julia variables for Inspecting

    While the addresses of many variables, like singletons, can be be useful to print for many failures, there are a number of additional variables (see julia.h for a complete list) that are even more useful.

    • (when in jl_apply_generic) mfunc and jl_uncompress_ast(mfunc->def, mfunc->code) :: for figuring out a bit about the call-stack
    • jl_lineno and jl_filename :: for figuring out what line in a test to go start debugging from (or figure out how far into a file has been parsed)
    • $1 :: not really a variable, but still a useful shorthand for referring to the result of the last gdb command (such as print)
    • jl_options :: sometimes useful, since it lists all of the command line options that were successfully parsed

    Useful Julia functions for Inspecting those variables

    • jl_gdblookup($rip) :: For looking up the current function and line. (use $eip on i686 platforms)
    • jlbacktrace() :: For dumping the current Julia backtrace stack to stderr. Only usable after record_backtrace() has been called.
    • jl_dump_llvm_value(Value*) :: For invoking Value->dump() in gdb, where it doesn’t work natively. For example, f->linfo->functionObject, f->linfo->specFunctionObject, and to_function(f->linfo).
    • Type->dump() :: only works in lldb. Note: add something like ;1 to prevent lldb from printing its prompt over the output
    • jl_eval_string("expr") :: for invoking side-effects to modify the current state or to lookup symbols
    • jl_typeof(jl_value_t*) :: for extracting the type tag of a Julia value (in gdb, call macro define jl_typeof jl_typeof first, or pick something short like ty for the first arg to define a shorthand)

    In your session, set a breakpoint in jl_breakpoint like so:

    1. (gdb) break jl_breakpoint

    Then within your Julia code, insert a call to jl_breakpoint by adding

    1. ccall(:jl_breakpoint, Cvoid, (Any,), obj)

    where obj can be any variable or tuple you want to be accessible in the breakpoint.

    It’s particularly helpful to back up to the jl_apply frame, from which you can display the arguments to a function using, e.g.,

    1. (gdb) call jl_(args[0])
    1. #2  0x00007ffff7928bf7 in to_function (li=0x2812060, cstyle=false) at codegen.cpp:584
    2. 584          abort();
    3. (gdb) p jl_(jl_uncompress_ast(li, li->ast))

    Inserting breakpoints upon certain conditions

    Let’s say the file is sysimg.jl:

    Calling a particular method

    1. (gdb) break jl_apply_generic if strcmp((char*)(jl_symbol_name)(jl_gf_mtable(F)->name), "method_to_break")==0

    Since this function is used for every call, you will make everything 1000x slower if you do this.

    Dealing with signals

    Julia requires a few signal to function property. The profiler uses SIGUSR2 for sampling and the garbage collector uses SIGSEGV for threads synchronization. If you are debugging some code that uses the profiler or multiple threads, you may want to let the debugger ignore these signals since they can be triggered very often during normal operations. The command to do this in GDB is (replace SIGSEGV with SIGUSRS or other signals you want to ignore):

    1. (gdb) handle SIGSEGV noprint nostop pass

    The corresponding LLDB command is (after the process is started):

    1. (lldb) pro hand -p true -s false -n false SIGSEGV

    If you are debugging a segfault with threaded code, you can set a breakpoint on jl_critical_error (sigdie_handler should also work on Linux and BSD) in order to only catch the actual segfault rather than the GC synchronization points.

    Errors that occur during make need special handling. Julia is built in two stages, constructing sys0 and sys.ji. To see what commands are running at the time of failure, use make VERBOSE=1.

    At the time of this writing, you can debug build errors during the sys0 phase from the base directory using:

    1. julia/base$ gdb --args ../usr/bin/julia-debug -C native --build ../usr/lib/julia/sys0 sysimg.jl

    You might need to delete all the files in usr/lib/julia/ to get this to work.

    You can debug the sys.ji phase using:

      Once an error is caught, a useful technique is to walk up the stack and examine the function by inspecting the related call to jl_apply. To take a real-world example:

      The most recent jl_apply is at frame #3, so we can go back there and look at the AST for the function julia_convert_16886. This is the uniqued name for some method of convert. f in this frame is a jl_function_t*, so we can look at the type signature, if any, from the specTypes field:

      1. (gdb) f 3
      2. #3  0x00007ffff6541154 in jl_apply (f=0x7ffdf367f630, args=0x7fffffffc2b0, nargs=2) at julia.h:1281
      3. 1281            return f->fptr((jl_value_t*)f, args, nargs);
      4. (gdb) p f->linfo->specTypes
      5. $4 = (jl_tupletype_t *) 0x7ffdf39b1030
      6. (gdb) p jl_( f->linfo->specTypes )
      7. Tuple{Type{Float32}, Float64}           # <-- type signature for julia_convert_16886

      Then, we can look at the AST for this function:

      1. (gdb) p jl_( jl_uncompress_ast(f->linfo, f->linfo->ast) )
      2. Expr(:lambda, Array{Any, 1}[:#s29, :x], Array{Any, 1}[Array{Any, 1}[], Array{Any, 1}[Array{Any, 1}[:#s29, :Any, 0], Array{Any, 1}[:x, :Any, 0]], Array{Any, 1}[], 0], Expr(:body,
      3. Expr(:line, 90, :float.jl)::Any,
      4. Expr(:return, Expr(:call, :box, :Float32, Expr(:call, :fptrunc, :Float32, :x)::Any)::Any)::Any)::Any)::Any

      Finally, and perhaps most usefully, we can force the function to be recompiled in order to step through the codegen process. To do this, clear the cached functionObject from the jl_lamdbda_info_t*:

      1. (gdb) p f->linfo->functionObject
      2. $8 = (void *) 0x1289d070
      3. (gdb) set f->linfo->functionObject = NULL

      Then, set a breakpoint somewhere useful (e.g. emit_function, emit_expr, emit_call, etc.), and run codegen:

      1. (gdb) p jl_compile(f)
      2. ... # your breakpoint here

      Debugging precompilation errors

      Module precompilation spawns a separate Julia process to precompile each module. Setting a breakpoint or catching failures in a precompile worker requires attaching a debugger to the worker. The easiest approach is to set the debugger watch for new process launches matching a given name. For example:

        or:

        Then run a script/command to start precompilation. As described earlier, use conditional breakpoints in the parent process to catch specific file-loading events and narrow the debugging window. (some operating systems may require alternative approaches, such as following each from the parent process)

        Mozilla’s Record and Replay Framework (rr)

        Julia now works out of the box with , the lightweight recording and deterministic debugging framework from Mozilla. This allows you to replay the trace of an execution deterministically. The replayed execution’s address spaces, register contents, syscall data etc are exactly the same in every run.