XRay Instrumentation

More high level information about XRay can be found in the XRay whitepaper.

This document describes how to use XRay as implemented in LLVM.

XRay consists of three main parts:

• Compiler-inserted instrumentation points.

• A suite of tools for analysing the traces.

NOTE: As of July 25, 2018 , XRay is only available for the followingarchitectures running Linux: x86_64, arm7 (no thumb), aarch64, powerpc64le,mips, mipsel, mips64, mips64el, NetBSD: x86_64, FreeBSD: x86_64 andOpenBSD: x86_64.

The compiler-inserted instrumentation points come in the form of nop-sleds inthe final generated binary, and an ELF section named whichcontains entries pointing to these instrumentation points. The runtime libraryrelies on being able to access the entries of the xray_instr_map, andoverwrite the instrumentation points at runtime.

You can use XRay in a couple of ways:

• Instrumenting your C/C++/Objective-C/Objective-C++ application.
• Generating LLVM IR with the correct function attributes.

The rest of this section covers these main ways and later on how to customisewhat XRay does in an XRay-instrumented binary.

The easiest way of getting XRay instrumentation for your application is byenabling the -fxray-instrument flag in your clang invocation.

By default, functions that have at least 200 instructions will get XRayinstrumentation points. You can tweak that number through the-fxray-instruction-threshold= flag:

clang -fxray-instrument -fxray-instruction-threshold=1 ...

You can also specifically instrument functions in your binary to either alwaysor never be instrumented using source-level attributes. You can do it using theGCC-style attributes or C++11-style attributes.

When linking a binary, you can either manually link in the or use clang to link it in automatically with the-fxray-instrument flag. Alternatively, you can statically link-in the XRayruntime library from compiler-rt – those archive files will take the name oflibclang_rt.xray-{arch} where {arch} is the mnemonic supported by clang(x86_64, arm7, etc.).

LLVM Function Attribute

If you’re using LLVM IR directly, you can add the function-instrumentstring attribute to your functions, to get the similar effect that theC/C++/Objective-C source-level attributes would get:

define i32 @always_instrument() uwtable "function-instrument"="xray-always" {
  ; ...
}
 
define i32 @never_instrument() uwtable "function-instrument"="xray-never" {
  ; ...
}

You can also set the xray-instruction-threshold attribute and provide anumeric string value for how many instructions should be in the function beforeit gets instrumented.

Special Case File

Attributes can be imbued through the use of special case files instead ofadding them to the original source files. You can use this to mark certainfunctions and classes to be never, always, or instrumented with first-argumentlogging from a file. The file’s format is described below:

# Comments are supported
[always]
fun:always_instrument
 
[never]
fun:never_instrument

These files can be provided through the -fxray-attr-list= flag to clang.You may have multiple files loaded through multiple instances of the flag.

The XRay Runtime Library is part of the compiler-rt project, which implementsthe runtime components that perform the patching and unpatching of insertedinstrumentation points. When you use clang to link your binaries and the-fxray-instrument flag, it will automatically link in the XRay runtime.

The default implementation of the XRay runtime will enable XRay instrumentationbefore main starts, which works for applications that have a shortlifetime. This implementation also records all function entry and exit eventswhich may result in a lot of records in the resulting trace.

Also by default the filename of the XRay trace is xray-log.XXXXXX where theXXXXXX part is randomly generated.

These options can be controlled through the XRAY_OPTIONS environmentvariable, where we list down the options and their defaults below.

• __xray_log_register_mode(…): Register a logging implementation againsta string Mode identifier. The implementation is an instance ofXRayLogImpl defined in xray/xray_log_interface.h.
• xray_log_select_mode(…): Select the mode to install, associated witha string Mode identifier. Only implementations registered withxray_log_register_mode(…) can be chosen with this function.
• __xray_log_init_mode(…): This function allows for initializing andre-initializing an installed logging implementation. Seexray/xray_log_interface.h for details, part of the XRay compiler-rtinstallation.

Once a logging implementation has been initialized, it can be “stopped” byfinalizing the implementation through the xray_log_finalize() function.The finalization routine is the opposite of the initialization. When finalized,an implementation’s data can be cleared out through thexray_log_flushLog() function. For implementations that support in-memoryprocessing, these should register an iterator function to provide access to thedata via the xray_log_set_buffer_iterator(…) which allows code callingthe xray_log_process_buffers(…) function to deal with the data inmemory.

All of this is better explained in the xray/xray_log_interface.h header.

Basic Mode

XRay supports a basic logging mode which will trace the application’sexecution, and periodically append to a single log. This mode can beinstalled/enabled by setting xray_mode=xray-basic in the XRAY_OPTIONSenvironment variable. Combined with patch_premain=true this can allow fortracing applications from start to end.

Like all the other modes installed through , theimplementation can be configured through the xray_log_init_mode(…)function, providing the mode string and the flag options. Basic-mode specificdefaults can be provided in the XRAY_BASIC_OPTIONS environment variable.

Flight Data Recorder Mode

XRay supports a logging mode which allows the application to only capture afixed amount of memory’s worth of events. Flight Data Recorder (FDR) mode worksvery much like a plane’s “black box” which keeps recording data to memory in afixed-size circular queue of buffers, and have the data availableprogrammatically until the buffers are finalized and flushed. To use FDR modeon your application, you may set the xray_mode variable to xray-fdr inthe XRAY_OPTIONS environment variable. Additional options to the FDR modeimplementation can be provided in the XRAY_FDR_OPTIONS environmentvariable. Programmatic configuration can be done by calling__xray_log_init_mode("xray-fdr", <configuration string>) once it has beenselected/installed.

When the buffers are flushed to disk, the result is a binary trace formatdescribed by XRay FDR format

When FDR mode is on, it will keep writing and recycling memory buffers untilthe logging implementation is finalized – at which point it can be flushed andre-initialised later. To do this programmatically, we follow the workflowprovided below:

The default settings for the FDR mode implementation will create logs namedsimilarly to the basic log implementation, but will have a different logformat. All the trace analysis tools (and the trace reading library) willsupport all versions of the FDR mode format as we add more functionality andrecord types in the future.

We currently have the beginnings of a trace analysis tool in LLVM, which can befound in the tools/llvm-xray directory. The llvm-xray tool currentlysupports the following subcommands:

• extract: Extract the instrumentation map from a binary, and return it asYAML.
• account: Performs basic function call accounting statistics with variousoptions for sorting, and output formats (supports CSV, YAML, andconsole-friendly TEXT).
• convert: Converts an XRay log file from one format to another. We canconvert from binary XRay traces (both basic and FDR mode) to YAML, friendly textformats, as well as Chrome Trace Viewer (catapult) formats.
• graph: Generates a DOT graph of the function call relationships betweenfunctions found in an XRay trace.
• stack: Reconstructs function call stacks from a timeline of functioncalls in an XRay trace.These subcommands use various library components found as part of the XRaylibraries, distributed with the LLVM distribution. These are:

• llvm/XRay/Trace.h : A trace reading library for conveniently loadingan XRay trace of supported forms, into a convenient in-memory representation.All the analysis tools that deal with traces use this implementation.
• llvm/XRay/Graph.h : A semi-generic graph type used by the graphsubcommand to conveniently represent a function call graph with statisticsassociated with edges and vertices.
• llvm/XRay/InstrumentationMap.h: A convenient tool for analyzing theinstrumentation map in XRay-instrumented object files and binaries. Theextract and stack subcommands uses this particular library.

There are a number of ongoing efforts for expanding the toolset building aroundthe XRay instrumentation system.

Trace Analysis Tools

• With a large instrumented binary, the size of generated XRay traces canquickly become unwieldy. We are working on integrating pruning techniques andheuristics for the analysis tools to sift through the traces and surface onlyrelevant information.

More Platforms

We’re looking forward to contributions to port XRay to more architectures andoperating systems.