Hello people!

This is it: the day you’ve been waiting for has finally arrived.

Today we’re releasing Babeltrace 2.0 “Amqui”! 🥳

Amqui (/ɒmkwiː/) is a town in eastern Québec, Canada, at the base of the Gaspé peninsula in Bas-Saint-Laurent.

Located at the confluence of the Humqui and Matapédia Rivers, its proximity to woodlands makes it a great destination for outdoor activities such as camping, hiking, and mountain biking.

By the way, Babeltrace has a new dedicated domain name: babeltrace.org.

Background

Just a few words about why Babeltrace 2 took about five years to complete.

First, let us clarify that from day one, one of the most important imperatives of the project was long-term stability: we wanted Babeltrace 2 to last for a long time and for its library’s API to become a well-known reference in the trace processing territory, a bit like Libxml2 has had major version 2 for about 20 years. All the other Babeltrace 2 parts depend on the library’s C API: the command-line interface (CLI), the plugins, and the Python bindings.

So, as we needed to modify the library, CLI, and plugins to adapt to new requirements during the development phase, we postponed the release so as not to have to release Babeltrace 3 a year after and Babeltrace 4 another year after.

Second, the scope and requirements of the project evolved. Because there were at most two developers working on Babeltrace 2 at the same time during its first years, the project took enough time to develop for new ideas to emerge and translate into changes, sometimes significant, to the project’s modules.

For example, the very first requirement was for the libbabeltrace2 API to use the same objects as Babeltrace 1’s CTF writer API. We put a lot of effort backfitting new concepts into the CTF writer API and making sure not to break existing user code. But in the end, we realized that this condition was too limiting and that we had to split the APIs completely to have the freedom to introduce all the features we wanted.

When the API finally met our needs feature-wise in 2018, we measured and found that a simple CTF-to-text conversion task with Babeltrace 2 was approximately 23 times slower than the equivalent with Babeltrace 1! Hence we entered an optimization phase in which we had to redefine core concepts of the library (for example, introduce unique objects to create object pools) so that, today, Babeltrace 2’s performance is comparable to Babeltrace 1’s.

Revamping the C API to optimize the library means we had to readapt the CLI, plugins, and Python bindings. We spent the beginning of 2019 synchronizing the project’s parts, but also improving and uplifting the test suite and fixing a lot of lingering bugs.

The last few months were invested in writing a complete C API documentation and in more bug fixes, tests, and build system improvements.

Today ends this journey to Babeltrace 2.0. It’s been a long one, but we made it. All the next releases of Babeltrace 2 will be minor ones for the foreseeable future, never breaking the library’s API.

Babeltrace 2 is:

  • 120,000 lines of C.

  • 7000 lines of Python.

  • 21,000 tests.

  • 30,000 lines of C API documentation.

  • 24 manual pages

We hope you enjoy using our software as much as we enjoyed making and refining it over the last years.

What’s new since Babeltrace 1?

The Babeltrace 2 project is completely independent from Babeltrace 1. In fact, you can install both on the same system as they share no common files. We only made sure to keep the babeltrace2 CLI tool backward compatible with the original babeltrace tool.

See babeltrace2-intro(7) to learn more about Babeltrace 2.

General

  • Full plugin support: any user can distribute a Babeltrace 2 plugin and, as long as libbabeltrace2 finds it, any application linked to libbabeltrace2 can load it and use it.

    Plugins are not just trace format encoders and decoders: they package source, filter, and sink component classes so that you can connect specialized, reusable components together in a trace processing graph to create a customized trace conversion or analysis device.

    This modular strategy is much like how the FFmpeg, GStreamer, and DirectShow projects approach media stream processing.

  • All the parts of the Babeltrace 2 project run on the major operating systems, including Linux, macOS, and Windows.

  • Some component classes, such as sink.text.pretty (similar to the text output format of the babeltrace CLI) and sink.text.details, can write color codes to the standard output when it’s connected to a color-enabled terminal.

    The Babeltrace 2 log, printed to the standard output, can also be colorized.

Command-line interface

  • Whereas you can convert traces from one format to another with Babeltrace 1’s CLI tool, babeltrace, you can also execute a custom trace manipulation task with babeltrace2 thanks to its run command.

  • The convert command features an automatic source component discovery algorithm to find the best suited components to create for a given non-option argument (file or directory path, or custom string like an LTTng live URL).

    For example:

    $ babeltrace2 /path/to/ctf/trace
    $ babeltrace2 net://localhost/host/myhost/my-session
CTF input/output

  • The source.ctf.fs component class, which is more or less the equivalent of Babeltrace 1’s ctf input format, has features not found in Babeltrace 1:

    • The component handles many trace quirks which are the results of known tracer bugs (which are fixed today) and corner cases in LTTng-UST, LTTng-modules, and barectf, making it possible to decode malformed packets.

    • The component merges CTF traces sharing the same UUID into a single, logical trace.

      This feature supports LTTng 2.11’s tracing session rotation trace chunks.

  • With a sink.ctf.fs component, you can create CTF traces on the file system.

    With the babeltrace2 tool, you can use the --output-format=ctf and --output options to create an implicit sink.ctf.fs component.

    For example:

    $ babeltrace2 /path/to/input/trace --output-format=ctf --output=trace-dir
LTTng live input

Babeltrace 1’s babeltrace command exits successfully when it cannot find an LTTng live (--input-format=lttng-live option) tracing session.

The session-not-found-action initialization parameter controls what a source.ctf.lttng-live message iterator does when it cannot find the remote tracing session:

  • If the action is end, the message iterator does like babeltrace and simply ends successfully.

  • If the action is continue (the default), the message iterator never ends: it keeps on trying until the tracing session exists, indeed “subscribing” to the session.

Library

  • libbabeltrace2 shares nothing with libbabeltrace.

    The Babeltrace 2 library C API has features such as:

    • A single header file.

    • Function precondition and postcondition checking.

    • Object-oriented model with shared and unique objects.

    • Strict C typing and const correctness.

    • User-extensible classes.

    • Rich, thread-safe error reporting.

    • Per-component and per-subsystem logging levels.

    • Trace intermediate representation (IR) objects to make the API trace-format-agnostic.

    • A versioned protocol for the exchange of messages between components to enable forward and backward compatibility.

  • You can build the library in developer mode to enable an extensive set of function precondition and postcondition checks.

    The developer mode can help detect programming errors early when you develop a Babeltrace 2 plugin or an application using libbabeltrace2.

    See the project’s README for build-time requirements and detailed build instructions.

Python 3 bindings

  • The new bt2 package offers all the functionality of libbabeltrace2 and even more through a brand new, Pythonic programming interface.

  • With the bt2 package, you can load existing plugins and create a trace processing graph, but also create Babeltrace 2 plugins.

    libbabeltrace2 can load both shared object and Python plugins.

  • What has the const qualifier in libbabeltrace2 has a dedicated constant type in the bt2 package so that methods to modify it simply do not exist. This can help catch programming errors.

    For example, when you create an event message, you get a bt2._EventMessage object. When you consume an event message, you get an immutable bt2._EventMessageConst object. Both objects wrap the same C bt_event * (and const bt_event *) object.

What’s new since Babeltrace 2.0.0-rc4?

Improvements

General

  • Hide private symbols.

  • Add a mechanism to add extra information (name, description, and names of applied patches) to a custom build’s version.

    See 1f316595.

Command-line interface

Print full version, including the release’s name and name description, the Git revision’s description, and the extra information, with the --version option.

See 72fa419d.

Documentation

Write a comprehensive Babeltrace 2 library C API documentation.

See 7704a0af.

Library

  • Remove the bt_packet_context_field API (not needed anymore).

    See 9b98c3bf.

  • Standardize variant field option function names.

    See 7e0b3e01.

  • Graph API: remove “ports connected” listeners (not needed anymore).

    See d9d764cc.

  • Graph API: remove the “listener removed” callbacks (not needed anymore).

    See fecbdb3e.

  • Remove bt_graph_interrupt(); use the new bt_graph_borrow_default_interrupter() function instead.

    See f09b3ae9.

  • Remove bt_query_executor_interrupt(); use the new bt_query_executor_borrow_default_interrupter() function instead.

    See d9a20b20.

  • Add the message iterator class concept.

    Instead of having an implicit message iterator class within source and filter component classes, make its creation explicit and pass it to bt_component_class_source_create() and bt_component_class_filter_create().

    See 68e2deed, 8e1fe2a4, and f615b250.

  • Rename BT_PLUGIN_INITIALIZE() to BT_PLUGIN_INITIALIZE_FUNC() and BT_PLUGIN_FINALIZE() to BT_PLUGIN_FINALIZE_FUNC().

    See 1906e012.

  • For many functions of the graph and component APIs, return a borrowed reference of an added object instead of a new reference.

    See 246d5223.

  • Rename bt_message_message_iterator_inactivity_borrow_default_clock_snapshot_const() to bt_message_message_iterator_inactivity_borrow_clock_snapshot_const().

    See 62988c56.

  • Rename bt_version_get_extra() to bt_version_get_development_stage().

    See 19e20ba9.

  • Add the following functions to get more information about the library’s version:

    • bt_version_get_vcs_revision_description()

    • bt_version_get_name()

    • bt_version_get_name_description()

    • bt_version_get_extra_name()

    • bt_version_get_extra_description()

    • bt_version_get_extra_patch_names()

    See 2491e1b5, 58df4c80, and a0446fa8.

Python bindings

  • Adapt to the library changes above.

  • Make bt2.Graph.add_*_listener() return None.

    See d9d764cc.

  • Reverse the order of the printed causes in bt2._Error.__str__() to match what the CLI does.

    See 701698c8.

source.ctf.fs component class

Sort the paths of the data stream files to handle so that the output port order is predictable (for debugging purposes).

See 18118efb.

Notable bug fixes

Command-line interface

Print error causes in all error paths.

See f2ff3e07.

Python bindings

  • Fix shared reference issues in listener handles.

    See d4024c86.

  • Don’t print previous error causes in causes created from bt2._Error.

    See b53cd768.

What’s next?

The current road map includes:

  • Make it easier to create an efficient filter message iterator which needs to modify or augment event messages.

    Currently, if you need to modify an event message, considering that you cannot change an event class once you use it, you need to create a new event class, which means you also need to create a new stream class and a new trace class. You also need to copy all the metadata and data objects you don’t alter. This is what filter.lttng-utils.debug-info does.

    We do realize that trace-transforming filter message iterators represent an important Babeltrace 2 use case. While it’s possible to write one for Babeltrace 2.0, it is cumbersome to do and the result is not as performant as we’d like. We decided to release Babeltrace 2 now and reserve improvements to this part of the library for a future minor release.

  • Add a filter component class to drop messages (like events) based on a filter expression, similar to the --filter option of the lttng command.

  • Make the ctf plugin’s component classes support CTF 2, the upcoming major revision of the Common Trace Format.

  • Optimize specific project component classes, especially source.ctf.fs, source.ctf.lttng-live, and filter.utils.muxer.

    For source.ctf.fs, implement the “seek ns from origin” method using the packet indexes to make a message iterator seek a specific time faster.

Important links