Where a Java agent can complement eBPF

Recent advancements in observability provide instant access to encrypted traffic, using eBPF. This includes headers, errors, and even the plaintext form of the data. However, these neat techniques just don’t cut it when it comes to special, unique cases, the toughest being encrypted Java traffic. Why is Java different? And how can we still reach the new state-of-the-art observability standard that after getting used to, we just can’t live without?

What has eBPF brought to the table?

With the introduction of eBPF, observability went through a wild make-over. With the ability to install kprobes and uprobes, eBPF bears the news: making sense of your environment is a 5-minute jog, instead of a 5-month marathon. Installation is not only fast, but also secure, and poses minimal overhead, compared with old alternatives based on manual or automatic instrumentation.

The following example encapsulates the magic brought forth by eBPF, that allowed this revolution in observability to commence. Consider libssl, the most widely-used encryption library out there. When your application uses libssl, it essentially calls two functions: SSL_write and SSL_read. These functions encrypt and decrypt the traffic. With tools such as bcc, it has become rather easy to inspect this traffic with eBPF uprobes. The following piece of code prints any outgoing encrypted traffic, in its plaintext form:

This is very impressive, but not always usable for anyone willing to view the traffic, let alone the logic behind it, such as whether anything went wrong. To make a very long story short, this raw traffic needs to be categorized into the underlying protocol, which then needs to be parsed, and different messages should be matched together. Only then can the information be visualized. Luckily, tools such as Flora do all that, automatically and efficiently.

The problem with encrypted Java traffic

For those unfamiliar with Java, now is probably not the best time getting acquainted with it. For years it has been the go-to programming language for almost every challenge, and for good reasons. However, some of its features are relatively out-of-date. For instance, Java applications run on a virtual machine: a revolutionary idea at the time, that allowed the same compiled application to run on multiple platforms. However, in today’s containerized world in the cloud, it is pretty redundant, and only raises difficulties.

An application that runs on a VM is tricky to observe. eBPF is event-driven, meaning it attaches to functions. Like a debug breakpoint, only when a function starts running can we access the information stored in the arguments passed to it - just like in the SSL_write example above.

Speaking of the example above, we want to attach to the Java alternative to the SSL_write function. However, this function is deep inside the virtual machine. Since the functions run inside a VM, attaching eBPF probes to them is tricky: the VM doesn’t expose the existence of these functions in the manner needed for eBPF to attach to them. Technically, what eBPF requires is the addresses of the code of these functions in the process’s memory: but the VM may change their location on every run.

With eBPF, the best we can hope for is attaching to a VM function that starts running every application function, and within that context, diagnosing the running function to see if it’s one we actually want to trace. This solution would understandably incur major performance issues.

To put another nail in the coffin of that idea, there are multiple implementations of encryption in Java, all of them we wish to observe. Sometimes, libssl is used. In other times, Java’s Secure Socket Extension (JSSE) is used. Other applications use conscrypt, or boringssl. Some of these solutions are linux shared objects, such as libssl, and some of them are implemented directly in Java.

When eBPF is not enough

We resort to the second best candidate for the job: automatic instrumentation. Before eBPF emerged, this was the best way to observe application information. Like eBPF, automatic instrumentation is, well… automatic. This means that we can still achieve what we were aiming for in terms of a one-click, immediate installation. Thanks to that, the integration with eBPF is seamless.

However, instrumentation has many disadvantages when compared with eBPF, the most relevant one in this scenario is due to it being highly specific. As we’re about to see, the solution we explore is built precisely for the purpose of Java encrypted traffic. The different implementations of encryptions, some of them mentioned above, sometimes require an alteration. Different versions of java may require alterations as well.

The first step toward automatic instrumentation is finding the Java code we wish to instrument. In the case of libssl, these were the SSL_write and SSL_read functions. Java has two types of encrypted traffic: synchronous and asynchronous. The relevant java classes are SSLSocket and SSLEngine, respectively. These are object-oriented interfaces: they declare functions that all implementations have in common.

Synchronous traffic is handled by the SSLSocket object. Since java works closely with the concept of streams, the socket has two streams: for incoming and for outgoing data. Instrumenting the streams’ write and read functions yields the wanted result. 

Asynchronous traffic is handled by the SSLEngine object. The engine does not have the streams as intermediaries, and instead has two simple functions: wrap encrypts, and unwrap decrypts.

Some implementations don’t behave as nicely as simply implementing these functions, which is one of the reasons that the solution may need custom-tailoring.

While SSLEngine seems simpler to instrument than SSLSocket is, in the manner that there are no stream intermediaries, it is in practice more complex. Since the streams are not referenced by the engine, the identity of the peer is unknown from the scope of our code. The identities of both communication endpoints (IP addresses and TCP ports) are very important in making sense of the traffic, which is why further instrumentation is needed. Specifically, the streams are instrumented in an efficient way, which also allows matching the relevant streams to the engines that generated the traffic, creating a match between the metadata (including the endpoint identities) and the plaintext form of the data.

Instrumentation in Java

Java instrumentation is done by changing the Java bytecode. The concept is quite simple: we alter functions of our choosing, inserting code when they are executed, just like eBPF does. Since this is not eBPF, this may trigger potential harmful situations, such as performance and memory complications, for which reason we have to be very careful and perform meticulous tests.

The details are pretty complex, but using libraries such as bytebuddy, it becomes easy. The following example shows how to instrument the wrap function of every implementation of SSLEngine. Using this, we can insert any code we’d like whenever wrap is executed.

This technique can be used once we’re in the context of a Java agent inside Java applications. To reach this context, we need to:

1. Detect Java applications
2. Attach an agent to the application

Detection is done with ease from Flora. As an agent on the server that runs any application, Flora has access to both already-running applications and newly-created applications. 

Attachment is done using Java’s dynamic attachment mechanism, as used by jattach. While Java agents are usually attached statically (upon an application’s startup), in order to provide the instant, one-click installation that eBPF has gotten us used to, we use dynamic attachment.

Finally, the extracted information needs to pass back to Flora, in order to be seamlessly integrated with data from eBPF (such as encrypted data from other languages). The entire automatic instrumentation process is as follows:

eBPF + Java agent = instant, full coverage

It’s true, eBPF has many advantages over automatic instrumentation. A Java agent can never be as performant or as secure as the eBPF alternative does. Because of this, some unjustifiably believe that if your stack is not modern enough to be fully covered by eBPF, you can not reach the new standard of observability that eBPF has brought to the world.

But having an older stack doesn’t mean that you have to compromise. When done right, a conjunction of eBPF and a Java agent lives up to all expectations, while withstanding the harshest of performance and security specifications.

Monitor your Java applications with groundcover, to instantly and fully cover every type of plaintext and encrypted communication. Seamlessly integrate eBPF with a dynamic Java agent to reap the fruits of both worlds.