Falco – Syscalls

Blog: Tracing System Calls Using eBPF - Part 2

Fri, 06 Oct 2023 00:00:00 +0000

Introduction

In Tracing System Calls Using eBPF Part 1, we lay the groundwork, introducing you to the fundamentals of eBPF and its predecessor, BPF (Berkeley Packet Filter). We delve into the evolution of this technology, its safety, performance, and observability advantages over traditional kernel modules, and its pivotal role in securing modern cloud native environments. We guide you through the intricate process of working with eBPF programs, from compilation to execution, highlighting its power in tracing system calls.

In the second installment, Tracing System Calls Using eBPF Part 2, we elevate our understanding of eBPF's capabilities. We unravel the world of Uprobes and Uretprobes, demonstrating how these features empower developers to instrument and monitor user-space applications seamlessly. We then venture into Kprobes and Kretprobes, unlocking the potential to dynamically trace and debug kernel functions, offering insights into system behavior and performance analysis.

Uprobes

Uprobes, short for user probes, are a feature in the Linux kernel that enables developers to instrument and monitor user-space applications without modifying their code directly. They allow for the insertion of breakpoints at specific points of interest within an application, facilitating the collection of data, tracing of function calls, debugging, and performance analysis.

Uretprobes

Uretprobes, short for User Return Probes, are a feature in the Linux kernel that allows developers to trace and monitor the return paths of functions in user-space applications. While uprobes are used to instrument and intercept the entry points of functions, URETprobes specifically focus on the exit points or return paths. They enable developers to set up probes that are triggered when a specific function returns to its caller.

Here is an eBPF program that uses user probes to trace the printf function present in glibc (the standard GNU C Library).

In accordance with the instructions outlined in our Tracing System Calls Using eBPF Part 1 blog, we can create a loader to effectively load this eBPF program and read the logs from the file /sys/kernel/tracing/trace_pipe .

Kprobes

Kprobes, short for Kernel Probes, are a feature in the Linux kernel that allow dynamic tracing and debugging of kernel functions. They are particularly useful for tasks like performance analysis, bug diagnosis, and system monitoring. They provide a non-intrusive way to gather runtime information from the kernel without requiring modifications to the kernel code itself. Additionally, they can be used to trace specific function calls, track parameters and return values, and gather statistical data on function execution

Kretprobes

Kretprobes, short for Kernel Return Probes, are a feature in the Linux kernel that complements Kprobes by allowing dynamic tracing and debugging of kernel function return points. While Kprobes focus on probing the entry points of kernel functions, kretprobes specifically target the return points of these functions.Similar to Kprobes, kretprobes work by inserting a probe handler function that gets executed when a specific kernel function is about to return. This allows developers and system administrators to gather information, modify return values, or perform additional actions at the point of function return.

Here is an eBPF program that uses kernel probes to trace a kernel function named prepare_kernel_cred. This function is used to create a new struct cred object that represents the credentials or privileges associated with a kernel task. It is commonly used in privilege escalation exploits for gaining root access. By tracing this function, we can identify all processes that invoke it, providing valuable insight for analyzing potential malicious activity.

SEC(“kprobe/prepare_kernel_cred”) indicates that an eBPF program is associated with the kprobe event for the "prepare_kernel_cred" kernel function. This event allows dynamic tracing and debugging by intercepting the entry point of the function.

struct pt_regs is a data structure that provides access to the register state of the program when it is executed. It contains information about the CPU registers at the time of the eBPF program invocation. It is defined as :

To facilitate the loading of the aforementioned eBPF program, we’ll use the following program.

Here is a Makefile for compiling the eBPF program and the loader

Conclusion

In this two-part exploration of Tracing System Calls Using eBPF, we've embarked on a fascinating journey through the inner workings of this powerful technology. Part 1 laid the foundation by introducing us to the fundamentals of eBPF and its predecessor, BPF, shedding light on their evolution and significance in modern cloud native environments. We uncovered how eBPF's safety, performance, and observability advantages empower us to trace system calls with unmatched efficiency.

In Part 2, we took our understanding to new heights. We delved into the world of Uprobes and Uretprobes, showcasing how they enable seamless instrumentation and monitoring of user-space applications. We then ventured into Kprobes and Kretprobes, unlocking the ability to dynamically trace and debug kernel functions. Armed with these advanced techniques, we gained valuable insights into system behavior, performance analysis, and even the detection of potential malicious activity.

As we conclude this journey into the heart of eBPF, we stand equipped with a powerful set of tools and knowledge. Whether you're a seasoned sysadmin, a curious developer, or a vigilant security enthusiast, the capabilities of eBPF open new doors to real-time monitoring and analysis.

Stay tuned for further insights and practical guidance in the world of eBPF, where innovation meets security, and the future of system monitoring becomes a reality.

Blog: Tracing System Calls Using eBPF - Part 1

Mon, 11 Sep 2023 00:00:00 +0000

Introduction:

In this article, we will delve into the details of eBPF (extended Berkeley Packet Filter) and explore its significance in tracing system calls. This particular blog will be in two parts; in the first blog, we will discuss eBPF, and in the subsequent section, we will delve into probes. eBPF is a powerful technology that allows for the dynamic and efficient tracing of events within the kernel space of an operating system. You have probably heard of the acronyms BPF and eBPF being used interchangeably. That's why we will aim to address both BPF and eBPF before discussing how and why Falco uses this technology.

BPF (Berkeley Packet Filter)

BPF is a technology used for network packet filtering and analysis. It is a powerful tool for implementing network security features, such as firewalls and intrusion detection systems. It can also be used to examine network traffic in real-time, detect suspicious patterns, and take appropriate actions to protect the network.

eBPF (Extended Berkeley Packet Filter)

The Extended Berkeley Filter (eBPF) is an evolution of the original BPF technology. It extends the capabilities of BPF by providing a more powerful and flexible way to perform dynamic tracing, network analysis, and performance monitoring. It allows developers to write and load programs into the kernel which can be attached to various hooks and events in the system. These programs can provide real-time insights and control over system activities.

Working on an eBPF program

The process of compiling and running an eBPF program involves several steps:

The eBPF program is converted into bytecode by using a compiler, ready to be loaded by a loader program.
The eBPF verifier checks the program for safety, correctness and adherence to specific rules and constraints. First of all, it performs a depth-first search on all possible execution paths to ensure that the program always proceeds to completion. Next, it performs a static analysis on the bytecode and ensures that the program doesn't violate memory access rules, and doesn't cause instability.
Once the eBPF program passes the verification process, it can be loaded into the kernel. The loader ensures that the program is securely loaded and attached to the desired hooks or targets in the system.
At runtime, the eBPF bytecode is further optimised through JIT (Just-in-time) compilation. This step converts the eBPF bytecode into machine code that can be executed by the CPU.

Kernel Modules

Apart from eBPF, the other approach we previously discussed for process tracing in Linux is the use of kernel modules. Kernel modules allow developers to write custom code that can be loaded into the kernel to extend its functionality.

By leveraging kernel modules, it is possible to hook into various points of the kernel's process management code and capture detailed information about process execution. This includes events such as process creation, termination, and context switches.

By accessing the kernel's internal data structures and functions, the module can gather valuable insights such as process IDs, parent-child relationships, execution time, system calls, and more.

So why does Falco use eBPF?

The integration of eBPF brings significant advantages to projects like Falco, empowering them to securely and efficiently monitor and analyze system calls in real-time. You might be wondering why eBPF is necessary when Falco already has real-time detection capabilities through its kprobe (kernel probe) that handles syscall events.

One compelling reason for incorporating eBPF support is to enable Falco to seamlessly operate in modern cloud native environments, where the traditional kernel probe may encounter limitations or face restrictions imposed by the control plane nodes.

By embracing eBPF, Falco ensures the continuity of its real-time detection capabilities in a secure manner, allowing for the prompt and accurate identification of security incidents, regardless of the underlying environment.

Later in the article, we will delve into the various considerations surrounding the adoption of an eBPF probe for Falco, providing valuable insights for determining when it becomes advantageous to leverage this functionality.

eBPF programs vs kernel modules

Safety and Isolation

eBPF programs are subjected to a thorough verification process before they are loaded into the kernel. This step provides an extra layer of protection and helps prevent security vulnerabilities. In contrast, kernel modules have direct access to the kernel code, which can pose a threat to the system if not implemented correctly.

Performance

eBPF programs are JIT compiled into machine code, which significantly improves the performance. JIT compilation optimizes the program for the specific CPU architecture, enabling efficient execution. Despite all these efforts, an eBPF instrumentation will always cause a greater overhead in the system than a kernel module one, the reason is that in the kernel module instrumentation there are no calls to the BPF subsystem.

Observability and Debugging

eBPF provides powerful tracing and observability capabilities. eBPF programs can be attached to various events, such as network packets, system calls, or kernel functions, allowing detailed visibility into the system behaviour. This makes eBPF a valuable tool for debugging, performance analysis, and security monitoring. Kernel modules typically require more invasive and complex mechanisms for achieving similar observability.

Attaching eBPF programs to hooks and events

There are various instrumentation points defined in the Linux kernel. An instrumentation is a specific point in a computer program where additional code, known as instrumentation code, is inserted to gather information about the program's execution. Instrumentation code can be injected at runtime using JIT compilation. Kernel probes, tracepoints, user-space probes, kretprobes are examples of instrumentation points.

Here is an eBPF program that runs when the execve system call is made.

In the eBPF programming context, the macro SEC() from the bpf/bpf_helper.h header file plays a crucial role. It allows the programmer to specify the section in which a function or variable will be placed within the eBPF object file. This becomes essential when loading eBPF programs into the kernel using mechanisms like the bpf() system call.
By organizing functions and variables into named sections, the eBPF loader can efficiently locate and load the required code and data. Specifically, when dealing with tracepoint events, the SEC format follows the pattern SEC("tp/<category>/<name>"), where <category> and <name> represent the respective tracepoint category and event name.
tp/syscalls/sys_enter_execve refers to a tracepoint that records when a process spawns the execve system call.
A list of all the available tracepoints is present in the /sys/kernel/debug/tracing/available_events file. The format for each line in the file is <category>:<name>. For example, syscalls:sys_enter_execve.

Before compiling the program, we need to do some basic configuration:

Let's compile the program. The following command can be used to do this task:

Now, we need to write a loader program that loads and attaches this program. This loader program is used to load and attach an eBPF program to the Linux kernel. It opens and loads the eBPF object file, checks for errors during the process, finds a specific eBPF program within the loaded object, and attaches it to the kernel. Once attached, the eBPF program will be executed when certain events occur. The program enters an infinite loop at the end, indicating that it will continue running until it is manually terminated.

Let's compile and run this program

To get the logs generated by the function bpf_printk, we can read the file: /sys/kernel/tracing/trace_pipe

Manually reading messages from the tracepipe doesn't seem to be an efficient approach. It would be advantageous to establish a mechanism for the eBPF program to send messages to the loader program. One viable solution is to utilize ring buffers. Let’s review more details about ring buffers.

Ring buffers

eBPF ring buffer, also known as bpf_ringbuf, is a mechanism provided by the Linux kernel for efficient communication between eBPF programs and user-space programs.

It allows the exchange of data and events between eBPF programs running in the kernel and user-space applications. It is a MPSC (Multi Producer Single Consumer) queue and can be safely shared across multiple CPUs simultaneously.

The eBPF ring buffer, being shared across all CPUs, offers a unified and efficient solution for managing memory utilisation, mitigating issues of overuse or under-allocation that commonly occur with perfbuf.

Let's have a look at a few functions that we'll be using to write an eBPF program that sends data to userspace.

bpf_ringbuf_reserve

This function is used to reserve size bytes of space in a BPF ring buffer.

bpf_probe_read_user_str

This function is used to read a null terminated string from user-space memory into the destination dst. The dst parameter is a pointer to the destination buffer in the kernel space. unsafe_ptr is a pointer to the source string in the user-space.

bpf_ringbuf_submit

This function is used to submit data that had previously been reserved in a ringbuf.

bpf_object__find_map_fd_by_name

This function is used to find the file descriptor of a named map.

bpf_program__attach_tracepoint

This function is used to attach an eBPF program to a kernel tracepoint.

ring_buffer__new

This function is used for creating and opening a new ringbuf manager.

ring_buf__consume

Used to remove or consume data from a ring buffer.

BTF (BPF Type Format)

It provides a way to describe the types of data structures used by eBPF programs, allowing for improved type safety, debugging, and introspection.

Now, let's write a program that sends data to userspace using ringbuf.

Having created the program, we can write a loader to load this eBPF program.

The infinite loop is necessary to ensure that the program continuously checks for new events in the ring buffer. Without the loop, the program would only consume events that were already in the buffer at the time of the initial ring_buffer__consume() call. By looping and calling ring_buffer__consume() repeatedly, the program can retrieve events as soon as they become available and process them in real-time. The sleep(1) call within the loop serves to reduce the CPU usage of the program by introducing a one-second delay between each call to ring_buffer__consume().

Great, we were able to recover the process name as well as the PID!

Conclusion

In conclusion, this article has provided a comprehensive overview of eBPF (extended Berkeley Packet Filter) and its significance in tracing system calls. We have explored the evolution from BPF to eBPF, discussed why Falco uses this technology, and delved into the process of working with eBPF programs and ring buffers for efficient data communication between the kernel and user-space applications.

As we journeyed through the capabilities of eBPF in this first part, we uncovered its benefits in terms of safety, performance, and observability when compared to traditional kernel modules. eBPF empowers us to securely and efficiently monitor and analyze system calls in real-time, making it a valuable tool in modern cloud native environments.

In the upcoming second part of this blog series, we will further expand our exploration by delving into the realm of probes and additional advanced topics. We will dive deeper into how eBPF probes can be leveraged for enhanced system tracing, performance analysis, and security monitoring. Stay tuned for more insights and practical guidance on harnessing the power of eBPF.

Keep an eye out for Part 2, where we'll continue our journey into the world of eBPF and system call tracing.