Falco – EBPF

Blog: Deploy Falco on a Talos cluster

Mon, 22 Jul 2024 00:00:00 +0000

Talos Linux is an OS designed for Kubernetes, with in mind to be secure, immutable and minimal. It offers a solution for having secure nodes for your Kubernetes cluster. Running Falco on them requires some configurations we'll see in this blog post. The good news is everything is available to collect the syscalls with eBPF and also the audit logs from the Kubernetes control plane.

In this tutorial we'll use a local Talos cluster created with Docker containers for convenience, adapt the configurations to your own context.

Requirements

For this tutorial, you'll need several tools installed:

Set up the Talos cluster

We'll start with a 2 workers cluster:

talosctl cluster create --workers 2 --wait-timeout 5m

After a few minutes, your containers and so your cluster should be up and running. You can check the status with:

talosctl cluster show --nodes 10.5.0.2,10.5.0.3,10.5.0.4

Output:

PROVISIONER docker
NAME talos-default
NETWORK NAME talos-default
NETWORK CIDR 10.5.0.0/24
NETWORK GATEWAY
NETWORK MTU 1500

NODES:

NAME TYPE IP CPU RAM DISK
talos-default-controlplane-1 controlplane 10.5.0.2 - - -
talos-default-worker-1 worker 10.5.0.3 - - -
talos-default-worker-2 worker 10.5.0.4 - - -

Get the kubeconfig

The talosctl CLI allows to easily set up your kubeconfig file for managing the apps in your fresh new cluster:

talosctl kubeconfig -n 10.5.0.2 -f

Check you have access to the cluster:

kubectl cluster-info

Output:

Kubernetes control plane is running at https://10.5.0.2:6443
CoreDNS is running at https://10.5.0.2:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Patch the cluster

When you deploy Falco with Helm in a Kubernetes cluster, an initContainer is bootstrapped to inject the eBPF probe into the kernel of each node. This behavior requires some privileges but Talos, designed to be secured, doesn't allow that by default. It's possible anyway by patching the nodes.

Create this patch.yaml file:

cluster:
 apiServer:
 admissionControl:
 - name: PodSecurity
 configuration:
 exemptions:
 namespaces:
 - falco

As you can see, we allow the pods in the namespace falco to use PodSecurity settings.

And now patch the cluster:

talosctl patch machineconfig --patch @patch.yaml --nodes 10.5.0.2

Output:

patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.2
Applied configuration without a reboot

Install Falco

We'll use Helm to deploy Falco.

Install the Helm registry for the Falco chart:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Create the values.yaml file:

cat << EOF > values.yaml
driver:
 kind: modern_ebpf
tty: true

falcosidekick:
 enabled: true
 replicaCount: 1
 webui:
 enabled: true
 replicaCount: 1
 redis:
 storageEnabled: false
 service:
 type: NodePort
 port: 2802
 targetPort: 2802
 nodePort: 30128

falcoctl:
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 artifact:
 install:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]
 follow:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]

services:
 - name: k8saudit-webhook
 type: ClusterIP
 ports:
 - port: 9765
 targetPort: 9765
 protocol: TCP
 name: http

falco:
 rules_files:
 - /etc/falco/falco_rules.yaml
 - /etc/falco/falco-incubating_rules.yaml
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit
 library_path: libk8saudit.so
 init_config: ""
 open_params: "http://:9765/k8s-audit"
 - name: json
 library_path: libjson.so
 init_config: ""
 load_plugins: [k8saudit, json]
EOF

Deploy Falco:

helm upgrade -i falco falcosecurity/falco -n falco --create-namespace -f values.yaml

Follow the deployment:

kubectl get pods -w -n falco

Before moving on, let's take time to explain why some of these values.

 driver:
 kind: modern_ebpf
 tty: true

We use the modern_epbf probe to collec the syscall events.
tty: true allows to get the alerts in the stdout immediatly, without any buffering.

 falcosidekick:
 enabled: true
 replicaCount: 1
 webui:
 enabled: true
 replicaCount: 1
 redis:
 storageEnabled: false
 service:
 type: NodePort
 port: 2802
 targetPort: 2802
 nodePort: 30128

We install Falcosidekick and its UI. All settings for the forwarding of the events between Falco and Falcosidekick are managed by the Helm chart.
As it's local cluster, we set the replicaCounts to 1, it loses the HA but save resources.
The UI will be exposed directly by the nodes on the port 30128, very convenient for a local cluster, prefer an ingress or just a port-forward for production.

 falcoctl:
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 artifact:
 install:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]
 follow:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]

falcoctl controls which plugins and rules to install and follow.
We install the stable and incubating rules for Falco
We install and follow the rules for the Kubernetes audit logs, the relevant plugins k8saudit and json will be automatically installed by falcoctl.

 services:
 - name: k8saudit-webhook
 type: ClusterIP
 ports:
 - port: 9765
 targetPort: 9765
 protocol: TCP
 name: http

The k8saudit plugin requires to create a Service listen the incoming events from the control plane.

 falco:
 rules_files:
 - /etc/falco/falco_rules.yaml
 - /etc/falco/falco-incubating_rules.yaml
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit
 library_path: libk8saudit.so
 init_config: ""
 open_params: "http://:9765/k8s-audit"
 - name: json
 library_path: libjson.so
 init_config: ""
 load_plugins: [k8saudit, json]

We load the rules for the syscalls and for the audit logs.
We load the plugins and their config. The k8saudit plugin will listen on the same port than configured in the services section.

Forward the audit logs to Falco

In a classic context, the control plane is configured to send its audit logs to an endpoint, like the k8saudit plugin. With Talos, it's not yet possible to configure this endpoint, but fortunately for us, these audit logs are written as files in the /var/log/audit/kube/ folder of the master nodes.

We'll use Fluent Bit to parse these files and forward them to the k8saudit plugin.

Install the Helm registry for the Fluent Bit chart:

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

Create the values.yaml file:

cat << EOF > values.yaml
podAnnotations:
 fluentbit.io/exclude: 'true'

daemonSetVolumes:
 - name: varlog
 hostPath:
 path: /var/log

daemonSetVolumeMounts:
 - name: varlog
 mountPath: /var/log

tolerations:
 - operator: Exists
 effect: NoSchedule

nodeSelector:
 node-role.kubernetes.io/control-plane: ""

config:
 service: |
 [SERVICE]
 Flush 5
 Daemon Off
 Log_Level warn
 HTTP_Server On
 HTTP_Listen 0.0.0.0
 HTTP_Port 2020
 Health_Check On
 Parsers_File /fluent-bit/etc/parsers.conf
 Parsers_File /fluent-bit/etc/conf/custom_parsers.conf

 inputs: |
 [INPUT]
 Name tail
 Alias audit
 Path /var/log/audit/kube/*.log
 Parser audit
 Tag audit.*
 Ignore_older true

 customParsers: |
 [PARSER]
 Name audit
 Format json
 Time_Key requestReceivedTimestamp
 Time_Format %Y-%m-%dT%H:%M:%S.%L%z

 outputs: |
 [OUTPUT]
 Name http
 Alias http
 Match *
 Host falco-k8saudit-webhook.falco.svc.cluster.local
 Port 9765
 URI /k8s-audit
 Format json
EOF

Deploy Fluent Bit:

helm upgrade -i fluent-bit fluent/fluent-bit -n kube-system -f values.yaml

To be allowed to mount the folder with the logs, we install Fluent Bit in the namespace kube-system.

Follow the deployment:

kubectl get pods -n kube-system -w -l app.kubernetes.io/name=fluent-bit

Some explanations of the values.yaml.

daemonSetVolumes:
 - name: varlog
 hostPath:
 path: /var/log

daemonSetVolumeMounts:
 - name: varlog
 mountPath: /var/log

The host folder with the logs is mounted inside the Fluent Bit pod.

tolerations:
 - operator: Exists
 effect: NoSchedule

nodeSelector:
 node-role.kubernetes.io/control-plane: ""

These settings are there to deploy Fluent Bit on the master nodes only.

config:
 inputs: |
 [INPUT]
 Name tail
 Alias audit
 Path /var/log/audit/kube/*.log
 Parser audit
 Tag audit.*
 Ignore_older true

Fluent Bit will parse the files *.logs from the folder /var/log/audit/kube/.

config:
 outputs: |
 [OUTPUT]
 Name http
 Alias http
 Match *
 Host falco-k8saudit-webhook.falco.svc.cluster.local
 Port 9765
 URI /k8s-audit
 Format json

The logs are forwarded to the endpoint falco-k8saudit-webhook.falco.svc.cluster.local:9765/k8s-audit, which is listened by the k8saudit plugin.

Visalize the alerts

Everything should be set up and running from now. You can access to the Falcosidekick-UI by the URL http://10.5.0.2:30128.

The default credentials are admin/admin.

Conclusion

Talos Linux is a more and more famous solution for creating resilient and secure Kubernetes clusters, but the trust doesn't exclude control. Mixing Talos and Falco makes you gain a step upper in term of security for your applications. Thanks to our modern eBPF probe and our k8saudit plugin, you can see how easy and quick it is to install Falco in Talos and start to observe what's happening.

Thanks to Quentin Joly for his blog post about Talos which helped me a lot to write this one.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falcosidekick UI project on GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Tracing System Calls Using eBPF - Part 2

Fri, 06 Oct 2023 00:00:00 +0000

Introduction

In Tracing System Calls Using eBPF Part 1, we lay the groundwork, introducing you to the fundamentals of eBPF and its predecessor, BPF (Berkeley Packet Filter). We delve into the evolution of this technology, its safety, performance, and observability advantages over traditional kernel modules, and its pivotal role in securing modern cloud native environments. We guide you through the intricate process of working with eBPF programs, from compilation to execution, highlighting its power in tracing system calls.

In the second installment, Tracing System Calls Using eBPF Part 2, we elevate our understanding of eBPF's capabilities. We unravel the world of Uprobes and Uretprobes, demonstrating how these features empower developers to instrument and monitor user-space applications seamlessly. We then venture into Kprobes and Kretprobes, unlocking the potential to dynamically trace and debug kernel functions, offering insights into system behavior and performance analysis.

Uprobes

Uprobes, short for user probes, are a feature in the Linux kernel that enables developers to instrument and monitor user-space applications without modifying their code directly. They allow for the insertion of breakpoints at specific points of interest within an application, facilitating the collection of data, tracing of function calls, debugging, and performance analysis.

Uretprobes

Uretprobes, short for User Return Probes, are a feature in the Linux kernel that allows developers to trace and monitor the return paths of functions in user-space applications. While uprobes are used to instrument and intercept the entry points of functions, URETprobes specifically focus on the exit points or return paths. They enable developers to set up probes that are triggered when a specific function returns to its caller.

Here is an eBPF program that uses user probes to trace the printf function present in glibc (the standard GNU C Library).

In accordance with the instructions outlined in our Tracing System Calls Using eBPF Part 1 blog, we can create a loader to effectively load this eBPF program and read the logs from the file /sys/kernel/tracing/trace_pipe .

Kprobes

Kprobes, short for Kernel Probes, are a feature in the Linux kernel that allow dynamic tracing and debugging of kernel functions. They are particularly useful for tasks like performance analysis, bug diagnosis, and system monitoring. They provide a non-intrusive way to gather runtime information from the kernel without requiring modifications to the kernel code itself. Additionally, they can be used to trace specific function calls, track parameters and return values, and gather statistical data on function execution

Kretprobes

Kretprobes, short for Kernel Return Probes, are a feature in the Linux kernel that complements Kprobes by allowing dynamic tracing and debugging of kernel function return points. While Kprobes focus on probing the entry points of kernel functions, kretprobes specifically target the return points of these functions.Similar to Kprobes, kretprobes work by inserting a probe handler function that gets executed when a specific kernel function is about to return. This allows developers and system administrators to gather information, modify return values, or perform additional actions at the point of function return.

Here is an eBPF program that uses kernel probes to trace a kernel function named prepare_kernel_cred. This function is used to create a new struct cred object that represents the credentials or privileges associated with a kernel task. It is commonly used in privilege escalation exploits for gaining root access. By tracing this function, we can identify all processes that invoke it, providing valuable insight for analyzing potential malicious activity.

SEC(“kprobe/prepare_kernel_cred”) indicates that an eBPF program is associated with the kprobe event for the "prepare_kernel_cred" kernel function. This event allows dynamic tracing and debugging by intercepting the entry point of the function.

struct pt_regs is a data structure that provides access to the register state of the program when it is executed. It contains information about the CPU registers at the time of the eBPF program invocation. It is defined as :

To facilitate the loading of the aforementioned eBPF program, we’ll use the following program.

Here is a Makefile for compiling the eBPF program and the loader

Conclusion

In this two-part exploration of Tracing System Calls Using eBPF, we've embarked on a fascinating journey through the inner workings of this powerful technology. Part 1 laid the foundation by introducing us to the fundamentals of eBPF and its predecessor, BPF, shedding light on their evolution and significance in modern cloud native environments. We uncovered how eBPF's safety, performance, and observability advantages empower us to trace system calls with unmatched efficiency.

In Part 2, we took our understanding to new heights. We delved into the world of Uprobes and Uretprobes, showcasing how they enable seamless instrumentation and monitoring of user-space applications. We then ventured into Kprobes and Kretprobes, unlocking the ability to dynamically trace and debug kernel functions. Armed with these advanced techniques, we gained valuable insights into system behavior, performance analysis, and even the detection of potential malicious activity.

As we conclude this journey into the heart of eBPF, we stand equipped with a powerful set of tools and knowledge. Whether you're a seasoned sysadmin, a curious developer, or a vigilant security enthusiast, the capabilities of eBPF open new doors to real-time monitoring and analysis.

Stay tuned for further insights and practical guidance in the world of eBPF, where innovation meets security, and the future of system monitoring becomes a reality.

Blog: Introducing a framework for regression testing against Linux kernels

Thu, 21 Sep 2023 00:00:00 +0000

There are a few foundational technologies that empower the Cloud Native ecosystem. Containers is one. And one of the basis for containerization is the Linux Kernel itself. With Falco, we are developing a runtime security tool that hooks directly in the kernel to collect information about the system and notifies about malicious behavior.
We have found the need to validate our drivers against various versions of the Linux kernel, to properly ensure that with each iteration of our drivers, supported kernels remained unaffected.
To elaborate, we lacked a means to guarantee that a new driver release could:

Successfully compile on multiple kernel versions.
Pass the eBPF verifier when executed on various kernel versions.
Operate as expected, such as retrieving kernel events, across multiple kernel versions.

To address this issue, we started a major intervention. Initially, a proposal was discussed and incorporated into the libs repository.

Since this was a pretty novel area, there were no pre-existing tools available to tackle it. Consequently, we embarked on the development of a completely new framework.
Allow us to introduce you to the kernel testing framework.

Components of a kernel testing framework

Considering the inherent characteristics of the challenge, we need to set up a complete virtual machine for each distinct kernel version.
These tests should be executed automatically each time new code is integrated into our drivers, serving as a means to promptly identify any issue or flaw in the tested kernel versions.
With these objectives in mind, our approach should fulfill the following requirements:

Rapid and cost-effective VM creation: the process of creating these virtual machines should be efficient and budget-friendly.
Effortless distribution of VM images: we should ensure easy sharing and deployment of the virtual machine images.
Parallel execution of tests on multiple VMs: tests should run concurrently on each virtual machine to expedite the process.
Reproducibility in local environments for debugging purposes: developers should be able to replicate the test environment locally to investigate and troubleshoot issues.
Straightforward and user-friendly presentation of the test results: they should be presented in a simple and intuitive manner to immediately spot failures.

Ignite a Firecracker microVM

Weave Ignite is used to provision the firecracker microVMs. Weave Ignite is an open source tool designed for lightweight and fast virtual machine management. It enables users to effortlessly create and manage virtual machines (VMs) for various purposes, such as development, testing, and experimentation. One of the main reasons why we chose to use this tool was its capability to create firecracker microVMs from kernels and rootfs packed as OCI images. Currently, we are using a patched version located at a forked repository. These patches were essential to enable the booting of kernels that necessitated the use of an initrd (initial ramdisk).

Kernel & Rootfs OCI images

Virtual machines consist of two essential layers: the kernel and the rootfs. These layers are packaged and distributed as OCI (Open Container Initiative) images. The kernel image encompasses the kernel that the virtual machine relies on, in contrast the rootfs image serves as the fundamental building block of a virtual machine, offering the essential filesystem necessary for booting the VM. Typically, these rootfs images incorporate a Linux distribution. For more info on how we build them please check the available images documentation.

Ansible Playbooks

Automation is accomplished through the utilization of Ansible. A collection of playbooks is responsible for:

Orchestrating the provisioning of microVMs.
Configuring the machines.
Retrieving the code to be tested.
Eliminating the microVMs once the testing process is completed.

Presenting test results

We wanted the test data to be publicly and easily accessible by anyone, thus we had to find a way to represent the test output.
Since there are 3 possible ways of instrumenting the kernel, that are using a kernel module or one of the available eBPF probes, the playbooks perform up to 3 tests. Taking into account that the modern eBPF probe is built in the Falco libraries, only 2 drivers need to be compiled. We have 3 possible results for each of them:

success, when the test goes fine
error, when the test fails
skipped, when the test is not runnable for the kernel (for example, skipping modern eBPF tests where it is unsupported)

The natural way of dealing with all of this was to develop a small tool that, given as input the output root folder, would generate a markdown matrix with the results.

While scrutinizing the first version of the markdown matrix, we understood that it would have been even better if errors were also attached to the markdown, allowing for a more streamlined visualization of the results.
This is the format we settled with; it can be found at libs github pages:

How we use it

We implemented a new Github action workflow in the libs repository that triggers on pushes to master, using an x86_64 and an aarch64 nodes with virtualization capabilities provided by the CNCF.
The workflow itself is very simple since it runs the testing framework on self-hosted nodes just like you would run it locally:

jobs:
test-kernels:
strategy:
fail-fast: false
matrix:
architecture: [X64, ARM64] # We use a matrix to run our job on both supported arch
# Since github actions do not support arm64 runners and they do not offer virtualization capabilities, we need to use self hosted nodes.
runs-on: [ "self-hosted", "linux", "${{matrix.architecture}}" ]
steps:
# We clone the kernel-testing repo, generate vars.yaml (ie: input options for the kernel-testing run)
# and run needed ansible playbooks one by one, directly on each node.
- name: Checkout
uses: actions/checkout@v3
with:
repository: falcosecurity/kernel-testing
ref: v0.2.3
- name: Generate vars yaml
working-directory: ./ansible-playbooks
run: |
LIBS_V=${{ github.event.inputs.libsversion }}
LIBS_VERSION=${LIBS_V:-${{ github.ref_name }}}
cat > vars.yml <<EOF
run_id: "id-${{ github.run_id }}"
output_dir: "~/ansible_output_${{ github.run_id }}"
repos:
libs: {name: "falcosecurity-libs", repo: "https://github.com/falcosecurity/libs.git", version: "$LIBS_VERSION"}
EOF
- name: Bootstrap VMs
working-directory: ./ansible-playbooks
run: |
ansible-playbook bootstrap.yml --extra-vars "@vars.yml"
- name: Common setup
working-directory: ./ansible-playbooks
run: |
ansible-playbook common.yml --extra-vars "@vars.yml"
- name: Prepare github repos
working-directory: ./ansible-playbooks
run: |
ansible-playbook git-repos.yml --extra-vars "@vars.yml"
- name: Run scap-open tests
working-directory: ./ansible-playbooks
run: |
ansible-playbook scap-open.yml --extra-vars "@vars.yml" || :
# Once test finished, we collect its output folder and upload it to the github workflow space
- name: Tar output files
run: |
tar -cvf ansible_output.tar ~/ansible_output_${{ github.run_id }}
- uses: actions/upload-artifact@v3
with:
name: ansible_output_${{matrix.architecture}}
path: ansible_output.tar
# We then build the matrix and upload them too
- name: Build matrix_gen
working-directory: ./matrix_gen
env:
GOPATH: /root/go
GOCACHE: /root/.cache/go-build
run: |
go build .
- name: Generate new matrix
working-directory: ./matrix_gen
run: |
./matrix_gen --root-folder ~/ansible_output_${{ github.run_id }} --output-file matrix_${{matrix.architecture}}.md
- uses: actions/upload-artifact@v3
with:
name: matrix_${{matrix.architecture}}
path: ./matrix_gen/matrix_${{matrix.architecture}}.md
# Always run the cleanup playbook to avoid leaving garbage on the nodes
- name: Cleanup
if: always()
working-directory: ./ansible-playbooks
run: |
ansible-playbook clean-up.yml --extra-vars "@vars.yml" || :

In the Generate new matrix step, the kernel matrix gets generated and then uploaded.
Once this workflow runs successfully for both architectures, another workflow gets triggered,
that is responsible for generating and pushing updated Github pages.
The end result can be seen at https://falcosecurity.github.io/libs/matrix/.

Moreover, the kernel-testing workflow gets also triggered on each driver's tag; then a supplementary workflow takes care of attaching matrixes to the release body;
here is an example: https://github.com/falcosecurity/libs/releases/tag/6.0.0%2Bdriver.

Pretty nice, uh?

What's next for the framework

There are quite a few gaps that still need to be addressed by our framework. First of all, the images being used by Ignite to spawn FireCracker VMs are still under a development Docker repository and need to be moved under Falcosecurity.
Moreover, we need to implement a CI to automatically build and push those images.

As previously said, the kernel tests are currently running scap-open binary to check whether any event gets received. It is a small libscap C example that loads a driver and waits for events, nothing more.
It would be great to run drivers tests instead, to fully test the expected behavior of the drivers.

Finally, an utopian idea: imagine if we were able to run kernel-crawler to fetch kernel images, and then automatically build new kernel testing matrix entries for newly discovered images.
This would mean that our kernel testing matrix coverage increases steadily week after week, giving users even more guarantees about the stability of the Falco drivers!

Here is the libs tracking issue: https://github.com/falcosecurity/libs/issues/1224.

We would love to hear back from you! If you try out the framework and find any issues, please file them at https://github.com/falcosecurity/kernel-testing/issues. If you want to help us to improve, please join our Slack, and feel free to open a Pull Request!

Blog: Tracing System Calls Using eBPF - Part 1

Mon, 11 Sep 2023 00:00:00 +0000

Introduction:

In this article, we will delve into the details of eBPF (extended Berkeley Packet Filter) and explore its significance in tracing system calls. This particular blog will be in two parts; in the first blog, we will discuss eBPF, and in the subsequent section, we will delve into probes. eBPF is a powerful technology that allows for the dynamic and efficient tracing of events within the kernel space of an operating system. You have probably heard of the acronyms BPF and eBPF being used interchangeably. That's why we will aim to address both BPF and eBPF before discussing how and why Falco uses this technology.

BPF (Berkeley Packet Filter)

BPF is a technology used for network packet filtering and analysis. It is a powerful tool for implementing network security features, such as firewalls and intrusion detection systems. It can also be used to examine network traffic in real-time, detect suspicious patterns, and take appropriate actions to protect the network.

eBPF (Extended Berkeley Packet Filter)

The Extended Berkeley Filter (eBPF) is an evolution of the original BPF technology. It extends the capabilities of BPF by providing a more powerful and flexible way to perform dynamic tracing, network analysis, and performance monitoring. It allows developers to write and load programs into the kernel which can be attached to various hooks and events in the system. These programs can provide real-time insights and control over system activities.

Working on an eBPF program

The process of compiling and running an eBPF program involves several steps:

The eBPF program is converted into bytecode by using a compiler, ready to be loaded by a loader program.
The eBPF verifier checks the program for safety, correctness and adherence to specific rules and constraints. First of all, it performs a depth-first search on all possible execution paths to ensure that the program always proceeds to completion. Next, it performs a static analysis on the bytecode and ensures that the program doesn't violate memory access rules, and doesn't cause instability.
Once the eBPF program passes the verification process, it can be loaded into the kernel. The loader ensures that the program is securely loaded and attached to the desired hooks or targets in the system.
At runtime, the eBPF bytecode is further optimised through JIT (Just-in-time) compilation. This step converts the eBPF bytecode into machine code that can be executed by the CPU.

Kernel Modules

Apart from eBPF, the other approach we previously discussed for process tracing in Linux is the use of kernel modules. Kernel modules allow developers to write custom code that can be loaded into the kernel to extend its functionality.

By leveraging kernel modules, it is possible to hook into various points of the kernel's process management code and capture detailed information about process execution. This includes events such as process creation, termination, and context switches.

By accessing the kernel's internal data structures and functions, the module can gather valuable insights such as process IDs, parent-child relationships, execution time, system calls, and more.

So why does Falco use eBPF?

The integration of eBPF brings significant advantages to projects like Falco, empowering them to securely and efficiently monitor and analyze system calls in real-time. You might be wondering why eBPF is necessary when Falco already has real-time detection capabilities through its kprobe (kernel probe) that handles syscall events.

One compelling reason for incorporating eBPF support is to enable Falco to seamlessly operate in modern cloud native environments, where the traditional kernel probe may encounter limitations or face restrictions imposed by the control plane nodes.

By embracing eBPF, Falco ensures the continuity of its real-time detection capabilities in a secure manner, allowing for the prompt and accurate identification of security incidents, regardless of the underlying environment.

Later in the article, we will delve into the various considerations surrounding the adoption of an eBPF probe for Falco, providing valuable insights for determining when it becomes advantageous to leverage this functionality.

eBPF programs vs kernel modules

Safety and Isolation

eBPF programs are subjected to a thorough verification process before they are loaded into the kernel. This step provides an extra layer of protection and helps prevent security vulnerabilities. In contrast, kernel modules have direct access to the kernel code, which can pose a threat to the system if not implemented correctly.

Performance

eBPF programs are JIT compiled into machine code, which significantly improves the performance. JIT compilation optimizes the program for the specific CPU architecture, enabling efficient execution. Despite all these efforts, an eBPF instrumentation will always cause a greater overhead in the system than a kernel module one, the reason is that in the kernel module instrumentation there are no calls to the BPF subsystem.

Observability and Debugging

eBPF provides powerful tracing and observability capabilities. eBPF programs can be attached to various events, such as network packets, system calls, or kernel functions, allowing detailed visibility into the system behaviour. This makes eBPF a valuable tool for debugging, performance analysis, and security monitoring. Kernel modules typically require more invasive and complex mechanisms for achieving similar observability.

Attaching eBPF programs to hooks and events

There are various instrumentation points defined in the Linux kernel. An instrumentation is a specific point in a computer program where additional code, known as instrumentation code, is inserted to gather information about the program's execution. Instrumentation code can be injected at runtime using JIT compilation. Kernel probes, tracepoints, user-space probes, kretprobes are examples of instrumentation points.

Here is an eBPF program that runs when the execve system call is made.

In the eBPF programming context, the macro SEC() from the bpf/bpf_helper.h header file plays a crucial role. It allows the programmer to specify the section in which a function or variable will be placed within the eBPF object file. This becomes essential when loading eBPF programs into the kernel using mechanisms like the bpf() system call.
By organizing functions and variables into named sections, the eBPF loader can efficiently locate and load the required code and data. Specifically, when dealing with tracepoint events, the SEC format follows the pattern SEC("tp/<category>/<name>"), where <category> and <name> represent the respective tracepoint category and event name.
tp/syscalls/sys_enter_execve refers to a tracepoint that records when a process spawns the execve system call.
A list of all the available tracepoints is present in the /sys/kernel/debug/tracing/available_events file. The format for each line in the file is <category>:<name>. For example, syscalls:sys_enter_execve.

Before compiling the program, we need to do some basic configuration:

Let's compile the program. The following command can be used to do this task:

Now, we need to write a loader program that loads and attaches this program. This loader program is used to load and attach an eBPF program to the Linux kernel. It opens and loads the eBPF object file, checks for errors during the process, finds a specific eBPF program within the loaded object, and attaches it to the kernel. Once attached, the eBPF program will be executed when certain events occur. The program enters an infinite loop at the end, indicating that it will continue running until it is manually terminated.

Let's compile and run this program

To get the logs generated by the function bpf_printk, we can read the file: /sys/kernel/tracing/trace_pipe

Manually reading messages from the tracepipe doesn't seem to be an efficient approach. It would be advantageous to establish a mechanism for the eBPF program to send messages to the loader program. One viable solution is to utilize ring buffers. Let’s review more details about ring buffers.

Ring buffers

eBPF ring buffer, also known as bpf_ringbuf, is a mechanism provided by the Linux kernel for efficient communication between eBPF programs and user-space programs.

It allows the exchange of data and events between eBPF programs running in the kernel and user-space applications. It is a MPSC (Multi Producer Single Consumer) queue and can be safely shared across multiple CPUs simultaneously.

The eBPF ring buffer, being shared across all CPUs, offers a unified and efficient solution for managing memory utilisation, mitigating issues of overuse or under-allocation that commonly occur with perfbuf.

Let's have a look at a few functions that we'll be using to write an eBPF program that sends data to userspace.

bpf_ringbuf_reserve

This function is used to reserve size bytes of space in a BPF ring buffer.

bpf_probe_read_user_str

This function is used to read a null terminated string from user-space memory into the destination dst. The dst parameter is a pointer to the destination buffer in the kernel space. unsafe_ptr is a pointer to the source string in the user-space.

bpf_ringbuf_submit

This function is used to submit data that had previously been reserved in a ringbuf.

bpf_object__find_map_fd_by_name

This function is used to find the file descriptor of a named map.

bpf_program__attach_tracepoint

This function is used to attach an eBPF program to a kernel tracepoint.

ring_buffer__new

This function is used for creating and opening a new ringbuf manager.

ring_buf__consume

Used to remove or consume data from a ring buffer.

BTF (BPF Type Format)

It provides a way to describe the types of data structures used by eBPF programs, allowing for improved type safety, debugging, and introspection.

Now, let's write a program that sends data to userspace using ringbuf.

Having created the program, we can write a loader to load this eBPF program.

The infinite loop is necessary to ensure that the program continuously checks for new events in the ring buffer. Without the loop, the program would only consume events that were already in the buffer at the time of the initial ring_buffer__consume() call. By looping and calling ring_buffer__consume() repeatedly, the program can retrieve events as soon as they become available and process them in real-time. The sleep(1) call within the loop serves to reduce the CPU usage of the program by introducing a one-second delay between each call to ring_buffer__consume().

Great, we were able to recover the process name as well as the PID!

Conclusion

In conclusion, this article has provided a comprehensive overview of eBPF (extended Berkeley Packet Filter) and its significance in tracing system calls. We have explored the evolution from BPF to eBPF, discussed why Falco uses this technology, and delved into the process of working with eBPF programs and ring buffers for efficient data communication between the kernel and user-space applications.

As we journeyed through the capabilities of eBPF in this first part, we uncovered its benefits in terms of safety, performance, and observability when compared to traditional kernel modules. eBPF empowers us to securely and efficiently monitor and analyze system calls in real-time, making it a valuable tool in modern cloud native environments.

In the upcoming second part of this blog series, we will further expand our exploration by delving into the realm of probes and additional advanced topics. We will dive deeper into how eBPF probes can be leveraged for enhanced system tracing, performance analysis, and security monitoring. Stay tuned for more insights and practical guidance on harnessing the power of eBPF.

Keep an eye out for Part 2, where we'll continue our journey into the world of eBPF and system call tracing.

Blog: Modern eBPF probe is ready to shine

Wed, 14 Jun 2023 00:00:00 +0000

Introducing the brand-new eBPF probe: a game-changing addition to Falco's toolkit. Curious to learn more? Dive into our first blog post where we spill the beans on its exciting features, what you need to get started, and real-world use cases.

Initially a Falco 0.34.0 experimental feature, we've put in months of hard work to refine it for production use. The wait is finally over! Falco 0.35.0 now ships with the modern probe as an official driver, alongside the trusted Kernel module and current eBPF probe. Falco is now more prepared than ever to scale with your infrastructure.

Supported Syscalls

In our driver history, we've supported syscalls in two possible ways:

Fully instrumented 🟢
Generically instrumented 🟡

Fully instrumented means that Falco notifies when a syscall is invoked in the system and enables the user to examine its parameters. You can find the list of available parameters for each syscall on https://falco.org/docs/reference/rules/supported-events/.

Generically instrumented means that Falco notifies when a syscall is invoked in the system, but nothing more.

Excitingly, in the latest Falco 0.35.0 version, the modern probe extends its support to reach syscall parity with the other drivers. This report outlines the current instrumentation state.

Advanced support checks

Before Falco 0.35.0, the modern probe was restricted to running exclusively on machines with kernel versions >= 5.8. This limitation posed challenges for certain users, prompting us to implement a more intelligent support detection mechanism. Leveraging the capabilities of the libbpf library, we can now search for specific features within the target system, precisely addressing our needs.

To elaborate further, we currently search for two crucial features that are essential for running the modern probe:

BPF Ring Buffer
BTF Tracing Programs

For a more in-depth understanding of these two concepts, we invite you to explore the previous blog discussing the modern eBPF driver.

Now, let's delve into the potential errors that you may encounter if any of these features are missing.

If the BPF ring buffer is not available, Falco will present you with an error message similar to:
```
ring buffer map type is not supported
```
If BTF tracing programs are absent, the error message you can expect to encounter would be along the lines of:
```
tracing program type is not supported
```

Buffers allocation

Every Falco driver utilizes shared buffers to facilitate the transmission of security events between the kernel and the userspace. To be more specific, there is an individual buffer allocated for each online CPU, ensuring efficient handling of events.

We have always followed this particular approach for utilizing shared buffers, and recently we introduced a new feature that allows you to modify the size of these buffers using a custom Falco configuration option called syscall_buf_size_preset. By default, each buffer is set to 8 MB, but you have the flexibility to adjust it anywhere between 1 MB and 512 MB.

Increasing the buffer size can be beneficial when you encounter syscall drops. A larger buffer size can help mitigate syscall drops in systems with high production loads. However, it's important to note that very large buffers may also impact the overall system performance, potentially slowing down the machine.

Conversely, reducing the buffer size can help enhance system speed, but it may also result in an increased number of syscall drops. It's crucial to exercise caution when experimenting with this configuration option, taking into consideration the trade-off between performance and syscall drops ⚠️

While the ability to adjust buffer sizes is a cool feature, it is available to all drivers. So, what sets the modern probe apart? Well, with this new driver, you have the added capability to manipulate the number of buffers. This means that the traditional rule of having one buffer per CPU is no longer a strict requirement. Unlike the old drivers, where the only possible configuration was one buffer per CPU, the modern probe introduces flexibility in this aspect, opening up new possibilities and alternative scenarios.

As an illustration, one such scenario is the allocation of a ring buffer for every 3 CPUs.

Observe that the second buffer can still be used by another CPU.

To adjust the number of buffers, you can use the Falco configuration option called modern_bpf.cpus_for_each_syscall_buffer. Unlike other drivers that have a 1:1 mapping between buffers and CPUs, the modern probe has a default value of one buffer for every two CPUs. This distinction arises because the BPF ring buffer requires more memory compared to other drivers, necessitating a reduction in the number of buffers.

However, feel free to experiment and find the configuration that best suits your system. Just remember the following guideline: having more buffers can reduce the likelihood of drops but will increase the overall memory footprint. On the other hand, reducing the number of buffers can help decrease memory consumption but may lead to an increased risk of drops.

Least privileged mode

Similar to the current probe, the modern probe can operate in least privileged mode. However, to ensure proper functionality, Falco always mandates a minimum of two capabilities: CAP_SYS_RESOURCE and CAP_SYS_PTRACE. Additional required capabilities vary depending on your specific kernel version, like the CAP_SYS_ADMIN capability for older kernels, which can be replaced by the CAP_PERFMON and CAP_BPF ones when running on a kernel newer than 5.8.

Here's an example command to run Falco in least privileged mode using the modern probe:

docker run --rm -i -t \
 --cap-drop all \
 --cap-add sys_admin \
 --cap-add sys_resource \
 --cap-add sys_ptrace \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -v /proc:/host/proc:ro \
 falcosecurity/falco-no-driver:latest falco --modern-bpf

Observe we cannot use CAP_BPF and CAP_PERFMON here since docker doesn't support them yet.

Try it out

The modern eBPF probe is compatible with all the installation methods available for other drivers.

Also, you can test it live in this interactive environment that we have prepared for you.

Click on it to start playing with it 🎮

Blog: Getting started with modern BPF probe in Falco

Wed, 30 Nov 2022 00:00:00 +0000

The new BPF probe has landed among us 👽 and it brings to the table new shiny features. The BPF world grows continuously and every new kernel release introduces some unbelievable novelties!

To take advantage of these we have created a completely new architecture, new BPF programs and maps. The main goal is to improve performance, maintainability, and user experience, shipping a unique, powerful, self-contained Falco executable. (... Oops, I said too much but don't burn the steps, first things first.)

What's new 🗞️

We wouldn't be able to cover all the details here so we will just go through the most outstanding features. If you are interested in technical aspects you can always take a look at the proposal merged upstream some months ago with its corresponding discussion.

CO-RE paradigm

Portability is one of the biggest issues we have with the current BPF driver. Our infrastructure tries to compile a BPF probe for every supported kernel since version 4.14! As you can imagine, this is not a simple task, and it causes some pain to both, users and maintainers. For this reason, the Falco maintainers have been working hard to adopt the so-called CO-RE paradigm.

CO-RE stands for "Compile-once-run-everywhere", so as you may imagine, this paradigm allows compiling the BPF probe just once for all kernels!

You understood well: No more missing drivers, and no more painful local builds requiring the much-loved KERNEL HEADERS.

BPF Ring Buffer map

Today, whenever a BPF program needs to send data to userspace, it first copies it into a BPF map, and then it pushes its content to a shared buffer located between userspace and kernel: The so-called perf-buffer.

This solution works great, but it has two major shortcomings that prove to be inconvenient in practice: Inefficient APIs and extra data copying.

The ring buffer can be considered an evolution of the perf buffer. We still use a shared buffer but with some advantages.

First, the APIs to interact with are more efficient. Second, and more importantly, we now have the possibility of writing data directly into this shared buffer, without having to write it twice!

This is a game changer in scenarios where high throughput is required, like running Falco, since it could save many cycles during the collection phase.

BTF-enabled program

Kernel engineers recently introduced a sort of debugging information for BPF programs/maps. This is called BTF and stands for "BPF Type Format". This feature shook the foundations of the BPF world because it finally offers the possibility to write code without BPF helpers like the famous bpf_probe_read().

This will not only increase the readability of the BPF code but will also reduce the bytecode dimension, allowing the crafting of smaller and more efficient programs!

BPF global variables

The addition of global variables points in the same direction as BTF: Simplify the code experience, increase the readability and reduce the performance overhead due to BPF helpers.

While BTF allows to deference kernel pointer without the use of the bpf_probe_read() helper, global variables allow to access BPF maps without the use of helpers like bpf_map_lookup_elem().

You can find further technical information about these 2 new features in the probe proposal.

BPF skeleton

Do you remember when we talked about Falco being a self-contained binary?

Well the BPF skeleton concept allows us to achieve the dream: Ship Falco as a unique, self-contained executable!

Under the hood, the probe is compiled once into Falco when the package is built. Hence, when you deploy Falco on different machines it will automatically inject the code without any extra effort on your side. Hard to believe? Try it here.

Multi-arch support

The modern BPF probe also supports multiple architectures by design. The actual targets are x86_64, arm64, and s390x, but new ones can be added at any time.

If you have a project that needs BPF instrumentation for one of these architectures you could simply link the Falco libraries (libsinsp, libscap) to obtain a working solution out of the box. We would to thank Hendrik Brueckner for the huge help he gives in reviewing and implementing the multi-arch support!

At this moment, we ship Falco for x86_64 and arm64 architectures only, due to their popularity in the community.

Requirements ⛓️

To use the modern BPF probe, there are 2 main requirements:

A kernel that implements all the aforementioned features. Linux kernel version 5.8 is the first kernel that supports all of them, and for this reason, we consider it the minimum required one.

Nevertheless, these features could also be backported into older kernels, so it wouldn't be completely fair to define the 5.8 as the first supported version. This is just a strict assumption that we put in place since we still miss the logic to detect all the necessary features of your kernel.

Until that day arrives, if your kernel is older than 5.8, you might find the following error when trying to start the modern probe:
```
Error: Actual kernel version is: '5.4.0' while the minimum required is: '5.8.0'
```
A kernel that exposes BTF types. This shouldn't be a big issue since we already require a kernel version newer than 5.8 and most recent Linux distributions come with kernel BTF capabilities.

If you want to be sure you can easily check their presence by typing:
```
ls /sys/kernel/btf/vmlinux
```
If your kernel supports them you should see:
```
/sys/kernel/btf/vmlinux
```

Now, if your machine satisfies these 2 requirements, you are ready to have fun with the modern probe! 🚀

Modern BPF in action 🏎️

Falco provides you with pre-release packages to try the modern BPF probe, but the full release will take place in Falco 0.34.

If you happen find some bugs or misbehaviors, please be kind with us and open an issue on Falco 🙏

Having said that, you have 3 main ways to try Falco with the modern probe.

1. Pre-built `deb`, `rpm` packages

You can find here the x86_64 packages and here the arm64 ones.

Taking as an example the deb package, you have simply to download it and type the following command:

sudo dpkg -i <your_deb_package.deb>

Now there are 2 possibilities:

If you have the dialog binary installed on your system you should see a dialog window like this (the dialog is a new feature that will be regularly shipped in Falco 0.34 for all the drivers!):

You have simply to choose the "Modern eBPF" option.

If you don't have dialog installed, you must start the service manually:

sudo systemctl start falco-modern-bpf.service

Typing sudo systemctl status falco-modern-bpf.service should either way show you a similar output to:

Nov 15 14:50:46 ip-172-31-13-74 systemd[1]: Started Falco: Container Native Runtime Security with modern ebpf.
Nov 15 14:50:46 ip-172-31-13-74 falco[1587330]: Falco version: 0.32.1-291+5d1b0c5 (x86_64)
Nov 15 14:50:46 ip-172-31-13-74 falco[1587330]: Falco initialized with configuration file: /etc/falco/falco.yaml
Nov 15 14:50:46 ip-172-31-13-74 falco[1587330]: Loading rules from file /etc/falco/falco_rules.yaml
Nov 15 14:50:46 ip-172-31-13-74 falco[1587330]: Loading rules from file /etc/falco/falco_rules.local.yaml
Nov 15 14:50:46 ip-172-31-13-74 falco[1587330]: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Nov 15 14:50:46 ip-172-31-13-74 falco[1587330]: Starting health webserver with threadiness 8, listening on port 8765
Nov 15 14:50:46 ip-172-31-13-74 falco[1587330]: Enabled event sources: syscall
Nov 15 14:50:46 ip-172-31-13-74 falco[1587330]: Opening capture with modern BPF probe

2. Tar archive

You can download here the x86_64 tar.gz and here the arm64 one.

Here the procedure is very simple, you can extract the contents of the archive and execute the Falco binary:

tar -xvf <targz_package.tar.gz>
cd <untar_folder>
sudo ./usr/bin/falco --modern-bpf -c ./etc/falco/falco.yaml -r ./etc/falco/falco_rules.yaml

Please note: The command line option required to run Falco with the modern BPF probe is --modern-bpf

3. Docker image

In an environment where you can start containers, you can simply use the docker images from DockerHub:

# x86_64
docker pull andreater/falco-modern-x86:latest
docker run --rm -i -t \
 --privileged \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -v /proc:/host/proc:ro \
 andreater/falco-modern-x86:latest

# arm64
docker pull andreater/falco-modern-arm64:latest
docker run --rm -i -t \
 --privileged \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -v /proc:/host/proc:ro \
 andreater/falco-modern-arm64:latest

Please note: The helm chart is not available yet since this is a pre-release but it will be shipped as expected with Falco 0.34

Current syscall support ⏲️

The modern BPF probe doesn't yet support all the syscalls supported by the current probe. It only supports the "simple consumer set", which means that we support all the syscalls necessary to run Falco without -A option, also known as the default Falco mode. At the moment, adding the -A flag wouldn't have any further effect.

Here you can find the complete list of syscalls currently supported by the modern probe.

The good news is that we are actively working to include all the syscalls supported by the current probe 💥

Next steps 🔮

Release Falco 0.34 with the modern probe as an experimental feature.
Reach the full support of syscall.
Introduce new points of instrumentation like LSM hooks to collect security events.

Falco and modern BPF at eBPF summit 🐝

If you want to know more about how the modern BPF probe works under the hood take a look at the eBPF summit presentation.

Blog: Falco Driverkit with Docker on Debian

Mon, 05 Sep 2022 00:00:00 +0000

We use different technologies on a daily basis. Tools like Vagrant, Terraform, Ansible, and many more allow us to create and destroy digital resources in a matter of minutes, if not seconds. However, if you keep changing your running environment, you might also need to calibrate your workloads to these new changes. This is especially true when you deploy applications tightly dependent on the operating system.

In other words, every time you deploy an application like Falco there's a chance that you need to compile a new module or eBPF probe to get along with the current underlying kernel. This is the first of a series of posts where you will learn some interesting techniques related to how Falco generates the much needed driver and how you can make it available for your deployments.

Falco on Docker

There are many ways to run Falco: as a service, as a local container, as a Pod in Kubernetes, etc. Either way, if what we want to do is use Falco to detect threats based on syscalls, we will need a driver that has been compiled for the specific kernel running on the machine, be it a physical machine, a virtual one, or a Kubernetes node in the cloud.

Launching Falco as a container

The Falco image embeds a script, /usr/bin/falco-driver-loader, that will automatically try to find and download a kernel module or an eBPF probe. If that wasn't possible, it might try to compile it inside the container itself. We will learn a bit more about this process and how to control it.

Here is the output of a fresh instance of falco running on our local docker service:

$ docker run -it --privileged \
 --name falco-driver-test \
 -v /dev:/host/dev \
 -v /etc:/host/etc:ro \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 docker.io/falcosecurity/falco:0.32.2

Unable to find image 'falcosecurity/falco:0.32.2' locally
0.32.2: Pulling from falcosecurity/falco
7e6a53d1988f: Pull complete
... output omitted
f3102eb3e85f: Pull complete
Digest: sha256:5ceb23e5baae9c86fc0b160fed397facd2074ca398b770878adbb9c6d049d8a8
Status: Downloaded newer image for falcosecurity/falco:0.32.2

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.32.2, driver version=2.0.0+driver
* Running falco-driver-loader with: driver=module, compile=yes, download=yes

================ Cleaning phase ================

* 1. Check if kernel module 'falco' is still loaded:
- OK! There is no 'falco' module loaded.

* 2. Check all versions of kernel module 'falco' in dkms:
- There are some versions of 'falco' module in dkms.

* 3. Removing all the following versions from dkms:
2.0.0+driver

- Removing 2.0.0+driver...

------------------------------
Deleting module version: 2.0.0+driver
completely from the DKMS tree.
------------------------------
Done.

- OK! Removing '2.0.0+driver' succeeded.


[SUCCESS] Cleaning phase correctly terminated.

================ Cleaning phase ================

* Looking for a falco module locally (kernel 5.10.0-14-cloud-amd64)
* Trying to download a prebuilt falco module from https://download.falco.org/driver/2.0.0%2Bdriver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco module
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"

Creating symlink /var/lib/dkms/falco/2.0.0+driver/source ->
 /usr/src/falco-2.0.0+driver

DKMS: add completed.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2.0.0+driver/build/make.log (with GCC /usr/bin/gcc)
* Trying to dkms install falco module with GCC /usr/bin/gcc-8
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2.0.0+driver/build/make.log (with GCC /usr/bin/gcc-8)
* Trying to dkms install falco module with GCC /usr/bin/gcc-6
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2.0.0+driver/build/make.log (with GCC /usr/bin/gcc-6)
* Trying to dkms install falco module with GCC /usr/bin/gcc-5
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2.0.0+driver/build/make.log (with GCC /usr/bin/gcc-5)
* Trying to load a system falco module, if present
Consider compiling your own falco driver and loading it or getting in touch with the Falco community
2022-08-31T12:00:02+0000: Falco version 0.32.2
2022-08-31T12:00:02+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:00:02+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:00:02+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:00:02+0000: Starting internal webserver, listening on port 8765
2022-08-31T12:00:02+0000: Unable to load the driver.
2022-08-31T12:00:02+0000: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

There are some important reads from this output:

The driver version this image tries to load is the 2.0.0+driver. This information will be really useful when we need to compile and share the driver with the falco container.
By default, the container will look for a kernel module. It is possible to switch to an eBPF probe by using an environment variable, as you'll see later in this post.
The falco-driver-loader script always removes the driver from memory and tries to load a current one. This is done for security reasons and the way to avoid that is not running this script when creating the container. More on this later, too.
After looking in the system for a previously installed driver, the script tries to download it from the URL https://download.falco.org. Unfortunately, it doesn't seem to be able to find it and falls back to the local compilation method.
When the script tries to compile the driver inside the container, it doesn't succeed because we haven't fulfilled one important prerequisite: installing the kernel headers on the host machine. In this post, we won't address that method but you can always refer to the documentation.

Using Falco Driverkit

As mentioned, there are different ways to obtain a valid kernel: downloading it from https://download.falco.org, compiling it via the falco-driver-loader script, or the method we'll explain here: using driverkit.

We don't intend this post to be an exhaustive guide to driverkit. That's also why we've chosen a relatively easy and tested target operating system: Debian.

First of all, we need the driverkit tool which we'll compile ourselves. We can download the source code from https://github.com/falcosecurity/driverkit.

When compiling a tool, we like using a temporary container. In this case, we'll start our container using the docker.io/golang:1.19 image and a sleep process until we're done. The ./driverkit directory will help us to extract the binary to the host filesystem. Feel free to use any other method you prefer, like docker cp.

# This directory will contain the driverkit binary
# once it is compiled inside the container

$ mkdir ./driverkit

# Container with Go tools
$ docker run --rm -d --name golang-compiler \
 -v $PWD/driverkit:/export \
 golang:1.19 sleep infinity

Check that the container has been successfully created and still runs:

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1ff943cbf7f9 golang:1.19 "sleep infinity" 4 seconds ago Up 3 seconds golang-compiler

Next, create a shell with a terminal in the container:

$ docker exec -it golang-compiler /bin/bash

Remember, you are in the container context now. Whatever you do here will be lost unless you copy it to the /export directory. We will clone the driverkit code and compile it using the following commands:

git clone https://github.com/falcosecurity/driverkit
cd driverkit && make
cp _output/bin/driverkit /export/driverkit
exit

Once we are done with the Golang container, we can stop it and it'll be automatically removed thanks to the --rm parameter that we used to start it.

docker stop golang-compiler -t0

Creating a configuration file for the driver request

Time to create a configuration file. Do you remember the driver version: 2.0.0+driver? We will use that and additional information to create the configuration file.

# We've included some VARIABLES that will help you understand
# where the different values come from and what they represent

export DRIVER_VERSION=2.0.0+driver
export DRIVER_TARGET=debian
export DRIVER_ARCH=$(arch)

export KERNEL_VERSION=$(uname -v| cut -f1 -d' ' | tr -d \#)
export KERNEL_RELEASE=$(uname -r)
export DRIVER_NAME=falco
export PROBE_FILE=${DRIVER_NAME}_${DRIVER_TARGET}_${KERNEL_RELEASE}_${KERNEL_VERSION}.o
export MODULE_FILE=${DRIVER_NAME}_${DRIVER_TARGET}_${KERNEL_RELEASE}_${KERNEL_VERSION}.ko

mkdir -p drivers/${DRIVER_VERSION}/${DRIVER_ARCH}/

# Creating the actual file that we will pass to driverkit

cat > debian.yaml << EOF
target: ${DRIVER_TARGET}
driverversion: ${DRIVER_VERSION}
kernelrelease: ${KERNEL_RELEASE}
kernelversion: ${KERNEL_VERSION}
output:
 module: ./drivers/${DRIVER_VERSION}/x86_64/${MODULE_FILE}
 probe: ./drivers/${DRIVER_VERSION}/x86_64/${PROBE_FILE}
EOF

The resulting file should look like this:

$ cat debian.yaml
target: debian
driverversion: 2.0.0+driver
kernelrelease: 5.10.0-14-cloud-amd64
kernelversion: 1
output:
 module: ./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
 probe: ./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o

In case you want to use a version previous to Falco 0.32.2 you might need to remove the x86_64/ string from the probe path. This is due to the expected path inside the falco-driver-loader script. These paths will be offered via an HTTP server at a later stage, so make sure they match in both steps.

This same file is the one we will pass to driverkit. If the driver is compiled satisfactorily, we should see a similar output in some seconds. Be patient.

$ ./driverkit/driverkit docker -c debian.yaml

INFO using config file file=debian.yaml
INFO driver building, it will take a few seconds processor=docker
INFO kernel module available path=./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
INFO eBPF probe available path=./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o

Make sure you use either the .yml or .yaml suffix. Otherwise, you'll get an error like:

$ ./driverkit/driverkit docker -c debian.unknown-ext
Error: exiting for validation errors
Usage:
 driverkit docker [flags]

Flags:

... output omitted

FATA error executing driverkit error="exiting for validation errors"

Alternatively, we could have used a bunch of parameters in the command line, like:

# Don't forget to \, to let the command continue after each line

$ ./driverkit/driverkit docker \
 --architecture amd64 \
 --target ${DRIVER_TARGET} \
 --driverversion ${DRIVER_VERSION} \
 --kernelversion ${KERNEL_VERSION} \
 --kernelrelease ${KERNEL_RELEASE} \
 --output-probe ./drivers/${DRIVER_VERSION}/x86_64/${PROBE_FILE} \
 --output-module ./drivers/${DRIVER_VERSION}/x86_64/${MODULE_FILE}

INFO driver building, it will take a few seconds processor=docker
INFO kernel module available path=./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
INFO eBPF probe available path=./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o

Either way, if driverkit manages to compile the drivers, you can continue with the next step. Otherwise, you might need to adjust some of the parameters in the configuration or even customize your builder image, but we will explain that in a different post where we will deep dive into driverkit.

Launching Falco with the new driver

There are different ways to load the driver when running Falco. We'll show you two of them: loading them manually and leaving this action to the script embedded in the container image.

Loading the driver manually

A kernel module only needs to be loaded once. So, if we load it manually before starting the container, Falco doesn't need to do it again.

There are two ways of achieving that, and both require avoiding the execution of the falco-driver-loader script:

Setting the SKIP_DRIVER_LOADER environment variable to any value when creating the container. By doing so, the container entrypoint will skip the existing falco-driver-loader script.
Using the image docker.io/falco/falco-no-driver, which doesn't contain that script.

First, try to load the driver on the host. Look for the .ko file in the directory structure we created and load it using insmod, for instance. If the compilation was successful and the kernel version chosen was the right one, you shouldn't see any message once the module is loaded. Don't forget to do it with the user root (i.e., via sudo).

$ find $HOME/drivers -type f
drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o
drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko

$ sudo insmod drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
$ lsmod | grep -i falco
falco 741376 0

This first method of starting the falco container will use the docker.io/falco/falco:0.32.2 image, passing the SKIP_DRIVER_LOADER variable. We've set it to one but the script doesn't check its value.

Observe that we're removing any existing container with that name before starting ours, but the container image remains.

$ docker rm -f falco-driver-test # Ignore any failure here
falco-driver-test

$ docker run -it --privileged \
 --name falco-driver-test \
 -e SKIP_DRIVER_LOADER=1 \
 -v /dev:/host/dev \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 docker.io/falcosecurity/falco:0.32.2

2022-08-31T12:07:30+0000: Falco version 0.32.2
2022-08-31T12:07:30+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:07:30+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:07:30+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:07:30+0000: Starting internal webserver, listening on port 8765

The second method uses the docker.io/falco/falco-no-driver image, so, as you can expect, it won't try to reload the driver this time.

This time, Docker will pull the image since we hadn't used it yet.

$ docker rm -f falco-driver-test # Ignore any failure here
falco-driver-test

$ docker run -i -t --privileged \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -v /dev:/host/dev \
 -v /proc:/host/proc:ro \
 --name falco-driver-test \
 docker.io/falcosecurity/falco-no-driver:0.32.2

Unable to find image 'falcosecurity/falco-no-driver:0.32.2' locally
0.32.2: Pulling from falcosecurity/falco-no-driver
1efc276f4ff9: Pull complete
e34e1870ff2c: Pull complete
Digest: sha256:14e6d3da56fe607ff9b0bfe91ec812ab4f4b030cea3ed88a2d31ac9b31f97fb4
Status: Downloaded newer image for falcosecurity/falco-no-driver:0.32.2

2022-08-31T12:12:40+0000: Falco version 0.32.2
2022-08-31T12:12:40+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:12:40+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:12:40+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:12:40+0000: Starting internal webserver, listening on port 8765

This method is a bit more complicated than the previous one but will also give us the flexibility of deploying falco at scale.

The idea is simple though. After starting your favorite webserver and publishing the ./drivers directory that we created before, we'll tell the falco container to use it as a repository and download the driver from there.

To keep things clean, we've used the docker.io/python:latest container image, which includes the Python module http.server. If you have Python already installed on your system, you can use it directly. Just remember to define a port accessible to the falco container and pass the right IP address.

$ docker run -d \
 --name falco-drivers-web \
 -v $PWD/drivers:/data:ro \
 docker.io/python:latest \
 bash -c "cd /data && python -m http.server"

Unable to find image 'python:latest' locally
latest: Pulling from library/python
1671565cc8df: Pull complete
... output omitted
4334b2fe8293: Pull complete
Digest: sha256:745efdfb7e4aac9a8422bd8c62d8bc35a693e8979a240d29677cb03e6aa91052
Status: Downloaded newer image for python:latest
f94cb601f85c312d62aab3e116619558239bada9f5d05e971fe26c0206828b6b

Our python web server is now available and offers the drivers to any local container that might need them. Retrieve the IP address of this container for later use:

$ docker inspect falco-drivers-web --format '{{ .NetworkSettings.IPAddress }}'
172.17.0.2

# Assign it to a variable for later use
$ export DRIVERS_REPO=$(docker inspect falco-drivers-web \
 --format '{{ .NetworkSettings.IPAddress }}'):8000

$ echo $DRIVERS_REPO
172.17.0.2:8000

It's always a good practice to test that the drivers are in the right place and accessible through the webserver.

# This is the checksum of the local files

$ find ./drivers -type f -name "*o" -exec cksum {} \;
3873827283 843080 ./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64.ko
914371530 4980536 ./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64.o

# This is the checksum of the files retrieved through HTTP

$ find ./drivers -type f -name "*o" | while read FILE
 do
 URL=$(echo ${FILE} | sed -e 's,./drivers,'${DRIVERS_REPO}',')
 echo "$(curl -s http://${URL} | cksum) http://${URL}"
 done
3873827283 843080 localhost:8000/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64.ko
914371530 4980536 localhost:8000/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64.o

As you can see, they are accessible and identical.

These values will be different depending on the version of the kernel and the Falco drivers.

Loading the kernel module

Let's start with the kernel module. In this case, the only variable we need to pass is the DRIVERS_REPO one, which has been carefully prepared in the previous step.

$ docker rm -f falco-driver-test
falco-driver-test

$ docker run -it --privileged \
 --name falco-driver-test \
 -v /dev:/host/dev \
 -v /etc:/host/etc:ro \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -e DRIVERS_REPO=${DRIVERS_REPO} \
 docker.io/falcosecurity/falco:0.32.2

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.32.2, driver version=2.0.0+driver
* Running falco-driver-loader with: driver=module, compile=yes, download=yes

================ Cleaning phase ================

* 1. Check if kernel module 'falco' is still loaded:
- OK! There is no 'falco' module loaded.

* 2. Check all versions of kernel module 'falco' in dkms:
- There are some versions of 'falco' module in dkms.

* 3. Removing all the following versions from dkms:
2.0.0+driver

- Removing 2.0.0+driver...

------------------------------
Deleting module version: 2.0.0+driver
completely from the DKMS tree.
------------------------------
Done.

- OK! Removing '2.0.0+driver' succeeded.


[SUCCESS] Cleaning phase correctly terminated.

================ Cleaning phase ================

* Looking for a falco module locally (kernel 5.10.0-14-cloud-amd64)
* Trying to download a prebuilt falco module from 172.17.0.2:8000/2.0.0%2Bdriver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
* Download succeeded
* Success: falco module found and inserted
2022-08-31T12:36:29+0000: Falco version 0.32.2
2022-08-31T12:36:29+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:36:29+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:36:29+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:36:29+0000: Starting internal webserver, listening on port 8765

It's a similar output as before, but this time we can see:

* Trying to download a prebuilt falco module from 172.17.0.2:8000/2.0.0%2Bdriver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
* Download succeeded
* Success: falco module found and inserted

The module has been successfully loaded and Falco can start properly.

Loading the eBPF Probe

For this, we will make use of another variable, FALCO_BPF_PROBE. Like it happened with the SKIP_DRIVER_LOADER variable, its value is not as relevant as the fact that the variable had been defined. We also need to keep the DRIVERS_REPO variable, since the falco-driver-loader script will look for the probe in that URL.

$ docker rm -f falco-driver-test
falco-driver-test

$ docker run -it --privileged \
 --name falco-driver-test \
 -v /dev:/host/dev \
 -v /etc:/host/etc:ro \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -e DRIVERS_REPO=${DRIVERS_REPO} \
 -e FALCO_BPF_PROBE="" \
 docker.io/falcosecurity/falco:0.32.2

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.32.2, driver version=2.0.0+driver
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
* Trying to download a prebuilt eBPF probe from 172.17.0.2:8000/2.0.0%2Bdriver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o
* Skipping compilation, eBPF probe is already present in /root/.falco/falco_debian_5.10.0-14-cloud-amd64_1.o
* eBPF probe located in /root/.falco/falco_debian_5.10.0-14-cloud-amd64_1.o
* Success: eBPF probe symlinked to /root/.falco/falco-bpf.o
2022-08-31T12:58:10+0000: Falco version 0.32.2
2022-08-31T12:58:10+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:58:10+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:58:10+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:58:10+0000: Starting internal webserver, listening on port 8765

This time the output is easier to read: The driver is set to bpf, the URL of the HTTP container points to our local webserver, and it also shows where it downloads the probe before starting Falco.

Debugging

As a final tip, if you want to start a container based on the regular falco image to test the falco-driver-loader script, we recommend starting the container with the --entrypoint /bin/bash parameter. This will keep the /docker-entrypoint.sh script from being executed (that one triggers /usr/bin/falco-driver-loader) and you'll have a much more comfortable environment to work with.

$ docker run -it --privileged \
 --name falco-driver-test \
 -v /dev:/host/dev \
 -v /etc:/host/etc:ro \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -e DRIVERS_REPO=${DRIVERS_REPO} \
 --entrypoint /bin/bash \
 docker.io/falcosecurity/falco:0.32.2

root@e0c391e0cee1:/#

Conclusion

Falco requires tapping into the kernel to be able to retrieve useful information from it. For that, it has two methods: loading a kernel module in the traditional way, or using an eBPF probe. Both of them instrumentalize the kernel and provide the functionality to retrieve the relevant data.

Due to the infinite number of combinations of Linux kernels and distributions, it is extremely difficult to offer all possible kernels as downloadable assets. Besides, in some environments, it'll be a requirement to compile the driver of such a critical component. Learning how to use Falco Driverkit will help you to easily deploy Falco in more environments.

Falco – EBPF

Blog: Deploy Falco on a Talos cluster

Requirements

Set up the Talos cluster

Get the kubeconfig

Patch the cluster

Install Falco

Forward the audit logs to Falco

Visalize the alerts

Conclusion

Blog: Tracing System Calls Using eBPF - Part 2

Introduction

Uprobes

Uretprobes

Kprobes

Kretprobes

Conclusion

Blog: Introducing a framework for regression testing against Linux kernels

Components of a kernel testing framework

Ignite a Firecracker microVM

Kernel & Rootfs OCI images

Ansible Playbooks

Presenting test results

How we use it

What's next for the framework

Blog: Tracing System Calls Using eBPF - Part 1

Introduction:

BPF (Berkeley Packet Filter)

eBPF (Extended Berkeley Packet Filter)

Working on an eBPF program

Kernel Modules

So why does Falco use eBPF?

eBPF programs vs kernel modules

Safety and Isolation

Performance

Observability and Debugging

Attaching eBPF programs to hooks and events

Ring buffers

bpf_ringbuf_reserve

bpf_probe_read_user_str

bpf_ringbuf_submit

bpf_object__find_map_fd_by_name

bpf_program__attach_tracepoint

ring_buffer__new

ring_buf__consume

BTF (BPF Type Format)

Conclusion

Blog: Modern eBPF probe is ready to shine

Supported Syscalls

Advanced support checks

Buffers allocation

Least privileged mode

Try it out

Blog: Getting started with modern BPF probe in Falco

What's new 🗞️

CO-RE paradigm

BPF Ring Buffer map

BTF-enabled program

BPF global variables

BPF skeleton

Multi-arch support

Requirements ⛓️

Modern BPF in action 🏎️

1. Pre-built deb, rpm packages

2. Tar archive

3. Docker image

Current syscall support ⏲️

Next steps 🔮

Falco and modern BPF at eBPF summit 🐝

Blog: Falco Driverkit with Docker on Debian

Falco on Docker

Launching Falco as a container

Using Falco Driverkit

Creating a configuration file for the driver request

Launching Falco with the new driver

Loading the driver manually

Sharing the probe and driver with the Falco container

Loading the kernel module

Loading the eBPF Probe

Debugging

1. Pre-built `deb`, `rpm` packages