Falco – Kmod

Blog: Introducing a framework for regression testing against Linux kernels

Thu, 21 Sep 2023 00:00:00 +0000

There are a few foundational technologies that empower the Cloud Native ecosystem. Containers is one. And one of the basis for containerization is the Linux Kernel itself. With Falco, we are developing a runtime security tool that hooks directly in the kernel to collect information about the system and notifies about malicious behavior.
We have found the need to validate our drivers against various versions of the Linux kernel, to properly ensure that with each iteration of our drivers, supported kernels remained unaffected.
To elaborate, we lacked a means to guarantee that a new driver release could:

Successfully compile on multiple kernel versions.
Pass the eBPF verifier when executed on various kernel versions.
Operate as expected, such as retrieving kernel events, across multiple kernel versions.

To address this issue, we started a major intervention. Initially, a proposal was discussed and incorporated into the libs repository.

Since this was a pretty novel area, there were no pre-existing tools available to tackle it. Consequently, we embarked on the development of a completely new framework.
Allow us to introduce you to the kernel testing framework.

Components of a kernel testing framework

Considering the inherent characteristics of the challenge, we need to set up a complete virtual machine for each distinct kernel version.
These tests should be executed automatically each time new code is integrated into our drivers, serving as a means to promptly identify any issue or flaw in the tested kernel versions.
With these objectives in mind, our approach should fulfill the following requirements:

Rapid and cost-effective VM creation: the process of creating these virtual machines should be efficient and budget-friendly.
Effortless distribution of VM images: we should ensure easy sharing and deployment of the virtual machine images.
Parallel execution of tests on multiple VMs: tests should run concurrently on each virtual machine to expedite the process.
Reproducibility in local environments for debugging purposes: developers should be able to replicate the test environment locally to investigate and troubleshoot issues.
Straightforward and user-friendly presentation of the test results: they should be presented in a simple and intuitive manner to immediately spot failures.

Ignite a Firecracker microVM

Weave Ignite is used to provision the firecracker microVMs. Weave Ignite is an open source tool designed for lightweight and fast virtual machine management. It enables users to effortlessly create and manage virtual machines (VMs) for various purposes, such as development, testing, and experimentation. One of the main reasons why we chose to use this tool was its capability to create firecracker microVMs from kernels and rootfs packed as OCI images. Currently, we are using a patched version located at a forked repository. These patches were essential to enable the booting of kernels that necessitated the use of an initrd (initial ramdisk).

Kernel & Rootfs OCI images

Virtual machines consist of two essential layers: the kernel and the rootfs. These layers are packaged and distributed as OCI (Open Container Initiative) images. The kernel image encompasses the kernel that the virtual machine relies on, in contrast the rootfs image serves as the fundamental building block of a virtual machine, offering the essential filesystem necessary for booting the VM. Typically, these rootfs images incorporate a Linux distribution. For more info on how we build them please check the available images documentation.

Ansible Playbooks

Automation is accomplished through the utilization of Ansible. A collection of playbooks is responsible for:

Orchestrating the provisioning of microVMs.
Configuring the machines.
Retrieving the code to be tested.
Eliminating the microVMs once the testing process is completed.

Presenting test results

We wanted the test data to be publicly and easily accessible by anyone, thus we had to find a way to represent the test output.
Since there are 3 possible ways of instrumenting the kernel, that are using a kernel module or one of the available eBPF probes, the playbooks perform up to 3 tests. Taking into account that the modern eBPF probe is built in the Falco libraries, only 2 drivers need to be compiled. We have 3 possible results for each of them:

success, when the test goes fine
error, when the test fails
skipped, when the test is not runnable for the kernel (for example, skipping modern eBPF tests where it is unsupported)

The natural way of dealing with all of this was to develop a small tool that, given as input the output root folder, would generate a markdown matrix with the results.

While scrutinizing the first version of the markdown matrix, we understood that it would have been even better if errors were also attached to the markdown, allowing for a more streamlined visualization of the results.
This is the format we settled with; it can be found at libs github pages:

How we use it

We implemented a new Github action workflow in the libs repository that triggers on pushes to master, using an x86_64 and an aarch64 nodes with virtualization capabilities provided by the CNCF.
The workflow itself is very simple since it runs the testing framework on self-hosted nodes just like you would run it locally:

jobs:
test-kernels:
strategy:
fail-fast: false
matrix:
architecture: [X64, ARM64] # We use a matrix to run our job on both supported arch
# Since github actions do not support arm64 runners and they do not offer virtualization capabilities, we need to use self hosted nodes.
runs-on: [ "self-hosted", "linux", "${{matrix.architecture}}" ]
steps:
# We clone the kernel-testing repo, generate vars.yaml (ie: input options for the kernel-testing run)
# and run needed ansible playbooks one by one, directly on each node.
- name: Checkout
uses: actions/checkout@v3
with:
repository: falcosecurity/kernel-testing
ref: v0.2.3
- name: Generate vars yaml
working-directory: ./ansible-playbooks
run: |
LIBS_V=${{ github.event.inputs.libsversion }}
LIBS_VERSION=${LIBS_V:-${{ github.ref_name }}}
cat > vars.yml <<EOF
run_id: "id-${{ github.run_id }}"
output_dir: "~/ansible_output_${{ github.run_id }}"
repos:
libs: {name: "falcosecurity-libs", repo: "https://github.com/falcosecurity/libs.git", version: "$LIBS_VERSION"}
EOF
- name: Bootstrap VMs
working-directory: ./ansible-playbooks
run: |
ansible-playbook bootstrap.yml --extra-vars "@vars.yml"
- name: Common setup
working-directory: ./ansible-playbooks
run: |
ansible-playbook common.yml --extra-vars "@vars.yml"
- name: Prepare github repos
working-directory: ./ansible-playbooks
run: |
ansible-playbook git-repos.yml --extra-vars "@vars.yml"
- name: Run scap-open tests
working-directory: ./ansible-playbooks
run: |
ansible-playbook scap-open.yml --extra-vars "@vars.yml" || :
# Once test finished, we collect its output folder and upload it to the github workflow space
- name: Tar output files
run: |
tar -cvf ansible_output.tar ~/ansible_output_${{ github.run_id }}
- uses: actions/upload-artifact@v3
with:
name: ansible_output_${{matrix.architecture}}
path: ansible_output.tar
# We then build the matrix and upload them too
- name: Build matrix_gen
working-directory: ./matrix_gen
env:
GOPATH: /root/go
GOCACHE: /root/.cache/go-build
run: |
go build .
- name: Generate new matrix
working-directory: ./matrix_gen
run: |
./matrix_gen --root-folder ~/ansible_output_${{ github.run_id }} --output-file matrix_${{matrix.architecture}}.md
- uses: actions/upload-artifact@v3
with:
name: matrix_${{matrix.architecture}}
path: ./matrix_gen/matrix_${{matrix.architecture}}.md
# Always run the cleanup playbook to avoid leaving garbage on the nodes
- name: Cleanup
if: always()
working-directory: ./ansible-playbooks
run: |
ansible-playbook clean-up.yml --extra-vars "@vars.yml" || :

In the Generate new matrix step, the kernel matrix gets generated and then uploaded.
Once this workflow runs successfully for both architectures, another workflow gets triggered,
that is responsible for generating and pushing updated Github pages.
The end result can be seen at https://falcosecurity.github.io/libs/matrix/.

Moreover, the kernel-testing workflow gets also triggered on each driver's tag; then a supplementary workflow takes care of attaching matrixes to the release body;
here is an example: https://github.com/falcosecurity/libs/releases/tag/6.0.0%2Bdriver.

Pretty nice, uh?

What's next for the framework

There are quite a few gaps that still need to be addressed by our framework. First of all, the images being used by Ignite to spawn FireCracker VMs are still under a development Docker repository and need to be moved under Falcosecurity.
Moreover, we need to implement a CI to automatically build and push those images.

As previously said, the kernel tests are currently running scap-open binary to check whether any event gets received. It is a small libscap C example that loads a driver and waits for events, nothing more.
It would be great to run drivers tests instead, to fully test the expected behavior of the drivers.

Finally, an utopian idea: imagine if we were able to run kernel-crawler to fetch kernel images, and then automatically build new kernel testing matrix entries for newly discovered images.
This would mean that our kernel testing matrix coverage increases steadily week after week, giving users even more guarantees about the stability of the Falco drivers!

Here is the libs tracking issue: https://github.com/falcosecurity/libs/issues/1224.

We would love to hear back from you! If you try out the framework and find any issues, please file them at https://github.com/falcosecurity/kernel-testing/issues. If you want to help us to improve, please join our Slack, and feel free to open a Pull Request!

Blog: Falco Driverkit with Docker on Debian

Mon, 05 Sep 2022 00:00:00 +0000

We use different technologies on a daily basis. Tools like Vagrant, Terraform, Ansible, and many more allow us to create and destroy digital resources in a matter of minutes, if not seconds. However, if you keep changing your running environment, you might also need to calibrate your workloads to these new changes. This is especially true when you deploy applications tightly dependent on the operating system.

In other words, every time you deploy an application like Falco there's a chance that you need to compile a new module or eBPF probe to get along with the current underlying kernel. This is the first of a series of posts where you will learn some interesting techniques related to how Falco generates the much needed driver and how you can make it available for your deployments.

Falco on Docker

There are many ways to run Falco: as a service, as a local container, as a Pod in Kubernetes, etc. Either way, if what we want to do is use Falco to detect threats based on syscalls, we will need a driver that has been compiled for the specific kernel running on the machine, be it a physical machine, a virtual one, or a Kubernetes node in the cloud.

Launching Falco as a container

The Falco image embeds a script, /usr/bin/falco-driver-loader, that will automatically try to find and download a kernel module or an eBPF probe. If that wasn't possible, it might try to compile it inside the container itself. We will learn a bit more about this process and how to control it.

Here is the output of a fresh instance of falco running on our local docker service:

$ docker run -it --privileged \
 --name falco-driver-test \
 -v /dev:/host/dev \
 -v /etc:/host/etc:ro \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 docker.io/falcosecurity/falco:0.32.2

Unable to find image 'falcosecurity/falco:0.32.2' locally
0.32.2: Pulling from falcosecurity/falco
7e6a53d1988f: Pull complete
... output omitted
f3102eb3e85f: Pull complete
Digest: sha256:5ceb23e5baae9c86fc0b160fed397facd2074ca398b770878adbb9c6d049d8a8
Status: Downloaded newer image for falcosecurity/falco:0.32.2

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.32.2, driver version=2.0.0+driver
* Running falco-driver-loader with: driver=module, compile=yes, download=yes

================ Cleaning phase ================

* 1. Check if kernel module 'falco' is still loaded:
- OK! There is no 'falco' module loaded.

* 2. Check all versions of kernel module 'falco' in dkms:
- There are some versions of 'falco' module in dkms.

* 3. Removing all the following versions from dkms:
2.0.0+driver

- Removing 2.0.0+driver...

------------------------------
Deleting module version: 2.0.0+driver
completely from the DKMS tree.
------------------------------
Done.

- OK! Removing '2.0.0+driver' succeeded.


[SUCCESS] Cleaning phase correctly terminated.

================ Cleaning phase ================

* Looking for a falco module locally (kernel 5.10.0-14-cloud-amd64)
* Trying to download a prebuilt falco module from https://download.falco.org/driver/2.0.0%2Bdriver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
curl: (22) The requested URL returned error: 404
Unable to find a prebuilt falco module
* Trying to dkms install falco module with GCC /usr/bin/gcc
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"

Creating symlink /var/lib/dkms/falco/2.0.0+driver/source ->
 /usr/src/falco-2.0.0+driver

DKMS: add completed.
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2.0.0+driver/build/make.log (with GCC /usr/bin/gcc)
* Trying to dkms install falco module with GCC /usr/bin/gcc-8
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2.0.0+driver/build/make.log (with GCC /usr/bin/gcc-8)
* Trying to dkms install falco module with GCC /usr/bin/gcc-6
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2.0.0+driver/build/make.log (with GCC /usr/bin/gcc-6)
* Trying to dkms install falco module with GCC /usr/bin/gcc-5
DIRECTIVE: MAKE="'/tmp/falco-dkms-make'"
* Running dkms build failed, couldn't find /var/lib/dkms/falco/2.0.0+driver/build/make.log (with GCC /usr/bin/gcc-5)
* Trying to load a system falco module, if present
Consider compiling your own falco driver and loading it or getting in touch with the Falco community
2022-08-31T12:00:02+0000: Falco version 0.32.2
2022-08-31T12:00:02+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:00:02+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:00:02+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:00:02+0000: Starting internal webserver, listening on port 8765
2022-08-31T12:00:02+0000: Unable to load the driver.
2022-08-31T12:00:02+0000: Runtime error: error opening device /host/dev/falco0. Make sure you have root credentials and that the falco module is loaded.. Exiting.

There are some important reads from this output:

The driver version this image tries to load is the 2.0.0+driver. This information will be really useful when we need to compile and share the driver with the falco container.
By default, the container will look for a kernel module. It is possible to switch to an eBPF probe by using an environment variable, as you'll see later in this post.
The falco-driver-loader script always removes the driver from memory and tries to load a current one. This is done for security reasons and the way to avoid that is not running this script when creating the container. More on this later, too.
After looking in the system for a previously installed driver, the script tries to download it from the URL https://download.falco.org. Unfortunately, it doesn't seem to be able to find it and falls back to the local compilation method.
When the script tries to compile the driver inside the container, it doesn't succeed because we haven't fulfilled one important prerequisite: installing the kernel headers on the host machine. In this post, we won't address that method but you can always refer to the documentation.

Using Falco Driverkit

As mentioned, there are different ways to obtain a valid kernel: downloading it from https://download.falco.org, compiling it via the falco-driver-loader script, or the method we'll explain here: using driverkit.

We don't intend this post to be an exhaustive guide to driverkit. That's also why we've chosen a relatively easy and tested target operating system: Debian.

First of all, we need the driverkit tool which we'll compile ourselves. We can download the source code from https://github.com/falcosecurity/driverkit.

When compiling a tool, we like using a temporary container. In this case, we'll start our container using the docker.io/golang:1.19 image and a sleep process until we're done. The ./driverkit directory will help us to extract the binary to the host filesystem. Feel free to use any other method you prefer, like docker cp.

# This directory will contain the driverkit binary
# once it is compiled inside the container

$ mkdir ./driverkit

# Container with Go tools
$ docker run --rm -d --name golang-compiler \
 -v $PWD/driverkit:/export \
 golang:1.19 sleep infinity

Check that the container has been successfully created and still runs:

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1ff943cbf7f9 golang:1.19 "sleep infinity" 4 seconds ago Up 3 seconds golang-compiler

Next, create a shell with a terminal in the container:

$ docker exec -it golang-compiler /bin/bash

Remember, you are in the container context now. Whatever you do here will be lost unless you copy it to the /export directory. We will clone the driverkit code and compile it using the following commands:

git clone https://github.com/falcosecurity/driverkit
cd driverkit && make
cp _output/bin/driverkit /export/driverkit
exit

Once we are done with the Golang container, we can stop it and it'll be automatically removed thanks to the --rm parameter that we used to start it.

docker stop golang-compiler -t0

Creating a configuration file for the driver request

Time to create a configuration file. Do you remember the driver version: 2.0.0+driver? We will use that and additional information to create the configuration file.

# We've included some VARIABLES that will help you understand
# where the different values come from and what they represent

export DRIVER_VERSION=2.0.0+driver
export DRIVER_TARGET=debian
export DRIVER_ARCH=$(arch)

export KERNEL_VERSION=$(uname -v| cut -f1 -d' ' | tr -d \#)
export KERNEL_RELEASE=$(uname -r)
export DRIVER_NAME=falco
export PROBE_FILE=${DRIVER_NAME}_${DRIVER_TARGET}_${KERNEL_RELEASE}_${KERNEL_VERSION}.o
export MODULE_FILE=${DRIVER_NAME}_${DRIVER_TARGET}_${KERNEL_RELEASE}_${KERNEL_VERSION}.ko

mkdir -p drivers/${DRIVER_VERSION}/${DRIVER_ARCH}/

# Creating the actual file that we will pass to driverkit

cat > debian.yaml << EOF
target: ${DRIVER_TARGET}
driverversion: ${DRIVER_VERSION}
kernelrelease: ${KERNEL_RELEASE}
kernelversion: ${KERNEL_VERSION}
output:
 module: ./drivers/${DRIVER_VERSION}/x86_64/${MODULE_FILE}
 probe: ./drivers/${DRIVER_VERSION}/x86_64/${PROBE_FILE}
EOF

The resulting file should look like this:

$ cat debian.yaml
target: debian
driverversion: 2.0.0+driver
kernelrelease: 5.10.0-14-cloud-amd64
kernelversion: 1
output:
 module: ./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
 probe: ./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o

In case you want to use a version previous to Falco 0.32.2 you might need to remove the x86_64/ string from the probe path. This is due to the expected path inside the falco-driver-loader script. These paths will be offered via an HTTP server at a later stage, so make sure they match in both steps.

This same file is the one we will pass to driverkit. If the driver is compiled satisfactorily, we should see a similar output in some seconds. Be patient.

$ ./driverkit/driverkit docker -c debian.yaml

INFO using config file file=debian.yaml
INFO driver building, it will take a few seconds processor=docker
INFO kernel module available path=./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
INFO eBPF probe available path=./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o

Make sure you use either the .yml or .yaml suffix. Otherwise, you'll get an error like:

$ ./driverkit/driverkit docker -c debian.unknown-ext
Error: exiting for validation errors
Usage:
 driverkit docker [flags]

Flags:

... output omitted

FATA error executing driverkit error="exiting for validation errors"

Alternatively, we could have used a bunch of parameters in the command line, like:

# Don't forget to \, to let the command continue after each line

$ ./driverkit/driverkit docker \
 --architecture amd64 \
 --target ${DRIVER_TARGET} \
 --driverversion ${DRIVER_VERSION} \
 --kernelversion ${KERNEL_VERSION} \
 --kernelrelease ${KERNEL_RELEASE} \
 --output-probe ./drivers/${DRIVER_VERSION}/x86_64/${PROBE_FILE} \
 --output-module ./drivers/${DRIVER_VERSION}/x86_64/${MODULE_FILE}

INFO driver building, it will take a few seconds processor=docker
INFO kernel module available path=./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
INFO eBPF probe available path=./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o

Either way, if driverkit manages to compile the drivers, you can continue with the next step. Otherwise, you might need to adjust some of the parameters in the configuration or even customize your builder image, but we will explain that in a different post where we will deep dive into driverkit.

Launching Falco with the new driver

There are different ways to load the driver when running Falco. We'll show you two of them: loading them manually and leaving this action to the script embedded in the container image.

Loading the driver manually

A kernel module only needs to be loaded once. So, if we load it manually before starting the container, Falco doesn't need to do it again.

There are two ways of achieving that, and both require avoiding the execution of the falco-driver-loader script:

Setting the SKIP_DRIVER_LOADER environment variable to any value when creating the container. By doing so, the container entrypoint will skip the existing falco-driver-loader script.
Using the image docker.io/falco/falco-no-driver, which doesn't contain that script.

First, try to load the driver on the host. Look for the .ko file in the directory structure we created and load it using insmod, for instance. If the compilation was successful and the kernel version chosen was the right one, you shouldn't see any message once the module is loaded. Don't forget to do it with the user root (i.e., via sudo).

$ find $HOME/drivers -type f
drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o
drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko

$ sudo insmod drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
$ lsmod | grep -i falco
falco 741376 0

This first method of starting the falco container will use the docker.io/falco/falco:0.32.2 image, passing the SKIP_DRIVER_LOADER variable. We've set it to one but the script doesn't check its value.

Observe that we're removing any existing container with that name before starting ours, but the container image remains.

$ docker rm -f falco-driver-test # Ignore any failure here
falco-driver-test

$ docker run -it --privileged \
 --name falco-driver-test \
 -e SKIP_DRIVER_LOADER=1 \
 -v /dev:/host/dev \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 docker.io/falcosecurity/falco:0.32.2

2022-08-31T12:07:30+0000: Falco version 0.32.2
2022-08-31T12:07:30+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:07:30+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:07:30+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:07:30+0000: Starting internal webserver, listening on port 8765

The second method uses the docker.io/falco/falco-no-driver image, so, as you can expect, it won't try to reload the driver this time.

This time, Docker will pull the image since we hadn't used it yet.

$ docker rm -f falco-driver-test # Ignore any failure here
falco-driver-test

$ docker run -i -t --privileged \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -v /dev:/host/dev \
 -v /proc:/host/proc:ro \
 --name falco-driver-test \
 docker.io/falcosecurity/falco-no-driver:0.32.2

Unable to find image 'falcosecurity/falco-no-driver:0.32.2' locally
0.32.2: Pulling from falcosecurity/falco-no-driver
1efc276f4ff9: Pull complete
e34e1870ff2c: Pull complete
Digest: sha256:14e6d3da56fe607ff9b0bfe91ec812ab4f4b030cea3ed88a2d31ac9b31f97fb4
Status: Downloaded newer image for falcosecurity/falco-no-driver:0.32.2

2022-08-31T12:12:40+0000: Falco version 0.32.2
2022-08-31T12:12:40+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:12:40+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:12:40+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:12:40+0000: Starting internal webserver, listening on port 8765

This method is a bit more complicated than the previous one but will also give us the flexibility of deploying falco at scale.

The idea is simple though. After starting your favorite webserver and publishing the ./drivers directory that we created before, we'll tell the falco container to use it as a repository and download the driver from there.

To keep things clean, we've used the docker.io/python:latest container image, which includes the Python module http.server. If you have Python already installed on your system, you can use it directly. Just remember to define a port accessible to the falco container and pass the right IP address.

$ docker run -d \
 --name falco-drivers-web \
 -v $PWD/drivers:/data:ro \
 docker.io/python:latest \
 bash -c "cd /data && python -m http.server"

Unable to find image 'python:latest' locally
latest: Pulling from library/python
1671565cc8df: Pull complete
... output omitted
4334b2fe8293: Pull complete
Digest: sha256:745efdfb7e4aac9a8422bd8c62d8bc35a693e8979a240d29677cb03e6aa91052
Status: Downloaded newer image for python:latest
f94cb601f85c312d62aab3e116619558239bada9f5d05e971fe26c0206828b6b

Our python web server is now available and offers the drivers to any local container that might need them. Retrieve the IP address of this container for later use:

$ docker inspect falco-drivers-web --format '{{ .NetworkSettings.IPAddress }}'
172.17.0.2

# Assign it to a variable for later use
$ export DRIVERS_REPO=$(docker inspect falco-drivers-web \
 --format '{{ .NetworkSettings.IPAddress }}'):8000

$ echo $DRIVERS_REPO
172.17.0.2:8000

It's always a good practice to test that the drivers are in the right place and accessible through the webserver.

# This is the checksum of the local files

$ find ./drivers -type f -name "*o" -exec cksum {} \;
3873827283 843080 ./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64.ko
914371530 4980536 ./drivers/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64.o

# This is the checksum of the files retrieved through HTTP

$ find ./drivers -type f -name "*o" | while read FILE
 do
 URL=$(echo ${FILE} | sed -e 's,./drivers,'${DRIVERS_REPO}',')
 echo "$(curl -s http://${URL} | cksum) http://${URL}"
 done
3873827283 843080 localhost:8000/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64.ko
914371530 4980536 localhost:8000/2.0.0+driver/x86_64/falco_debian_5.10.0-14-cloud-amd64.o

As you can see, they are accessible and identical.

These values will be different depending on the version of the kernel and the Falco drivers.

Loading the kernel module

Let's start with the kernel module. In this case, the only variable we need to pass is the DRIVERS_REPO one, which has been carefully prepared in the previous step.

$ docker rm -f falco-driver-test
falco-driver-test

$ docker run -it --privileged \
 --name falco-driver-test \
 -v /dev:/host/dev \
 -v /etc:/host/etc:ro \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -e DRIVERS_REPO=${DRIVERS_REPO} \
 docker.io/falcosecurity/falco:0.32.2

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.32.2, driver version=2.0.0+driver
* Running falco-driver-loader with: driver=module, compile=yes, download=yes

================ Cleaning phase ================

* 1. Check if kernel module 'falco' is still loaded:
- OK! There is no 'falco' module loaded.

* 2. Check all versions of kernel module 'falco' in dkms:
- There are some versions of 'falco' module in dkms.

* 3. Removing all the following versions from dkms:
2.0.0+driver

- Removing 2.0.0+driver...

------------------------------
Deleting module version: 2.0.0+driver
completely from the DKMS tree.
------------------------------
Done.

- OK! Removing '2.0.0+driver' succeeded.


[SUCCESS] Cleaning phase correctly terminated.

================ Cleaning phase ================

* Looking for a falco module locally (kernel 5.10.0-14-cloud-amd64)
* Trying to download a prebuilt falco module from 172.17.0.2:8000/2.0.0%2Bdriver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
* Download succeeded
* Success: falco module found and inserted
2022-08-31T12:36:29+0000: Falco version 0.32.2
2022-08-31T12:36:29+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:36:29+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:36:29+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:36:29+0000: Starting internal webserver, listening on port 8765

It's a similar output as before, but this time we can see:

* Trying to download a prebuilt falco module from 172.17.0.2:8000/2.0.0%2Bdriver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.ko
* Download succeeded
* Success: falco module found and inserted

The module has been successfully loaded and Falco can start properly.

Loading the eBPF Probe

For this, we will make use of another variable, FALCO_BPF_PROBE. Like it happened with the SKIP_DRIVER_LOADER variable, its value is not as relevant as the fact that the variable had been defined. We also need to keep the DRIVERS_REPO variable, since the falco-driver-loader script will look for the probe in that URL.

$ docker rm -f falco-driver-test
falco-driver-test

$ docker run -it --privileged \
 --name falco-driver-test \
 -v /dev:/host/dev \
 -v /etc:/host/etc:ro \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -e DRIVERS_REPO=${DRIVERS_REPO} \
 -e FALCO_BPF_PROBE="" \
 docker.io/falcosecurity/falco:0.32.2

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.32.2, driver version=2.0.0+driver
* Running falco-driver-loader with: driver=bpf, compile=yes, download=yes
* Mounting debugfs
* Trying to download a prebuilt eBPF probe from 172.17.0.2:8000/2.0.0%2Bdriver/x86_64/falco_debian_5.10.0-14-cloud-amd64_1.o
* Skipping compilation, eBPF probe is already present in /root/.falco/falco_debian_5.10.0-14-cloud-amd64_1.o
* eBPF probe located in /root/.falco/falco_debian_5.10.0-14-cloud-amd64_1.o
* Success: eBPF probe symlinked to /root/.falco/falco-bpf.o
2022-08-31T12:58:10+0000: Falco version 0.32.2
2022-08-31T12:58:10+0000: Falco initialized with configuration file /etc/falco/falco.yaml
2022-08-31T12:58:10+0000: Loading rules from file /etc/falco/falco_rules.yaml:
2022-08-31T12:58:10+0000: Loading rules from file /etc/falco/falco_rules.local.yaml:
2022-08-31T12:58:10+0000: Starting internal webserver, listening on port 8765

This time the output is easier to read: The driver is set to bpf, the URL of the HTTP container points to our local webserver, and it also shows where it downloads the probe before starting Falco.

Debugging

As a final tip, if you want to start a container based on the regular falco image to test the falco-driver-loader script, we recommend starting the container with the --entrypoint /bin/bash parameter. This will keep the /docker-entrypoint.sh script from being executed (that one triggers /usr/bin/falco-driver-loader) and you'll have a much more comfortable environment to work with.

$ docker run -it --privileged \
 --name falco-driver-test \
 -v /dev:/host/dev \
 -v /etc:/host/etc:ro \
 -v /proc:/host/proc:ro \
 -v /var/run/docker.sock:/host/var/run/docker.sock \
 -e DRIVERS_REPO=${DRIVERS_REPO} \
 --entrypoint /bin/bash \
 docker.io/falcosecurity/falco:0.32.2

root@e0c391e0cee1:/#

Conclusion

Falco requires tapping into the kernel to be able to retrieve useful information from it. For that, it has two methods: loading a kernel module in the traditional way, or using an eBPF probe. Both of them instrumentalize the kernel and provide the functionality to retrieve the relevant data.

Due to the infinite number of combinations of Linux kernels and distributions, it is extremely difficult to offer all possible kernels as downloadable assets. Besides, in some environments, it'll be a requirement to compile the driver of such a critical component. Learning how to use Falco Driverkit will help you to easily deploy Falco in more environments.