Falco – Kernel Events

Docs: Kernel Events Architecture

Mon, 01 Jan 0001 00:00:00 +0000

This document describes the overall architecture that allows events from kernel sources to be ingested by Falco, how to use the libraries to inspect the data collection flow and how Falco manages the boundary between the kernel and userspace. In order to make Falco compatible with a very large number of Linux Kernel versions, the internal APIs and low level communication mechanisms that are employed to cross the kernel and userspace boundary vary greatly between driver types and may be different between driver versions or kernel versions. However, they all implement the same event collection interface as described below.

How Falco interacts with kernel components

The component of the Falco libraries that gathers data from the syscalls and interacts with the kernel is called libscap. Internally, it implements all functionality required to use the drivers to collect kernel events.

When using the kernel module or legacy eBPF probe (deprecated), the driver will need to be installed and deployed separately as a kernel object or probe, while the modern eBPF probe can be installed directly by libscap.

Upon connection to its kernel counterpart, libscap will need to negotiate the API Version and Schema Version that the driver recognizes. These versions are expressed with a semver subset and are documented in the libs repository.

The API version refers to the communication mechanism between the kernel and userspace. Every driver has a different communication mechanism which changes between versions. The kernel module may use ioctls and a ring buffer, while the eBPF probe can use maps and different APIs depending on the kernel version and eBPF probe edition. Since some drivers can be deployed separately from Falco, at startup libscap will verify if the driver it's connecting to is compatible.
The Schema version refers to the type of events that the specific driver supports. The Syscall Events documentation page shows the list of fields that are supported for each version of Falco. Every time that list changes the version number is updated as well.

When running Falco it is possible to verify the currently compatible version numbers with falco --version. For instance, this is the output for Falco 0.35.1:

# falco --version
2023-07-01T16:23:43+0000: Falco version: 0.35.1 (x86_64)
2023-07-01T16:23:43+0000: Falco initialized with configuration file: /etc/falco/falco.yaml
Falco version: 0.35.1
Libs version: 0.11.3
Plugin API: 3.0.0
Engine: 17
Driver:
API version: 4.0.0
Schema version: 2.0.0
Default driver: 5.0.1+driver

Once Falco is running, a stream of events is returned directly from the kernel. libscap's API allow the data to flow with a consistent format from the kernel to userspace.

The main interface that governs this is the scap event format. Once the chosen driver is loaded and initialized in the kernel, the events are encoded with a specific header and a payload:

struct ppm_evt_hdr {
uint64_t ts; /* timestamp, in nanoseconds from epoch */
uint64_t tid; /* the tid of the thread that generated this event */
uint32_t len; /* the event len, including the header */
uint16_t type; /* the event type */
uint32_t nparams; /* the number of parameters of the event */
};

The payload contains an array of lengths of each parameter followed by the content of the parameters themselves. The parameter type is a numeric identifer that maps with each event documented in the reference.

For example, the dup3 event is defined in the reference as:

dup3(FD res, FD oldfd, FD newfd, FLAGS32 flags: O_LARGEFILE, O_DIRECTORY, O_DIRECT, O_TRUNC, O_SYNC, O_NONBLOCK, O_EXCL, O_DSYNC, O_APPEND, O_CREAT, O_RDWR, O_WRONLY, O_RDONLY, O_CLOEXEC, O_NONE, O_TMPFILE)

Meaning that its encoding will be composed of an header containing the timestamp and tid, nparams will be 4 and the complete encoding will be:

[header] [uint16(8)] [uint16(8)] [uint16(8)] [uint16(32)] [res] [oldfd] [newfd] [flags]

Use scap-open to inspect kernel data collection

Contributors and expert users can find a tool called scap-open in the libs repo. This tool allows to dump raw events from a variety of drivers. Building and usage instructions are included in the repository.

Docs: Advanced Performance Tuning

Mon, 01 Jan 0001 00:00:00 +0000

This document provides advanced performance tuning options for the syscall data source in Falco. It is intended for users who want to optimize the performance of their Falco deployment by customizing the syscall monitoring behavior.

Adaptive syscalls selection

Falco provides users flexibility to select different syscall monitoring behaviors tailored to their specific use cases. These options offer various degrees of control over system calls, directly configured through the falco.yaml file.

This section outlines the available configurations and their implications.

Default behavior

By default, Falco traces syscalls derived from:

Syscalls explicitly required by enabled Falco rules.
A predefined set essential for maintaining Falco's internal state engine, defined at compile-time.

With the default configuration:

base_syscalls.custom_set: []
base_syscalls.repair: false
base_syscalls.all: false

This ensures accurate state engine management but offers no end-user customization of the additional syscalls.

Monitoring all syscalls (`base_syscalls.all`)

Setting this option to true enables monitoring all events supported by Falco, including typically ignored events such as write:

base_syscalls.all: true

Use with caution, as this may negatively impact performance due to increased resource usage.

User-defined syscall set (`base_syscalls.custom_set`)

CAUTION: Misconfiguration may result in incomplete event logs or disrupt Falco's tracing capabilities.

This option allows you to explicitly define an additional set of syscalls to trace, supplementing those required by active Falco rules:

base_syscalls.custom_set: [clone, clone3, fork, execve, execveat, close]

It offers fine-grained control and can help optimize resource utilization according to your threat model and performance constraints.

Recommended syscall sets for typical scenarios:

Process monitoring: [clone, clone3, fork, vfork, execve, execveat, close]
Networking monitoring: [clone, clone3, fork, vfork, execve, execveat, close, socket, bind, getsockopt]
Accurate UID/GID tracking: Add [setresuid, setsid, setuid, setgid, setpgid, setresgid, capset, chdir, chroot, fchdir] to the relevant set.

Negative notation ("!syscall_name") is supported to explicitly exclude specific syscalls.

Automatic state engine management (`base_syscalls.repair`)

Recommended for most scenarios, enabling this option allows Falco to automatically select the minimal necessary set of syscalls beyond those explicitly required by enabled rules:

base_syscalls.repair: true
base_syscalls.custom_set: []
base_syscalls.all: false

This option ensures Falco's internal state engine integrity with minimal performance overhead, automatically incorporating best-practice syscall configurations.

Scenarios

Different configurations address various monitoring scenarios effectively:

Monitoring spawned processes under resource constraints
- Default: Insufficient
- custom_set and repair: Both viable, but repair is recommended for automatic correctness.
Monitoring spawned processes and network activity, excluding file opens
- Default: Insufficient
- custom_set and repair: Both suitable, with repair ensuring automatic correctness without manual intervention.
Flexible configurability for tailored monitoring
- Useful in environments requiring selective monitoring to optimize resources.
- Allows coexistence with other monitoring tools by minimizing duplication of work.
Comprehensive syscall monitoring
- All three configurations (default, custom_set, repair) can achieve complete syscall monitoring.
- Choice depends on user preference and performance trade-offs.

Notes

Use falco -i to list all events typically ignored in the default configuration.
Events marked EF_OLD_VERSION are not generated during live monitoring but may appear in .scap files.

Docs: Actions For Dropped System Call Events

Mon, 01 Jan 0001 00:00:00 +0000

Introduction

With the enhancements introduced in v0.15.0, Falco can now intelligently detect dropped system call events and take remedial actions, such as alerting or even exiting Falco entirely. When system call events are dropped, Falco might encounter problems building its internal view of the processes, files, containers, and orchestrator metadata in use, which in turn might affect the rules that depend on that metadata. The explicit signals that Falco now provides make it easier to detect dropped system calls.

For more information on this feature, see our blog post on CVE-2019-8339.

Implementation

Every second, Falco reads system call event counts that are populated by the kernel module/eBPF program. The reading includes the number of system calls processed, and most importantly, the number of times the kernel tried to write system call information to the shared buffer between the kernel and user space, but found the buffer was full. These failed write attempts are considered dropped system call events.

When at least one dropped event is detected, Falco takes one of the following actions:

ignore: no action is taken. If an empty list is provided, ignore is assumed.
log: log a CRITICAL message noting that the buffer was full.
alert: emit a Falco alert stating that the buffer was full.
exit: exit Falco with a non-zero rc.

Given below are a sample log message, an alert, and an exit message:

Wed Mar 27 15:28:22 2019: Falco initialized with configuration file /etc/falco/falco.yaml
Wed Mar 27 15:28:22 2019: Loading rules from file /etc/falco/falco_rules.yaml:
Wed Mar 27 15:28:24 2019: Falco internal: syscall event drop. 1 system calls dropped in last second.
15:28:24.000207862: Critical Falco internal: syscall event drop. 1 system calls dropped in last second.(n_drops=1 n_evts=1181)
Wed Mar 27 15:28:24 2019: Falco internal: syscall event drop. 1 system calls dropped in last second.
Wed Mar 27 15:28:24 2019: Exiting.

Actions Rate Throttling

To reduce the likelihood of a flood of log messages/alerts, Falco provides an alert throttling mechanism disabled by default. This feature can be enabled through the Falco configuration (see the outputs entry).

Before v0.33.0 this feature was enabled by default.

Configuration

The actions to take on a dropped system call event and the throttling parameters for the token bucket are configurable in the file falco.yaml. You can find them in syscall_event_drops.

Docs: Generating sample events

Mon, 01 Jan 0001 00:00:00 +0000

If you'd like to check if Falco is working properly, we have the event-generator tool that can perform an activity for both our syscalls and k8s audit related rules.

The tool provides a command to run either some or all sample events.

event-generator run [regexp]

Without arguments it runs all actions, otherwise only those actions matching the given regular expression.

The full command line documentation is here.

Downloads

Artifacts		Version
binaries	download link
container images	`docker pull falcosecurity/event-generator:latest`

Sample events

WARNING

Since some commands might alter your system, we strongly recommend that you run the program within a container (see below).
For example, some actions modify files and directories below /bin, /etc, /dev, etc.

System Call Activity

The syscall collection performs a variety of suspect actions that are detected by the default Falco ruleset.

docker run -it --rm falcosecurity/event-generator run syscall --loop

The above command loops forever, incessantly generating a sample event each second.

Kubernetes Auditing Activity

The k8saudit collection generates activity that matches the k8s audit event ruleset.

event-generator run k8saudit --loop

The above command loops forever, creating resources in the current namespace and deleting them after each iteration. Use the --namespace option to choose a different namespace.

Running the Event Generator in K8s

We've also provided a helm chart that make it easy to run the event generator in K8s Clusters.

First thing, we need to add the falcosecurity charts repository:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Once you have the helm repo configured, you can run the following to create the necessary objects in the event-generator namespace and then generate events continuously:

helm install event-generator falcosecurity/event-generator \
 --namespace event-generator \
 --create-namespace \
 --set config.loop=false \
 --set config.actions=""

The above command applies to the event-generator namespace. Use the --namespace option to deploy in a different namespace. Events will be generated in the same namespace.

You can also find more examples in the event-generator and charts repositories.

Falco – Kernel Events

Docs: Kernel Events Architecture

How Falco interacts with kernel components

Use scap-open to inspect kernel data collection

Docs: Advanced Performance Tuning

Adaptive syscalls selection

Default behavior

Monitoring all syscalls (base_syscalls.all)

User-defined syscall set (base_syscalls.custom_set)

Automatic state engine management (base_syscalls.repair)

Scenarios

Notes

Docs: Actions For Dropped System Call Events

Introduction

Implementation

Actions Rate Throttling

Configuration

Docs: Generating sample events

Downloads

Sample events

System Call Activity

Kubernetes Auditing Activity

Running the Event Generator in K8s

Monitoring all syscalls (`base_syscalls.all`)

User-defined syscall set (`base_syscalls.custom_set`)

Automatic state engine management (`base_syscalls.repair`)