Falco – Troubleshooting

Docs: Performance

Mon, 01 Jan 0001 00:00:00 +0000

First and foremost, if something goes seriously wrong during Falco deployment, it's usually noticeable immediately. On a longer time scale, continuous performance monitoring and quality assurance, driven by the right metrics, ensure Falco functions as expected 24/7.

As a security tool, Falco requires checking for a healthy deployment on multiple dimensions:

Resource utilization and system impact: Strive to minimize compute overhead while ensuring the desired monitoring scope is achieved.
Disruption/upgrades: Ensure minimal downtime and avoid interruptions to the service, minimizing restarts.
Data quality assurance: Verify that Falco outputs contain the desired quality with little to no missing data.
End-to-end data pipeline testing: Ensure alerts reach their end destination as quickly as possible.
Security monitoring capabilities: Adopting the right Falco rules and resilience to bypasses by attackers.

The Falco Project provides guidance on some of these aspects, and there are ongoing long-term efforts, including a partnership with the CNCF TAG Environmental Sustainability, aimed at enhancing Falco's performance and assessing its impact on the system. These efforts are intended to make it easier for adopters to assess the actual impact on their systems, enabling them to make informed decisions about the cost-security monitoring tradeoffs.

Resource Utilization and System Impact

The Falco Project provides native support for measuring resource utilization and statistics, including event drop counters, kernel tracepoint invocation counters, timeouts, and internal state handling. More detailed information is given in the Falco Metrics Guide.

CPU and Memory Utilization

On top of the mind for SREs or system admins is how much CPU and memory Falco will utilize on their hosts. They need to assess whether the cost is justified. To maintain excellent relationships with infrastructure teams, setting resource limits for your Falco deployment is crucial. This can be done through systemd or daemonset limits in a Kubernetes environment.

This is an essential consideration because running a kernel tool always comes with specific challenges and considerations. For example, it could slow down the kernel or the request rates of applications.

Top metrics:

CPU usage: Typically measured as a percentage of one CPU, it can be compared with the number of available CPUs on the host. Falco's hot path is single-threaded, so it should not be able to exceed the capacity of one full CPU.
Memory RSS: Resident Set Size is the portion of memory held in RAM by a process.
Memory VSZ: Virtual Memory Size is the total memory allocated to a process, including both RAM and swap space.
container_memory_working_set_bytes in Kubernetes settings: This is almost equivalent to the cgroups container memory_used metric natively exposed in Falco metrics.

Beyond monitoring the tool's utilization, check if your applications perform as before. This evaluation could include overall network, I/O, or general contention metrics.

Read Falco Metrics next.

Server Load and Falco Event Drops

A common misconception is to think that Falco has constant resource utilization. However, that is not accurate. Falco's utilization is directly dependent on the current workload on the host. The more system calls the applications make, the more work Falco has to handle. You can read our Kernel Testing Framework Proposal for more insights into this topic.

Read Falco Is Dropping Syscalls Events next.

Top metrics:

Kernel side and userspace event counts.
Kernel side and userspace event drop counts.
Kernel tracepoint invocation counts to deduce the overall server load.
Userspace timeouts.
Falco internal state counters.

Docs: Falco Is Not Starting Up

Mon, 01 Jan 0001 00:00:00 +0000

Action Items (TL;DR)

Read Install and Operate Guides and review falco.yaml and any local configuration file for necessary preconditions.
Address common startup issues by verifying and correcting config misconceptions.
Monitor for potential kernel driver bugs, though less frequent.
Be aware of userspace bugs that can also interfere with Falco startup.
First, always try running Falco with the default and/or easiest configuration without any plugins.

Let's find out!

Debugging Tips

Please acknowledge that The Falco Project performs a wide range of tests and provides pre-built kernel drivers, but perfection is not guaranteed.

How do I determine if Falco does not start up because of a kernel driver or userspace or pure config issue?

When you start Falco, watch the print statements.
If Falco crashes after passing load config stages, especially during syscall source setup, it signals potential kernel driver issues. These issues may include device unavailability, permission problems, or strange printouts. Alternatively, it could suggest that the kernel driver is not present in the first place, for instance, due to a download failure or missing mounts.
If Falco started up, but then crashed after, it's likely a genuine bug somewhere, we would have to find out.

Kernel Drivers

Falco kernel driver issues are the most common source of frustrating errors. Please note that since Falco 0.38.0, modern_ebpf driver is the new default driver, and it will be automatically used wherever is supported; this should help mitigate most of the following issues. Here are a few tips to demystify what can go wrong with respect to Falco's kernel drivers:

Check if all preconditions to start up the kernel drivers are met. Common issues include:
- For ebpf based drivers, the bpf syscall needs to be allowed and not blocked by SELinux or similar.
- Ensure the DKMS package is installed for the kmod driver, and your system may require custom-signed kernel modules. Also, verify the availability of the host /dev mount (e.g. /dev:/host/dev when running Falco over a container).
- In general, check that Falco has all host mounts when running from a container or as a daemonset in Kubernetes. Critical mounts for running Falco, assuming the kernel driver is available, include: /etc:/host/etc, /proc:/host/proc, /boot:/host/boot, /dev:/host/dev.
For ebpf and kmod drivers, the kernel object code needs to be available for the exact kernel release (uname -r) of your system. This invites a wide range of possible issues:
- If you use the Falco open source binary on Linux distributions such as stock Ubuntu, Fedora, Debian, Arch Linux, Oracle Linux, Rocky Linux, AlmaLinux, etc., you may encounter an issue if the pre-built kernel driver from The Falco Project is not available for download. Verify on the Driver Index page if the driver is available for your specific OS and kernel.
- Your network ACLs may be blocking the download.
- In case the download fails, building the driver on the fly (over the init container in Kubernetes, for example) can fail for many reasons.
- Lastly, if you run a custom kernel, you'll need to build your own drivers (ebpf or kmod only) or explore the option of using the modern_ebpf driver if applicable.
If your kernel version is >= 5.8 and you are enforcing either kmod or ebpf driver, consider switching to the modern_ebpf driver. It's bundled into the userspace binary and works out of the box, regardless of the kernel release, thanks to the eBPF feature called 'Compile Once Run Everywhere' (CO-RE).
If you are using the ebpf or modern_ebpf driver and encounter verbose and lengthy instruction printouts, you may have encountered a dreaded eBPF verifier failure. In such cases, kindly reach out to the Falco maintainers, providing the kernel release (uname -r). Resolving such instances involves modifying the driver code to ensure the eBPF verifier is happy again.

Userspace

Errors associated with Falco's rules or configurations are generally more understandable, and we provide warnings with clear instructions.
Historically, we have encountered edge case bugs with some newer features. Please bear with us in such cases, and we typically release patches to address them.
In the past, there have been instances where regressions were introduced, and certain configurations or combinations thereof may exhibit unexpected behavior. However, Falco's core functionality undergoes comprehensive testing, and we are committed to ensuring its continued reliability.

Restarts

Falco is a C/C++ application for performance reasons, and as such, it is not unheard of for Falco to crash and restart in some rare code paths or edge case conditions. However, if you deploy Falco with resource limits, for example the OOM killer can also kill the process and force a restart. Read more in the Falco Performance Guide.

References and Community Discussions

Docs: Falco Is Dropping Syscalls Events

Mon, 01 Jan 0001 00:00:00 +0000

Action Items (TL;DR)

Adjust the buf_size_preset in the falco.yaml config.
Utilize base_syscalls to limit the syscalls under monitoring.
Audit and optimize Falco rules to prevent unnecessary backpressure on the kernel, considering that Falco's main event stream is single-threaded.
Try running Falco without any plugins.

Background

Falco monitors each syscall based on deployed Falco rules. Additionally, Falco requires a few more syscalls to function properly, see Adaptive Syscalls Selection:

The default configuration is conservative; consequently, there is an opportunity that you could optimize and even eliminate Falco dropping events, depending on the scope of monitoring you are seeking.
Utilize the base_syscalls config for precise override control alongside a resource-friendly suggestion of the absolute minimum additional syscalls needed to ensure proper functioning of Falco (set repair: true).

Falco monitors syscalls by hooking into kernel tracepoints. To transfer events from the kernel to userspace, it uses buffers. For each CPU, Falco allocates separate buffers. If you're using the modern_ebpf driver, you can choose to have fewer, larger buffers shared among multiple CPUs (contention, according to kernel experts, should not be a problem). The buffer size is fixed but can be adjusted in the buf_size_preset config. Increasing the size helps, but keep in mind that the benefits may not increase proportionally. Also, remember that a larger buffer means more preallocated memory.

buf_size_preset of 5 or 6 could be a valid option for large machines assuming you use the kmod or ebpf drivers.
For the modern_ebpf driver try a modern_ebpf.buf_size_preset of 6 or 7, along with a modern_ebpf.cpus_for_each_buffer of 4 or 6. Feel free to experiment and adjust these values as needed.

Lastly, while it may sound appealing to push all filtering into the kernel, it is not that straightforward. In the kernel, you are in the application context, and yes, you can slow down both the kernel and the application (for example, apps may then experience lower request rates). Checkout the Driver Kernel Testing Framework for more information. Additionally, in the kernel, you only have raw syscall arguments and can't easily correlate them with other events. All this being said, we are actively looking into ways to improve this and make the kernel logic smarter without sacrificing performance.

Kernel-side Syscalls Drops Metrics

Falco's metrics config (see also Falco Metrics) enables you to measure Falco's kernel-side syscall drops and provides a range of useful metrics related to software functioning. Key settings include:

kernel_event_counters_enabled: true
libbpf_stats_enabled: true (for ebpf or modern_ebpf drivers, enable /proc/sys/kernel/bpf_stats_enabled)

Here is an example metrics log snippet highlighting the fields crucial for this analysis. Pay close attention to falco.evts_rate_sec and scap.evts_rate_sec, as well as the monotonic drop counters categorizing syscalls into coarse-grained (non-comprehensive) categories. For more details, refer to the dedicated metrics section in the Falco Performance guide for a more detailed explanation.

{
 "output_fields": {
 "evt.source": "syscall",
 "falco.host_num_cpus": 96, # Divide *rate_sec by CPUs
 "falco.evts_rate_sec": 93345.1, # Taken between 2 metrics snapshots
 "falco.num_evts": 44381403800,
 "falco.num_evts_prev": 44045361392,
 # scap kernel-side counters
 "scap.evts_drop_rate_sec": 0.0, # Taken between 2 metrics snapshots
 "scap.evts_rate_sec": 93546.8, # Taken between 2 metrics snapshots
 "scap.n_drops": 0, # Monotonic counter all-time kernel side drops
 # Coarse-grained (non-comprehensive) categories for more granular insights
 "scap.n_drops_buffer_clone_fork_exit": 0,
 "scap.n_drops_buffer_close_exit": 0,
 "scap.n_drops_buffer_connect_enter": 0,
 "scap.n_drops_buffer_connect_exit": 0,
 "scap.n_drops_buffer_dir_file_exit": 0,
 "scap.n_drops_buffer_execve_exit": 0,
 "scap.n_drops_buffer_open_enter": 0,
 "scap.n_drops_buffer_open_exit": 0,
 "scap.n_drops_buffer_other_interest_exit": 0,
 "scap.n_drops_buffer_proc_exit": 0,
 "scap.n_drops_buffer_total": 0,
 "scap.n_drops_bug": 0,
 "scap.n_drops_page_faults": 0,
 "scap.n_drops_perc": 0.0, # Taken between 2 metrics snapshots
 "scap.n_drops_prev": 0,
 "scap.n_drops_scratch_map": 0,
 "scap.n_evts": 48528636923,
 "scap.n_evts_prev": 48191868502,
 # libbpf stats -> all-time kernel tracepoints invocations stats for a x86_64 machine
 "scap.sched_process_e.avg_time_ns": 2041, # scheduler process exit tracepoint
 "scap.sched_process_e.run_cnt": 151463770,
 "scap.sched_process_e.run_time_ns": 181866667867268,
 "scap.sys_enter.avg_time_ns": 194, # syscall enter (raw) tracepoint
 "scap.sys_enter.run_cnt": 933995602769,
 "scap.sys_enter.run_time_ns": 181866667867268,
 "scap.sys_exit.avg_time_ns": 205, # syscall exit (raw) tracepoint
 "scap.sys_exit.run_cnt": 934000454069,
 "scap.sys_exit.run_time_ns": 192201218598457
 },
 "rule": "Falco internal: metrics snapshot"
}

Precise Control Over Monitored Syscalls

Since Falco 0.35.0, you have precise control over the syscalls Falco monitors. Refer to the Adaptive Syscalls Selection blog post and carefully read the base_syscalls config description for detailed information.

Run Tests for Data-Driven Insights

Falco's current metrics system lacks direct syscalls counters to pinpoint high-volume culprits. In the meantime, deriving insights step by step is necessary until syscall counters become available in Falco's metrics system.

Generate a dummy rule designed not to trigger any alerts:

- macro: spawned_process
 condition: (evt.type in (execve, execveat))

- rule: TEST Simple Spawned Process
 desc: "Test base_syscalls config option"
 enabled: true
 condition: spawned_process and proc.name=iShouldNeverAlert
 output: "%evt.type"
 priority: WARNING

Now, run Falco with the dummy rule and the specified test cases (edit base_syscalls config). If you're open to it, consider sharing anonymized logs for further assessment by Falco maintainers or the community to explore potential solutions.

For each test, run Falco in dry-run debug mode initially to print the final set of syscalls.

sudo /usr/bin/falco -c /etc/falco/falco.yaml -r falco_rules_dummy.yaml -o "log_level=debug" -o "log_stderr=true" --dry-run

# Example Output for Test 2
XXX: (2) syscalls in rules: execve, execveat
XXX: +(16) syscalls (Falco's state engine set of syscalls): capset, chdir, chroot, clone, clone3, fchdir, fork, prctl, procexit, setgid, setpgid, setresgid, setresuid, setsid, setuid, vfork
XXX: (18) syscalls selected in total (final set): capset, chdir, chroot, clone, clone3, execve, execveat, fchdir, fork, prctl, procexit, setgid, setpgid, setresgid, setresuid, setsid, setuid, vfork

Subsequently, run Falco normally.

sudo /usr/bin/falco -c /etc/falco/falco.yaml -r falco_rules_dummy.yaml

Test 1: spawned_process only

base_syscalls:
 custom_set: [clone, clone3, fork, vfork, execve, execveat, procexit]
 repair: false

If Test 1 already fails, and you see drops even after adjusting the buf_size_preset and other parameters, Falco may be less usable on this particular system, unfortunately.

Test 2: spawned_process + minimum syscalls needed for Falco state (internal process cache table)

base_syscalls:
 custom_set: []
 repair: true

Test 3: network accept*

base_syscalls:
 custom_set: [clone, clone3, fork, vfork, execve, execveat, getsockopt, socket, bind, accept, accept4, close]
 repair: false

Test 4: network connect

base_syscalls:
 custom_set: [clone, clone3, fork, vfork, execve, execveat, getsockopt, socket, connect, close]
 repair: false

Test 5: open* syscalls

base_syscalls:
 custom_set: [clone, clone3, fork, vfork, execve, execveat, open, openat, openat2, close]
 repair: false

Test n

Continue custom testing to ensure effective monitoring of all desired syscalls on your servers without experiencing event drops or with minimal acceptable drops.

At What Kernel Event Rates Do Problems Generally Start?

This question presents a challenge as it's not solely about the pure "kernel event rate". In less realistic benchmarking tests, you could artificially drive the rates very high without dropping events. Therefore, we believe it is more complex in real-life production, involving not just event rates but also the actual nature of the events, and possibly bursts of events in very short periods of time.

Additionally, we believe it's best to normalize the event rates by the number of CPUs (e.g. scap.evts_rate_sec / falco.host_num_cpus). Busier servers with 96, 128, or more CPUs naturally have higher event rates than VMs with 12 CPUs, for instance.

Nevertheless, here are some numbers we have heard from various adopters. Please take them with a grain of salt:

Less than ~1K kernel events per second per one CPU usually is not a problem, but it depends.
Less than ~1.5K kernel events per second per one CPU should not be a problem, but it depends.
More than 3K kernel events per second per one CPU likely could be more difficult to keep up, but it depends.
Consider 1-2% of all events dropped on a smaller subset of servers in your fleet (your busy servers/clusters) as acceptable.
More than 164K kernel events per second per CPU have been observed on a 128-CPU machine. Currently under exploration is how to solve this.

References and Community Discussions

Docs: Missing Fields in Falco Logs

Mon, 01 Jan 0001 00:00:00 +0000

Action Items (TL;DR)

Read Install and Operate Guides and review falco.yaml for necessary preconditions.
Refer to the relevant debugging guide based on suspected missing fields.
Acknowledge that certain missing fields or data in Falco are legitimate.

Background

Many of the Supported Output Fields are derived from multiple events and mechanisms. To provide a more concrete explanation, for each spawned process, Falco extracts and derives fields from the clone*/*fork/execve* syscalls. Falco generates a struct in userspace, stores the relevant information within this struct, and then adds it to the process cache table in memory. If a process makes additional system calls during its lifetime, such as opening a file, in a Falco rule, you typically also export process fields — assuming we haven't missed the spawned process event and the information is available. These details extend to various use cases, and, in essence, dropped events can lead to missing fields as well as race conditions.

As a result, Falco logs can never be perfect, and null values can occur. We are constantly aiming to improve the robustness in this regard. We encourage you to contribute to the project if you encounter such cases or have improvement ideas. Also be aware that, unfortunately, missing fields can have different natures. Sometimes the field may be an empty string, or the string <NA>, or, if numeric, the default numeric value. These inconsistencies may be more difficult to address, as many Falco rules rely on legacy declarations.

Furthermore, sometimes Linux may not operate exactly as expected. One concrete example is that shell built-ins like echo do not cause a new spawned process, and the echo command does not get logged with Falco. Similarly, if a base64 encoded string gets interpreted during decoding, you do not have the original base64 blob in the command args unless the command was passed with the sh -c flag. Lastly, some fields only work for certain kernel versions or system configs (e.g. proc.is_exe_upper_layer requires a container overlayfs).

Missing Container Images

Check the basics:

Is the container runtime socket correctly mounted? For Kubernetes, mount with the HOST_ROOT prefix: /host/run/k3s/containerd/containerd.sock. See deploy-kubernetes example template.
Is a custom path specified for the container runtime socket in Kubernetes? If yes, use the -o container_engines.cri.sockets[]=<socket_path> command line option when running Falco. The default paths include: /run/containerd/containerd.sock, /run/k3s/containerd/containerd.sock, /run/crio/crio.sock.
To expedite lookups, attempt to disable asynchronous CRI API calls by using the -o container_engines.cri.disable_async=true command line option when running Falco.
Falco monitors both host and container processes. If the container.id is set to host, it indicates that the process is running on the host, and therefore, no container image is associated with it.

The k8s.* fields are extracted from the container runtime socket simultaneously as we look up the container.* fields from the CRI API calls responses.

Using CRI with containerd

When using containerd as your container runtime, you should configure Falco to use the CRI engine to consume the containerd socket (/run/containerd/containerd.sock). This is important because:

The native containerd protocol does not support container names - it only provides container IDs
Containerd typically exposes two interfaces on the same socket: the native containerd protocol and the CRI (Container Runtime Interface) protocol
The CRI protocol provides richer metadata, including container names

If you are missing container.name or other container metadata fields while using containerd, ensure you are using the CRI engine configuration (not the containerd engine) in your Falco setup. For example, configure the container plugin with:

engines:
 cri:
 enabled: true
 sockets:
 - /run/containerd/containerd.sock
 containerd:
 enabled: false

Carefully read the field description documentation:

Supported Output Fields container.* retrieved from the container runtime socket
Supported Output Fields k8s.* also retrieved from the container runtime socket

The container info enrichment, while robust, depends on the speed of making API requests against the container runtime socket.

Falco's metrics config (see also Falco Metrics) provides a range of useful metrics related to software functioning, now also featuring metrics around Falco's internal state:

state_counters_enabled: true

Here is an example metrics log snippet highlighting the fields crucial for this analysis.

{
 "output_fields": {
 "evt.source": "syscall",
 "falco.n_containers": 50,
 "falco.n_missing_container_images": 0,
 },
 "rule": "Falco internal: metrics snapshot"
}

falco.n_containers indicates how many containers are running at a given time, typically less than 100-300 at maximum. falco.n_missing_container_images is an updated snapshot of how many containers are internally stored in Falco without a container image at any given time.

To complicate matters, some processes in Kubernetes run in the pod sandbox container, which has no container image in the API responses. In such cases, the container.id is the same as the k8s.pod.sandbox_id. If the container image is consistently missing throughout the lifetime of the container, it's likely a process in a pod sandbox container in the majority of the cases. However, sandbox containers likely constitute less than 1% of the distinct containers in your overall Falco logs. Note that this comparison will be fully supported by Falco 0.38 and is a work in progress.

Additionally, the improvement of the overall efficiency of the container engine, especially for the -o container_engines.cri.disable_async=true option, is also a work in progress. A more performant implementation is expected to be available by Falco 0.38. This improvement aims to address missing images observed by adopters and resolve most cases, leaving only some edge cases of race conditions where the lookup hasn't happened yet.

Missing User Names

Ensure proper mounts (e.g., /etc:/host/etc) when running Falco as a daemonset in Kubernetes, for example.
If you expect Falco to be aware of Kubernetes Control Plane users, especially when execing into a pod (kubectl exec), we must disappoint you. The Linux kernel lacks knowledge of the control plane. However, we are actively exploring ways to support this. Refer to this issue for more details.

Missing Process Tree Fields

Let's consider another example: the fields related to the process tree lineage (e.g. proc.aname*).

Falco adds processes to a cache in userspace when a new process starts and removes them when the process exits. The goal is to maintain a current view of running processes on the Linux host at any time. However, this also means that there are cases where the parent legitimately exits, re-parenting occurs, and/or PIDs get replaced or re-used.
As a result, missing processes in the process ancestry (process tree) may be due to dropped or missed events, failure to store the event, or the process exiting without proper tracking of re-parenting or orphan process cases by Falco.
Furthermore, a history of all spawned_process events is not equivalent to the current process tree on the system. Check out the Falco rules macro container_entrypoint for one such example and explore this resource.
In summary, Falco aims to closely preserve the true system state, similar to the Linux kernel itself.

Falco's metrics config (see also Falco Metrics) provides a range of useful metrics related to software functioning, now also featuring metrics around Falco's internal state:

state_counters_enabled: true

Here is an example metrics log snippet highlighting the fields crucial for this analysis.

{
 "output_fields": {
 "evt.source": "syscall",
 "falco.n_drops_full_threadtable": 0,
 "falco.n_store_evts_drops": 0,
 "falco.n_failed_fd_lookups": 0,
 "falco.n_failed_thread_lookups": 0,
 "falco.n_retrieve_evts_drops": 0
 },
 "rule": "Falco internal: metrics snapshot"
}

falco.n_drops_full_threadtable and falco.n_store_evts_drops reflect similar occurrences. They are monotonic counters indicating how often a spawned process event was dropped due to a full table (configurable by Falco 0.38 with a higher default value) and how frequently store actions to update the process structs in memory failed and were subsequently dropped. On the flip side, there are also counters keeping track of failed lookup or retrieve actions. Internally, Falco is granular and talks about threads, not processes.