Falco – Falco

Blog: Introducing Prempti: Falco meets AI coding agents

Tue, 12 May 2026 00:00:00 +0000

Today's developer workflow is increasingly reliant on AI coding agents. Tools like Claude Code sit in your terminal, read your files, run shell commands, make network requests, and write code, all on your behalf. They are fast, capable, and increasingly trusted with real tasks on real machines.

But with that trust comes a question worth taking seriously: what exactly is your coding agent doing on your machine?

Today, we're introducing an experimental project that brings Falco to this new frontier: Prempti.

Agents are a black box at runtime

When a coding agent runs a bash command, writes a file, or reads a configuration, those actions happen inside your user session, with your permissions, in your filesystem, against your credentials. Most developers using these tools have no structured visibility into that activity. You see the agent's chat output, but you don't see what's happening under the hood.

Here's a simple scenario: you ask your coding agent to refactor a module. It reads your source files. It makes edits. Then, perhaps prompted by a malicious dependency or an unexpected instruction in a file it just parsed, it attempts to read ~/.ssh/known_hosts or write a file to ~/.aws/. Should it be allowed to? Would you even know if it tried?

The demo below captures exactly this situation:

The agent tried to both read and write to sections it's not allowed to, and both were blocked. The agent itself received a structured message explaining why, and showed that to the user. This is detection and enforcement working together at the tool-call level.

How Prempti works

Prempti runs as a lightweight user-space service alongside your coding agent. It does not require root, kernel modules, or containers. When your agent makes a tool call such as a file write, a shell command, or a file read, Prempti intercepts it before it executes, evaluates it against Falco rules, and delivers a verdict:

Verdict	What Happens
Allow	The tool call proceeds normally
Deny	The tool call is blocked and the agent is told why
Ask	You are prompted to approve or reject interactively

The architecture looks like this:

Prempti's hook fires before each tool call
An interceptor sends the event to Falco via a Unix socket
Falco's rule engine evaluates the event against your policies
Matching rules produce verdicts (deny / ask / allow)
The interceptor delivers the verdict back to the agent

Prempti uses Falco's plugin system to define a new event source (coding_agent) with fields purpose-built for this context: tool.name, tool.input_command, tool.file_path, agent.cwd, and so on.

Two modes: Monitor and Guardrails

Prempti is designed to let you both observe what the agent is doing and align it with your security policy:

Monitor mode evaluates every tool call against your rules and logs the results, but does not enforce any action. This is what we recommend as a starting point: run it for a few sessions, see what your agent actually touches, and tune your rules before you enable blocking.

Guardrails mode (the default) fully enforces verdicts as explained above — deny blocks, ask prompts you, allow proceeds.

You can switch between modes at any time:

premptictl mode monitor # observe only
premptictl mode guardrails # enforce verdicts
premptictl logs # watch live events

Writing rules: Familiar territory

If you've written Falco rules before, agent security policies will feel very familiar. Here's a rule that blocks piping content directly to a shell interpreter, a classic vector for prompt injection attacks:

- rule: Deny pipe to shell
 desc: Block piping content to shell interpreters
 condition: >
 tool.name = "Bash"
 and (tool.input_command contains "| sh"
 or tool.input_command contains "| bash"
 or tool.input_command contains "| zsh")
 output: >
 Falco blocked piping to a shell interpreter (%tool.input_command)
 priority: CRITICAL
 source: coding_agent
 tags: [coding_agent_deny]

The output field is designed to be LLM-friendly, so that the agent receives it as a structured message it can surface directly to the user. Correlation IDs allow you to trace every event across your logs.

The default ruleset ships with policies covering six areas:

Working-directory boundary — monitor and ask on file access outside the session's project directory
Sensitive paths — deny reads and writes to /etc/, ~/.ssh/, ~/.aws/, cloud credentials, .env files, and similar
Sandbox disable — detect attempts to disable the agent's own sandbox configuration
Threats — credential access, destructive commands, pipe-to-shell, encoded payloads, exfiltration, IMDS access, reverse shells, and supply-chain installs from known-malicious hosts
MCP and skill content — MCP server config poisoning and slash-command file injection
Persistence vectors — hook injection, git hooks, package-registry redirects, AI API base-URL overrides, and API keys leaking into env files

You can add your own rules to ~/.prempti/rules/user/; they're preserved across upgrades.

Rule authoring with Claude Code

The project also includes a Claude Code skill for writing Falco rules for Prempti interactively. You can install it directly from the Prempti plugin marketplace:

/plugin marketplace add falcosecurity/prempti
/plugin install prempti-falco-rules@prempti-skills

Then you can ask Claude Code to create rules like:

"Block the agent from running git push"
"Deny any read outside the working directory"
"Create a rule that requires confirmation before editing Dockerfiles"

The skill guides you through writing the rule, placing it in the right directory, and validating it with Falco. It's a great example of the kind of human-AI collaboration this project is designed to enable: the agent helps you constrain itself.

Let's be honest about limitations

We want to be clear about what this project is and isn't.

Prempti intercepts tool calls as declared by the agent, not the system calls those tool calls produce. If an agent writes a malicious binary and runs it, Falco sees gcc main.c -o main and ./main, not what ./main does at the OS level. For deep syscall-level visibility on Linux, Falco's kernel instrumentation (eBPF/kmod) remains the right tool.

Prempti is also not a sandbox. It doesn't prevent a sufficiently determined agent from circumventing the hook mechanism if it can find a path the hook doesn't cover. Think of it as a policy layer at the agent level — a valuable complement to sandboxing and system hardening, not a replacement for them.

What it does provide is visibility and a programmable policy boundary that lives at the most natural enforcement point: the moment the agent decides to act.

Getting started

Download the latest release from the GitHub repository: https://github.com/falcosecurity/prempti/releases

macOS:

installer -pkg prempti-<version>-darwin-universal.pkg \
 -target CurrentUserHomeDirectory

The installer wizard handles everything. The service starts automatically on login.

Linux:

tar xzf prempti-<version>-linux-x86_64.tar.gz
cd prempti-<version>-linux-x86_64
bash install.sh

Windows:

msiexec /i prempti-<version>-windows-<arch>.msi

Verify your setup:

premptictl status
premptictl hook status

Explore together with us

Runtime security for AI coding agents is genuinely new territory. The threat models are still being defined. The right default policies are still being discovered. We believe our community of developers, security engineers, and the people running these agents day to day are the ones who will figure out what good looks like here. If you've used Prempti, we'd love to hear what you found:

What rules have you written? What did you catch?
What agents or platforms do you need support for?
What didn't work as expected?

Open an issue, start a discussion, or come chat with us in the Falco Slack. Every piece of feedback shapes what this project becomes.

Prempti is released under the Apache License 2.0. Currently supports Claude Code on Linux (x86_64, aarch64), macOS (Apple Silicon, Intel), and Windows (x86_64, ARM64). Codex integration is on the roadmap.

Blog: Falco at KubeCon Europe 2026 — See You in Amsterdam! 🐦

Wed, 18 Mar 2026 00:00:00 +0000

We're excited to share that the Falco community will be at KubeCon + CloudNativeCon Europe 2026 in Amsterdam! Whether you're a long-time contributor, a curious user, or just want to say hi, we'd love to see you there.

Falco is celebrating 10 years of development and adoption, and we are on the lookout for people who would like to say Happy Birthday to the project or share their best Falco story. Libby Schulze and I will be on the event floor with mic and camera to capture some amazing moments and memories from Falco's 10 years. So bring your best story, and we'll see you at the Falco booth!

Sneak peek

Psst... we have something really cool brewing that we will show at the Falco booth. You, our amazing reader, is the first to hear about this. It's a way to run Falco locally on your development machine, and make sure your AI coding agents are following new rules that are being defined. We'd love to get your feedback on this as we're currently building it!

Here’s where you can find us in Amsterdam and everything we have lined up:

Project lightning talk

Forensics With Falco
Speaker: Gerald Combs, Maintainer
When: Monday, March 23, 2026 — 10:27 to 10:32 CET
Where: Elicium 2

Falco has recently expanded its capabilities with capture recording, opening the door to seamless integration with forensic analysis tools like Stratoshark. In this lightning talk, Gerald will walk through how the two tools work together to provide deep visibility into container and system activity. He will demonstrate how captured event data can accelerate investigations and discuss key considerations for safely and efficiently deploying these features in production environments.

Sysdig-led workshop

Hands-On Cloud Native Security Workshop
When: Monday, March 23 — 2:00–4:00 PM CET

Run Atomic Red Team™ tests, then step into the Blue Team role to detect threats and create custom Falco™ detection rules in this hands‑on 90‑minute keyboard workshop.

Conference talk

In Falco's Nest: The Evolution of Cloud Native Runtime Security
Speakers: Iacopo Rozzo (Sysdig), Aldo Lacuku (Kong Inc.)
When: Tuesday, March 24, 2026 — 12:00 to 12:30 CET
Where: G102–103

Falco, the Cloud Native Runtime Security project, is constantly evolving to meet the demands of modern cloud environments. This maintainer track session, led by the Falco maintainers, will dive deep into the latest advancements and the strategic direction of the project. We will focus on two major areas of growth: the introduction of the new Falco Operator and the new features that enhance Falco's performance and reliability.

The new Falco Operator simplifies the deployment, configuration, and management of Falco across Kubernetes clusters, making it easier than ever for users to secure their runtime environments at scale.

Furthermore, we will explore the most significant new features integrated into Falco. This includes performance optimizations for high-throughput environments. The session will also touch upon community contributions, ecosystem integrations, and the roadmap for the upcoming release.

Booth demo

Pivoting from detection to investigation with Falco and Stratoshark
Speaker: Gerald Combs
When: Tuesday, March 24, 2026 — 15:45 CET
Where: Sysdig Booth #671

See how to move from “we detected something” to “here’s what happened” using Falco and Stratoshark. Stop by the Sysdig booth and say hello!

Thank you!

We couldn’t do this without you all in our community - the contributors, users, and everyone who shows up at events. If you’re in Amsterdam, come find us at the talks, the workshop, or the booth. We’d love to meet you and hear how you’re using Falco.

See you there! 🐦

Blog: Hey Falco Flock! 🐦 Let's Soar Into 2026

Wed, 25 Feb 2026 00:00:00 +0000

New year, new opportunities!

As we spread our wings and glide into 2026, we want to make sure this community is one you’re proud (and excited!) to be a part of. Falco has always been more than just a project: it’s a flock of builders, defenders, contributors, question-askers, doc-writers, rule-tuners, and runtime security enthusiasts.

And now we want to hear from you.

We’ve put together a quick community survey (5 minutes or less!) to better understand:

How connected you feel to the community
What you love about being a part of it
What could be better
What you’d like to see us focus on this year
What resources would make your life easier
How you’re using Falco and what tools you integrate it with

Your feedback directly shapes our focus on what we build, improve, prioritize, and invest in this year - from documentation and content to events, integrations, and contributor experience. A report detailing the responses will be shared at the same time as KubeCon Europe 2026.

Whether you’re building, using, learning, or just keeping an eye on things, your voice matters.

👉 Take the survey here

Thanks for being part of the flock. We couldn’t do this without you and we’re excited to build 2026 together!

Blog: Introducing Falco 0.43.0

Mon, 26 Jan 2026 00:00:00 +0000

Dear Falco Community, we are happy to announce the release of Falco 0.43.0 today!

This is a stabilization release that consolidates the changes introduced in 0.42.0, including the drop-enter initiative and the capture recording feature. It also introduces several deprecations to improve maintainability and fixes minor issues across falcoctl, plugins, and libs.

During this release cycle, we merged:

31 PRs on Falco, including 11 release note-worthy changes
48 PRs on Falco libs, including 17 release note-worthy changes
8 PRs on Falco drivers, including 3 release note-worthy changes

We upgraded libs to version 0.23.1 and drivers to 9.1.0+driver. Thank you to our maintainers and contributors. This would not have been possible without your support and dedication!

To learn everything about the changes, read on!

What's new? TL;DR

Key fixes:

evt.arg.filename field reintroduction
Falcoctl signature verification fixes
overflow and NULL pointer dereferences fixes for the container plugin, shipped with plugins/container/0.6.1
race condition fix for the k8smeta plugin, shipped with plugins/k8smeta/0.4.1

This release also comes with breaking changes that you should be aware of before upgrading.

Latest updates

Deprecations

In Falco 0.43.0, we are announcing the deprecation of three significant components to streamline the project, reduce maintenance burden, and focus on modern, more efficient alternatives. All these components are stable, and considering that the deprecation is first enforced in this version, they could be removed at any future version starting from 0.44.0.

Legacy eBPF probe deprecation

The "legacy" eBPF probe (configured via engine.kind=ebpf) was the original eBPF implementation in Falco. It required compiling a specific probe for each kernel version, often necessitating the dynamic usage of the falco-driver-loader or pre-built drivers. The Modern eBPF probe (engine.kind=modern_ebpf), which leverages CO-RE (Compile Once – Run Everywhere), has reached maturity and feature parity. It offers superior stability, portability (no need to compile drivers on the fly), flexibility and performance. Maintaining two eBPF drivers splits engineering effort and complicates the codebase. Users currently using the legacy eBPF probe are strongly encouraged to switch to the Modern eBPF probe by setting engine.kind=modern_ebpf in their falco.yaml, or to engine.kind=kmod if the used kernel doesn't provide support for the modern eBPF probe.

See the relevant section of the deprecation proposal for the detailed motivation behind the deprecation.

gVisor deprecation

The gVisor engine is a dedicated, internal C++ implementation designed to monitor system calls from gVisor sandboxes leveraging events coming from gVisor itself through gRPC. There is evidence that this engine is little used. Moreover, gVisor doesn't provide all information required to build all supported event types, indeed resulting in a system call source not completely equivalent to the ones provided by drivers. Finally, it requires libs being dependent on protobuf, this latter introducing a non-negligible build time overhead and maintainability burden.

See the relevant section of the deprecation proposal for the detailed motivation behind the deprecation.

gRPC output and server deprecation

The gRPC output was implemented to allow external consumers to subscribe to a stream of Falco security alerts over a gRPC connection. It was notably utilized by tools like the event-generator (in test mode) and custom integrations requiring a streaming API for alerts. The gRPC output and the gRPC server embedded in Falco add substantial complexity to the core codebase, including dependencies on specific protobuf and gRPC framework versions in Falco and libs. Over time, it has become clear that the community prefers standard, widespread integration patterns for alert consumption - primarily HTTP and the ecosystem enabled by Falcosidekick. Users consuming alerts via gRPC should migrate to the HTTP output or use Falcosidekick to forward events to their destination of choice.

See the relevant section of the deprecation proposal for the detailed motivation behind the deprecation.

GPG key rotation

In anticipation of the previous GPG key's expiration in January 2026, we have rotated the GPG key used to sign the official RPM and DEB packages. Pre-existing Falco installations (installed via apt or yum before the rotation) must manually import the new GPG key. Failure to do so may result in errors during package updates or verification failures. Please follow the "Trust the falcosecurity GPG key" step in the official documentation for your package manager:

apt (Debian/Ubuntu): Install with apt
yum/dnf (CentOS/RHEL/Fedora): Install with yum

Notice that new installations following the current documentation will automatically receive the updated key bundle and do not require additional steps.

For more details see [TRACKING] [deadline 2026-01-17] Rotate public GPG key for RPM/DEB package signing.

Container plugin improvements

The container plugin, which extracts metadata from container runtimes to enrich Falco events, includes important updates in version 0.6.1 to enhance its API capabilities and performance. This release exposes container.id, container.image, container.name, and container.type through the table API and adds comprehensive logging across all engines, while also preventing allocations by extensively using zero-allocation tools offered by the C++ (like std::string_view) and avoiding reflex matcher allocations during resolve operations.

Falcoctl tweaks and improvements

`follow` polling interval increase to 1 week

About three years ago, we started distributing Falco artifacts (rules files and plugins) via ghcr.io, and later added automatic rule updates in falcoctl with a 6h check interval. With years of data now, it’s clear we don’t need checks that frequent: our rule updates happen far less often. Moreover, due to the growth of Falco adoption, these frequent checks are now hitting ghcr.io rate limit. These two reasons drove the decision to increase the artifact follow interval from 6h to 1 week.

For more details see chore(scripts/falcoctl): increase follow interval to 1 week and Falco's Helm chart changelog.

Dependency resolution improvements

The artifact installation logic has been reworked to handle dependencies and references correctly. Previously, dependencies could be duplicated or incorrectly resolved, and signature verification was skipped for full registry references. Now dependencies are properly deduplicated, all refs are correctly resolved, and signatures are verified for all resolved dependencies, not just the top-level artifacts. This provides end-to-end verification of the entire dependency chain.

For more details see Inefficient deduplication logic and incorrect input handling for dependency resolution

Support for cosign v3

Falcoctl now supports Cosign v3 bundle format for signature verification. This is the new standard for signing OCI artifacts, replacing the legacy .sig tag format.

What this means for you:

Artifacts signed with cosign v3 are now fully supported
Backward compatibility with cosign v2 signatures is maintained

For more details see feat: Upgrade to Cosign v3 with Bundle Format

Key fixes

`evt.arg.filename` field reintroduction

As part of the recent "drop enter" optimization initiative (which removed enter events for most syscalls to improve performance), the filename argument - historically available only in the enter event for execve and execveat - was inadvertently made unavailable. This caused a regression where specific context, such as the exact path provided to the syscall (e.g., a symlink path versus the resolved binary path), was lost in the remaining exit event.

In Falco 0.43.0 (via libs 0.23.0), this has been fixed. The filename argument is now correctly populated in the exit events for these syscalls. Users can once again access this data using the evt.arg.filename field in their rules, ensuring that the critical execution context is preserved without needing the deprecated enter events.

For more details see Missing "filename" argument to execve syscall in libscap 0.22.x.

Falcoctl signature verification fixes

Signature verification fix for full reference artifacts

Fixed an issue where signature verification was skipped for artifacts specified with a full registry reference ( e.g., ghcr.io/falcosecurity/plugins/plugin/container:0.4.1). Now all artifacts are verified regardless of how they are referenced.

Signature verification fix for authenticated registries

Signature verification now works correctly on private/authenticated registries. Previously, verification would fail with authentication errors even though the artifact pull succeeded, and credentials were not being passed to the signature verification component.

Supported authentication methods:

Basic auth (Docker credentials)
OAuth2 client credentials
GCP Workload Identity (for GKE deployments)

For more details see fix(signature): pass registry credentials to cosign for signature verification

Breaking changes and deprecations

This version includes breaking changes you should be aware of before upgrading.

Bump drivers minimum required kernel version to `3.10`

Falco 0.43.0 introduces a breaking change regarding the Falco drivers. Starting with drivers version 9.1.0+driver, the minimum required Linux kernel version has been bumped to 3.10. In practice, this only affects the kmod driver and means that the kernel module will explicitly fail to compile on kernels older than 3.10. This choice is motivated by the fact that even Linux 3.10 is a 12-year-old kernel, and its support ended in 2017: maintaining support for older kernels is a maintenance burden and limits progress. This change enables the team to focus on modernizing the codebase and improving stability for current environments.

Deprecation warnings

Falco 0.43.0 introduces several deprecation warnings to help users migrate to newer components:

Legacy eBPF probe deprecation: using the legacy eBPF probe (engine.kind=ebpf) will now generate warnings
gVisor engine deprecation: using the gVisor engine (engine.kind=gvisor) will now generate warnings
gRPC deprecation: using the gRPC output or the gRPC server (grpc_output.enabled=true or grpc.enabled=true), will now generate warnings

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

Stay connected

Join us on social media and in our community calls, held every other Wednesday! It's always great to have new members in the community, and we're looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: GPG Key Rotation for Falco Packages (2026)

Fri, 12 Dec 2025 00:00:00 +0000

The GPG key used to sign official Falco packages (RPM and DEB) is set to expire on January 17, 2026. To ensure the security and continuity of our software distribution, the Falco maintainers will be rotating to a new 4096-bit RSA key.

We have designed a two-phase "Soft Launch" strategy to make this transition as smooth as possible, providing a one-month transition window before the old key is retired.

The Rotation Plan

To avoid immediate disruption, we are rolling out the new key in two distinct phases. You can follow the detailed progress in our tracking issue #3750.

Phase 1: Soft Launch (Dec 12, 2025)

What happens: The new GPG key has been published and added to our repository configuration.
Dev Builds: Will begin using the New Key immediately.
Stable Builds: No stable releases are planned for this phase. If any hotfixes are released, they will be signed with New Key as well.
Key Bundle: The official key URL has been updated to serve a bundle containing both the Old (valid) and New (valid) keys.

Phase 2: Hard Cut-Over (Jan 12–17, 2026)

What happens: This is the maintenance window where we fully switch to the new key.
Mass Resign: All existing stable packages on download.falco.org will be resigned with the New Key.
Revocation: The Old Key will be officially revoked and removed from the active bundle.
Impact: If you have not updated your keyring by this date, your package manager (apt or yum) will reject updates with a signature verification error.

Action Items for Users

We strongly recommend all users update their GPG keyring before January 12, 2026 to avoid interruption.

New Users

If you are installing Falco for the first time following our Install on a host (DEB,RPM) instructions, no action is required. The installation process will guide you to fetch the new key bundle, ensuring you are ready for both phases.

Existing Users

If you have an existing Falco installation, you must manually import the new key. We have updated the key file at our standard URL to include both the old and new keys, allowing you to transition safely.

For apt users, to update your keyring, run the following command:

# Download the updated key bundle (Old + New) and import it
curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \
 sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg

For yum users, to update your keyring, run the following command:

# Download the updated key bundle (Old + New) and import it
rpm --import https://falco.org/repo/falcosecurity-packages.asc

Note: These commands overwrite the existing keyring file with the new bundle. Since the bundle contains both keys, your current installation will continue to work immediately, and will remain working after the January cut-over.

For more details on apt and yum specific instructions, please refer to the Install on a host (DEB,RPM) page of our documentation.

Summary

Deadline: Update your keys before Jan 12, 2026.
Old Key (Expiring Jan 17, 2026): falcosecurity-14CB7A8D.asc (Fingerprint 2005399002D5E8FF59F28CE64021833E14CB7A8D)
New Key: falcosecurity-B35B1B1F.asc (Fingerprint 478B2FBBC75F4237B731DA4365106822B35B1B1F)
Tracking Issue: falcosecurity/falco#3750

If you encounter any issues during this transition, please reach out to us on the #falco channel on Kubernetes Slack or open an issue on GitHub.

Thank you for your attention and cooperation in keeping Falco secure!

Blog: Introducing Falco 0.42.0

Wed, 22 Oct 2025 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.42.0!

This release brings exciting new capabilities, including the capture feature, significant performance improvements, and important bug fixes that enhance Falco's capabilities. During this release cycle, we merged:

52 PRs on Falco, including 23 release note-worthy changes
110 PRs on Falco libs, including 47 release note-worthy changes
102 PRs on Falco drivers, including 29 release note-worthy changes

We upgraded libs to version 0.22.1 and drivers to v9.0.0+driver. Thank you to our maintainers and contributors. This would not have been possible without your support and dedication!

To learn everything about the changes, read on!

What's new? TL;DR

Key features:

Key fixes:

Fix thread table memory leak when parsing vfork (or equivalent clone/clone3 with CLONE_VFORK) exit from the caller process;
Enable handling of multiple actions configured with syscall_event_drops.actions;
Disable dry-run restarts when Falco runs with config-watching disabled;
Fix abseil-cpp for Alpine build;
Fix detection sandbox containers for CRI and containerd runtimes (container plugin);
Stability improvements for container plugin and static linking of libgcc/libstdc++ for legacy compatibility;

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.42.0 release contains a new capture feature and significant performance improvements. Here is a list of the key new capabilities.

Capture recording feature

Falco 0.42.0 introduces the new capture recording feature, now available at sandbox maturity. This capability allows Falco to generate .scap files whenever a detection rule is triggered automatically.

Each capture contains a detailed trace of system calls around the event, providing forensic-level visibility into what happened. The recordings can be opened directly in Stratoshark for Wireshark-style analysis of runtime behavior.

The capture system is fully configurable: you can enable global recording or tie captures to specific Falco rules for targeted runtime snapshots.

When targeting specific Falco rules (by setting mode: rules, as shown in the configuration below), users can modify individual rules to enable capture by adding capture: true and optionally capture_duration to specific rules. For example:

 - rule: Suspicious File Access
 desc: Detect suspicious file access patterns
 condition: >
 open_read and fd.name startswith "/etc/"
 output: >
 Suspicious file access (user=%user.name command=%proc.cmdline file=%fd.name)
 priority: WARNING
 capture: true
 capture_duration: 10000 # Capture for 10 seconds when this rule triggers

This configuration will capture events for 10 seconds whenever the "Suspicious File Access" rule is triggered, overriding the default duration.

Find below the configuration snippet to enable the capture feature in falco.yaml:

capture:
 # -- Set to true to enable event capturing.
 enabled: false
 # -- Prefix for capture files. Falco appends a timestamp and event number to ensure unique filenames.
 path_prefix: /tmp/falco
 # -- Capture mode. Can be "rules" or "all_rules".
 mode: rules
 # -- Default capture duration in milliseconds if not specified in the rule.
 default_duration: 5000

Learn more at KubeCon + CloudNativeCon North America 2025:

Project Lightning Talk: When Falco Spots Trouble, The Shark Swims In - Gerald Combs, Falco Core Maintainer
Beyond the Cloud(s): Falco's Ascent in Performance and Deep Visibility - Leonardo Grasso & Leonardo Di Giovanna, Sysdig

Drop enter initiative

We've just shipped a significant performance improvement: syscall enter events have been completely removed from our event pipeline.

In Falco, each system call traditionally used to generate two events: an enter event when syscall kernel processing starts (i.e., before its arguments are processed) and an exit event when the kernel processing completes. Now that we collect all relevant information on exit events, we can drop the generation and processing of enter events.

Nevertheless, for TOCTOU (Time-of-Check to Time-of-Use) mitigation, a few selected enter events are still monitored internally — their relevant data is captured and stored — but these events are no longer pushed downstream to the userspace processing pipeline.

By focusing solely on syscall exit events, we've nearly halved the number of events generated and processed by userspace, eliminating redundant data collection. This reduces the Falco instrumentation overhead, improving workloads' performance up to 20% (by reducing syscall execution latency). It also decreases Falco's CPU usage up to 30%, especially in high-syscall environments.

From a developer's perspective, this also removes ambiguity about where syscall parameters should be defined, streamlines event processing logic, and makes event handling code cleaner and easier to maintain.

Overall, you can expect better performance, leaner code, and a more predictable event model moving forward.

For more details, see:

Plugin event schema versioning

Falco 0.42.0 introduces plugin event schema validation, enabling plugins to specify their compatible event schema version.

It provides an event schema validation system for syscall events consumed by plugins that offer parsing and/or field extraction capabilities, ensuring backward compatibility and clear error reporting for plugins that depend on specific Event Schema Versions.

If the plugin does not declare a required Schema Version, it is assumed to be compatible with 3.0.0, the initial major version when the plugin event schema validation was introduced.

The plugins should implement a new function of the Plugin API to declare the required schema version. Find below the signature of the new API function:

// New plugin API functions for schema management
typedef struct {
...
// Event schema version check
//
// Return the minimum event schema version required by this plugin.
// Required: no
// Arguments:
// - s: the plugin state, returned by init(). Can be NULL.
// Return value: the event schema version string, in the following format:
// "<major>.<minor>.<patch>", e.g. "4.0.0".
// If the function is not implemented or NULL is returned, the plugin is assumed to be
// compatible with schema version 3.0.0.
//
const char* (*get_required_event_schema_version)(ss_plugin_t* s);
} plugin_api;

For more details, see:

Plugin system event schema versioning proposal

Thread table auto-purging configuration

We've added a few new falco_libs configurations for advanced users who want finer control over Falco's performance and resource usage. It introduces tunable parameters for Falco's internal thread table, which tracks active threads:

thread_table_size defines the maximum number of entries.
thread_table_auto_purging_interval_s controls how often stale threads are cleaned up.
thread_table_auto_purging_thread_timeout_s sets how long inactive threads are kept before removal.

These options let you balance memory efficiency, CPU usage, and state accuracy, with related metrics (n_drops_full_threadtable, n_store_evts_drops) available to guide tuning.

Static fields

Falco 0.42.0 adds a new static_fields configuration object allowing users to add statically defined fields to the Falco engine. The following example illustrates how to specify new static fields:

static_fields:
 foo: bar
 foo2: ${bar2}

Notice that foo2: ${bar2} leverages the Falco's behavior of expanding env variables in config YAML.

After specifying them, these fields can be used in normal rule conditions, by prepending the static. prefix (e.g.: evt.type=open and static.foo=bar). Moreover, if append_output.suggested_output is true, they'll be automatically appended to each rule output, in the form static_foo=bar.

For more details, see:

Breaking changes and deprecations ⚠️

This version comes with breaking changes that you should be aware of before upgrading.

Event direction and `evt.dir` deprecation

Following the enter events initiative, the evt.dir field, as well as the concept of "direction", have been deprecated in Falco 0.42.0 and will be removed in a future release. Until field removal and since Falco 0.42.0, specifying evt.dir='>' will match nothing, while specifying evt.dir='<' will match everything, with a warning informing the user about the deprecation. Users are encouraged to get rid of any reference to evt.dir, as its presence will result in an error at rules loading time after its removal.

Plugin API changes

Old plugins consuming syscall events not declaring the required event schema version will be incompatible with Falco 0.42.0 and later.

Deprecation warnings

Falco 0.42.0 introduces several deprecation warnings to help users migrate to newer APIs:

evt.dir field deprecation: Rules using the deprecated evt.dir field will now generate warnings;
Enter events drop stats: Prometheus metrics for enter events drop statistics have been deprecated;
Configuration warnings: Enhanced warning system for deprecated configuration options;

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

Stay connected

Join us on social media and in our community calls, held every other Wednesday! It's always great to have new members in the community, and we're looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Introducing Falco 0.41.0

Thu, 29 May 2025 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.41.0!

This version brings several new features, performance enhancements, and bug fixes that streamline Falco’s detection capabilities. During this release cycle, we merged more than 50 PRs on Falco and around 130 PRs for libs and drivers, version 0.21.0 and version 8.1.0, respectively. Thank you to our maintainers and contributors. This would not have been possible without your support and dedication!

To learn everything about the changes, read on!

What’s new? TL;DR

Key features:

Reimplemented container engines support from scratch;
A Kubernetes operator is taking shape;
Falco's config_files configuration gained support to specify the merge strategy;
Modern eBPF driver is now capable of trying to load multiple programs for each event; consequently, sendmmsg and recvmmsg will now make use of bpf_loop eBPF helper where available, boosting their performances;
New proc.aargs field available, ie: a lookup for an ancestor args field;
proc.args gained support for indexed access, to only check a certain argument;
json_include_output_fields configuration key for Falco to control whether output fields are included in the JSON message;
Ongoing work to improve libs code modularity;

Key fixes:

Avoid kmod crashing when a CPU gets enabled at runtime;
Fixed Falco Prometheus metrics with multiple event sources enabled;
Fixed RPM packages evaluation of RPM scripts;
-o options do now correctly override included config_files;

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.41.0 release contains a number of features and UX improvements. Here is a list of some of the key new capabilities.

Reimplemented container engines support

In the Falco 0.41.0 release, the Falco team has completely revised its support for container engines. Key improvements include:

Container support is now a plugin;
The plugin will attach a listener to the engine's SDKs onCreate signal; since onCreate comes way before onStart, we have plenty of time to deliver the container's metadata before the first process in the container is even started;
For now, it is bundled within Falco to avoid breaking changes, but in the future, it will need to be downloaded through falcoctl;

These changes should address all issues related to missing container metadata.

Kubernetes operator

In Falco 0.41.0, we worked hard to create a Falco k8s operator: https://github.com/falcosecurity/falco-operator/. For now, this is considered a technical preview, but we will deliver a fully functional operator very soon. Expect more news in a new blog post!

Security best practices improvements

We are grateful for the suggestions we received from security experts and adopters in our community, and so we implemented the following enhancements:

The modern eBPF probe will no longer store security sensitive settings in the .bss mmapable segment but will use dedicated maps instead. This is a security best practice because it prevents other processes running with elevated privileges from tampering with the map file descriptor, which would be harder to detect. We would like to thank Mouad Kondah for suggesting this change!

Falco will no longer consider rule files or contents of rule directories that do not have a .yml/.yaml extension. This prevents accidental processing of files that are not related to rules. We would like to thank our user Travis Smith for suggesting this change!

Breaking changes and deprecations ⚠️

This version comes with breaking changes, mostly in the configuration interface.

Removed command line options and equivalent configuration options

We removed the already deprecated options -S/--snaplen, -A, and -b, and it is now possible to achieve the same result through the Falco configuration:

for -S/--snaplen: falco_libs.snaplen config key;
for -A: base_syscalls.all config key;
for -b: buffer_format_base64 config key;

The configuration options for the container engines, added in 0.40.0, have been completely dropped in favor of the new plugin init configuration which can be found at https://github.com/falcosecurity/plugins/tree/main/plugins/container#configuration.

You can find more information on breaking changes in the tracking issue.

Behavior changes

Falco will now only consider and consequently load rules whose name ends in .yml or .yaml.

Dropped features

syslog related fields were dropped by libs, since they were unused.

Also, as a consequence of the new container plugin, some breaking changes had to take place:

the musl build is inherently not able to load plugins; that means that it loses container metadata support;
falcosecurity_scap_n_containers and falcosecurity_scap_n_missing_container_images metrics are now moved to the plugin, and their name now have the falcosecurity_plugins_ prefix;
-pc and -pk command line options are now ineffective; it is up to the container and k8smeta plugins to declare suggested fields to be used as output fields; consequently, container_image=%container.image.repository and k8s_ns=%k8s.ns.name changed their name to container_image_repository= and k8s_ns_name=;

Deprecations

In Falco 0.41.0, we have deprecated the following options:

-p cli flag; the only remaining user for it is gVisor, which will be ported to a plugin sooner or later and will then make use of the suggested output fields plugin API;

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

Falco is more mature with each release. Following its graduation, we have published the roadmap for version 1.0.0, which is guiding us in the next steps. For the next release, you can expect more stability, a refined k8s operator, improved performance, and, as always, new detections and fixes.

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Detecting Supply Chain Attacks with Falco Actions

Wed, 19 Mar 2025 00:00:00 +0000

The recently discovered CVE for the GitHub action tj-actions/changed-files brought to light a topic that is really critical for companies: supply chain attacks. With that, we want to discuss and show a bit about how Falco can help your organization detect this kind of attack and other suspect behaviors inside your CI/CD pipeline.

What is Falco?

Falco is a cloud native security tool that provides runtime security across hosts, containers, Kubernetes, and cloud environments. It leverages custom rules on Linux kernel events and other data sources through plugins, enriching event data with contextual metadata to deliver real-time alerts. Falco enables the detection of abnormal behavior, potential security threats, and compliance violations.

What is Falco Actions?

Falco Actions enable you to run Falco in GitHub Actions to detect suspicious behavior in your CI/CD workflows. If you run it in a pull request, the action will create a comment with the findings.

Thanks to ad-hoc Falco rules specific to this use case, these GitHub actions can monitor your GitHub runner and detect software supply chain attacks.

Using Falco Actions

To have Falco inside your pipeline, you need to add these two actions:

falcosecurity/falco-actions/start
falcosecurity/falco-actions/stop

Below you can see an example:

name: CI

on:
 push:
 pull_request:

jobs:
 build:
 runs-on: ubuntu-latest
 permissions:
 contents: read
 actions: read

 steps:
 - uses: actions/checkout@v4

 - name: Start Falco
 uses: falcosecurity/falco-actions/start@main
 with:
 mode: live
 falco-version: '0.40.0'
 verbose: true

 - name: My Custom Step
 run: |
 echo "This is my custom step"

 - name: Stop Falco
 uses: falcosecurity/falco-actions/start@main
 with:
 mode: live
 verbose: true

OBS: main is being used here only to simplify how it works, you should always pin your dependencies to a specific commit SHA.

After the execution, you will be able to see the results at the github action summary.

If you want a more detailed report, you can use the action falcosecurity/falco-actions/analyze; it will allow you to have a better report with information like:

Falco rules triggered during steps' execution.
Contacted IPs
Contacted DNS domains
SHA256 hash of spawned executables
Spawned container images
Written files
A summary of the report generated with OpenAI
Reputation of Contacted IPs
Reputation of SHA256 hashes

For more informations about the usage, you can check the github repository for the actions.

Default rules file

By default, Falco action will detect a variety of events, following the default CICD rules, that can be overridden if you want.

In the example from the tj-actions/changed-files exploit, one rule that would be triggered is the Process Dumping Memory of Others, which was used during the exploit to dump environment variables from the main process and print them as part of the Github runner execution.

The Falco team is always adding new rules to ensure our users get value out of the box, but you can also write your own rules according to your company policy.

Conclusions

These actions are just the beginning of having the Falco capabilities inside the CI/CD pipelines. You can customize and have your own set of rules, keeping all environments and scenarios covered and protected from supply chain attacks.

Let's meet!

As always, we meet every 2 weeks on Wednesday at 4pm UTC in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Enjoy 😎,

Igor and Edson

Blog: Detecting Threats in OVHcloud MKS Audit Logs with Falco

Thu, 13 Mar 2025 00:00:00 +0000

Detecting threats in a Kubernetes cluster can be challenging, we generally don't know where and how to start. The good news is that we have an amount of valuable logs that can help us to know what is happened in the cluster. Indeed, each action requested or done by a user or an app, in a cluster, is recorded in Audit Logs. Kubernetes events are key to understanding the behavior of a cluster.

We already provide plugins that let you parse Audit Logs and use Falco to detect threats from GKE, EKS and AKS clusters. With our latest plugin, you'll now have the same powerful threat detection capabilities for your OVHcloud MKS clusters 🎉.

What is Falco?

Falco is an Open Source cloud-native runtime security tool. It provides near real-time threat detection for cloud, container, and Kubernetes workloads by leveraging runtime insights. Falco can monitor events from various sources, including the Linux kernel, and enrich them with metadata from the Kubernetes API server, container runtime, and more.

Falco can receive Events, compare them to a set of Rules to determine the actions to perform and generate Alerts to different endpoints.

What is the OVH MKS Audit Logs plugin?

The OVH audit logs plugin (k8saudit-ovh) extends Falco's capabilities to OVHcloud Managed Kubernetes Service (MKS) clusters, providing you with the same security insights and threat detection Falco already offers for GKE, EKS and AKS environments.

With this plugin, you can seamlessly integrate MKS Audit Logs into Falco's event processing pipeline, enabling it to identify anomalies, suspicious activities, and policy violations within your MKS-based workloads.

Concretely, when a user executes some kubectl commands in an OVHcloud MKS cluster, Audit Logs will be generated. Falco is listening to them, and depending on the configured rules to follow, it will generate some alerts.

Using OVH MKS Audit Logs plugin

In order to use the OVH MKS Audit Logs plugin, you must follow several steps:

deploy an OVHcloud LDP (Logs Data Platform)
create a data stream into this LDP
connect an OVHcloud MKS cluster to the data stream (to send Audit Logs into it)

To be able to access our Kubernetes clusters' Audit Logs, you need to deploy an LDP. LDP is the managed platform for collecting, processing, analyzing, and storing your logs of the OVHcloud products. Deploy an LDP (Bare Metal Cloud universe) with whatever plan you want.

OVHcloud Kubernetes Audit Logs will be stored in a data stream. The OVHcloud Audit Logs Falco plugin receive the audit logs through Websocket so you need to enable Websocket broadcasting when you create the data stream on LDP.

Retrieve the Websocket URL, follow the guide to do so. The Websocket address have this kind of format: wss://gra.logs.ovh.com/tail/?tk=

Finally, you have to connect a MKS cluster to the LDP data stream.

Configuring Falco to use OVH Audit Logs plugin

Running locally

If you have a Falco running locally, using falcoctl, add the falcosecurity index (if it's not already the case) and install the k8saudit-ovh Falco plugin:

# Add falcosecurity index
sudo falcoctl index add falcosecurity https://falcosecurity.github.io/falcoctl/index.yaml
# Install k8saudit-ovh Falco plugin
sudo falcoctl artifact install k8saudit-ovh

Fill your falco.yaml file in order to add the plugin configuration:

plugins:
 - name: k8saudit-ovh
 library_path: /usr/share/falco/plugins/libk8saudit-ovh.so
 open_params: "<OVH LDP WEBSOCKET URL>" # gra<x>.logs.ovh.com/tail/?tk=<ID>
 - name: json
 library_path: libjson.so
 init_config: ""
load_plugins: [k8saudit-ovh, json]

stdout_output:
 enabled: true

Running in a Kubernetes cluster

If you have a Falco running in a Kubernetes cluster (on OVHcloud MKS or on another cluster), deployed with Helm, create a values.yaml file with the following content:

tty: true
kubernetes: false

# Just a Deployment with 1 replica (instead of a Daemonset) to have only one Pod that pulls the MKS Audit Logs from a OVHcloud LDP
controller:
 kind: deployment
 deployment:
 replicas: 1

falco:
 rule_matching: all
 rules_files:
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit-ovh
 library_path: libk8saudit-ovh.so
 open_params: "gra<x>.logs.ovh.com/tail/?tk=<ID>" # Replace with your LDP Websocket URL
 - name: json
 library_path: libjson.so
 init_config: ""
 # Plugins that Falco will load. Note: the same plugins are installed by the falcoctl-artifact-install init container.
 load_plugins: [k8saudit-ovh, json]

driver:
 enabled: false
collectors:
 enabled: false

# use falcoctl to install automatically the plugin and the rules
falcoctl:
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 indexes:
 - name: falcosecurity
 url: https://falcosecurity.github.io/falcoctl/index.yaml
 artifact:
 allowedTypes:
 - plugin
 - rulesfile
 install:
 resolveDeps: false
 refs: [k8saudit-rules:0, k8saudit-ovh:0.1, json:0]
 follow:
 refs: [k8saudit-rules:0]

This values.yaml file will install Falco with the k8saudit-ovh and the json plugins.

Install the latest version of Falco with helm install command:

$ helm install falco --create-namespace --namespace falco --values=values.yaml falcosecurity/falco

This command will install the latest version of Falco, with the k8saudit-ovh and json plugins, and create a new falco namespace.

Or if you already have Falco deployed in a Kubernetes cluster, you can use the helm update command instead:

$ helm update falco --create-namespace --namespace falco --values=values.yaml falcosecurity/falco

Once the Falco pod is ready, run the following command to see the logs:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco

You should see logs like that:

$ kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco

Mon Feb 10 09:15:35 2025: /etc/falco/k8s_audit_rules.yaml | schema validation: ok
Mon Feb 10 09:15:35 2025: Hostname value has been overridden via environment variable to: my-pool-1-node-921b61
Mon Feb 10 09:15:35 2025: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Mon Feb 10 09:15:35 2025: Starting health webserver with threadiness 2, listening on 0.0.0.0:8765
Mon Feb 10 09:15:35 2025: Loaded event sources: syscall, k8s_audit
Mon Feb 10 09:15:35 2025: Enabled event sources: k8s_audit
Mon Feb 10 09:15:35 2025: Opening 'k8s_audit' source with plugin 'k8saudit-ovh'
{"hostname":"my-pool-1-node-921b61","output":"09:15:40.698757000: Warning K8s Operation performed by user not in allowed list of users (user=csi-cinder-controller target=csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3/volumeattachments verb=patch uri=/apis/storage.k8s.io/v1/volumeattachments/csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3/status resp=200)","output_fields":{"evt.time":1739178940698757000,"ka.response.code":"200","ka.target.name":"csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3","ka.target.resource":"volumeattachments","ka.uri":"/apis/storage.k8s.io/v1/volumeattachments/csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3/status","ka.user.name":"csi-cinder-controller","ka.verb":"patch"},"priority":"Warning","rule":"Disallowed K8s User","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:15:40.698757000Z"}
{"hostname":"my-pool-1-node-921b61","output":"09:15:57.508657000: Warning K8s Operation performed by user not in allowed list of users (user=yacht target=my-pool-1.18051c0a88716868/events verb=patch uri=/api/v1/namespaces/default/events/my-pool-1.18051c0a88716868 resp=403)","output_fields":{"evt.time":1739178957508657000,"ka.response.code":"403","ka.target.name":"my-pool-1.18051c0a88716868","ka.target.resource":"events","ka.uri":"/api/v1/namespaces/default/events/my-pool-1.18051c0a88716868","ka.user.name":"yacht","ka.verb":"patch"},"priority":"Warning","rule":"Disallowed K8s User","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:15:57.508657000Z"}
{"hostname":"my-pool-1-node-921b61","output":"09:15:57.807013000: Warning K8s Operation performed by user not in allowed list of users (user=yacht target=my-pool-1/nodepools verb=update uri=/apis/kube.cloud.ovh.com/v1alpha1/nodepools/my-pool-1/status resp=200)","output_fields":{"evt.time":1739178957807013000,"ka.response.code":"200","ka.target.name":"my-pool-1","ka.target.resource":"nodepools","ka.uri":"/apis/kube.cloud.ovh.com/v1alpha1/nodepools/my-pool-1/status","ka.user.name":"yacht","ka.verb":"update"},"priority":"Warning","rule":"Disallowed K8s User","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:15:57.807013000Z"}

Let's test it!

In order to test Falco we need to know which rules are installed by default. In our case, as we defined it in the values.yaml file, the k8saudit-ovh plugin follow the k8s_audit_rules.yaml file. You can take a look at them in order to know them.

In this blog post we will test one of the well-known default k8s audit rules:

- rule: Attach/Exec Pod
 desc: >
 Detect any attempt to attach/exec to a pod
 condition: kevt_started and pod_subresource and (kcreate or kget) and ka.target.subresource in (exec,attach) and not user_known_exec_pod_activities
 output: Attach/Exec to pod (user=%ka.user.name pod=%ka.target.name resource=%ka.target.resource ns=%ka.target.namespace action=%ka.target.subresource command=%ka.uri.param[command])
 priority: NOTICE
 source: k8s_audit
 tags: [k8s]

This rule is interesting because an event will be generated if/when an user execute commands in a pod.

Let’s test the rule!

In a tab of your terminal, watch the coming logs:

$ kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco -f

In an another tab of your terminal, create a Nginx pod and execute a command into it:

$ kubectl run nginx --image=nginx

$ kubectl exec -it nginx -n hello-app -- cat /etc/shadow

Several seconds laters, in the logs you should see this you will see this Attach/Exec to pod logs:

...
{"hostname":"my-pool-1-node-921b61","output":"09:29:46.302906000: Notice Attach/Exec to pod (user=kubernetes-admin pod=nginx-676b6c5bbc-4xc6t resource=pods ns=hello-app action=exec command=cat)","output_fields":{"evt.time":1739179786302906000,"ka.target.name":"nginx-676b6c5bbc-4xc6t","ka.target.namespace":"hello-app","ka.target.resource":"pods","ka.target.subresource":"exec","ka.uri.param[command]":"cat","ka.user.name":"kubernetes-admin"},"priority":"Notice","rule":"Attach/Exec Pod","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:29:46.302906000Z"}
...

💪

Let's meet!

If you have planned to go to the KubeCon + CloudNative Con EU 2025 at London, don't hesitate to stop at the Falco booth in the Project Pavillon!

As always, we meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Enjoy 😎,

Aurélie

Blog: Falco plugin for collecting AKS audit logs

Sun, 09 Mar 2025 00:00:00 +0000

Troubleshooting Kubernetes events is challenging due to the multitude of data sources involved: container logs, Kubernetes events, cloud logs, and more. Among these sources, Kubernetes audit logs are especially valuable for identifying threats, as every action passing through the Kubernetes API server is recorded there.

We already provide plugins that let you parse and use Falco to detect threats in audit logs from GKE and EKS clusters. With our latest plugin, you'll now have the same powerful threat detection capabilities for your Azure AKS clusters.

What is Falco?

Falco is a Cloud Native Computing Foundation project that provides runtime threat detection. Out of the box, Falco examines syscalls to alert you to any suspicious activity. And, since containers share the same kernel as their host, Falco can monitor not only activity on the host but also activity on all of the containers running on that host. Moreover, Falco pulls data from both Kubernetes and the container runtime to add additional context to its alerts.

With Falco running on your GKE clusters you can be notified of a wide variety of events, such as:

Did someone start a container with high privileges?
Has someone shelled into a running container?
Has an executable been added to the container after it was deployed?

These are just a few examples. Falco has over 80 rules that can be used to make you aware of not only external threats but also when clusters aren't being operated in accordance with industry best practices.

What is the AKS audit logs plugin?

The AKS audit logs plugin extends Falco's capabilities to Microsoft Azure Kubernetes Service (AKS) clusters, providing you with the same security insights and threat detection Falco already offers for GKE and EKS environments. With this plugin, you can seamlessly integrate AKS audit logs into Falco's event processing pipeline, enabling it to identify anomalies, suspicious activities, and policy violations within your AKS-based workloads.

Using AKS audit logs plugin

In order to use the AKS audit log plugin, you must first configure your AKS cluster to ship the logs where we can fetch them.

The current supported output source is Event hub, so when following the guide to configure your AKS audit logs, you must have Eventhub enabled. You can also optionally send it to other sources:

Once you have the stream enabled, you must create or reuse a storage account blob container so that the plugin can track the last event that was consumed, which is done trough checkpoints.

Configuring Falco to use AKS audit logs plugin

First, using falcoctl, download the plugin:

sudo falcoctl artifact install k8saudit-aks

In your falco.yaml file, you must add the plugin configuration and later enable the plugin

config_files:
 - /etc/falco/config.d

watch_config_files: true

plugins:
# - name: k8saudit
# library_path: libk8saudit.so
# init_config: ""
# open_params: "http://:9765/k8s-audit"
# - name: json
# library_path: libjson.so
 - name: k8saudit-aks
 library_path: libk8saudit-aks.so
 init_config:
 event_hub_name: ${EVENTHUB_NAME}
 blob_storage_container_name: ${BLOB_STORAGE_CONTAINER_NAME}
 event_hub_namespace_connection_string: ${EVENTHUB_NAMESPACE_CONNECTION_STRING}
 blob_storage_connection_string: ${BLOB_STORAGE_CONNECTION_STRING}

load_plugins: [k8saudit-aks]

stdout_output:
 enabled: true

Once they are exported, run Falco and after some seconds you'll logs informing the k8saudit-aks plugin was loaded:

falco -c /etc/falco/falco.yaml -r /etc/falco/falco_rules.yaml

Tue Dec 17 18:02:07 2024: Opening 'k8s_audit' source with plugin 'k8saudit-aks'
2024/12/17 21:02:07 [k8saudit-aks] opened connection to blob storage
2024/12/17 21:02:07 [k8saudit-aks] opened blob checkpoint connection
2024/12/17 21:02:07 [k8saudit-aks] opened consumer client
2024/12/17 21:02:07 [k8saudit-aks] created eventhub processor

Testing out!

Append rule to falco_rules.yaml file:

- rule: K8s Audit Event Detected
 desc: A test rule that detects any Kubernetes audit event
 condition: ka.req exists
 output: "K8s Audit Event Detected: %ka.req"
 priority: DEBUG
 source: k8s_audit
 tags: [testing, k8s_audit]

$ falco -c /etc/falco/falco.yaml -r /etc/falco/falco_rules.yaml

Then, you should see initialization message, followed by some events from your AKS cluster. Since we have debug enabled, you should see some events from the aksService:

Thu Dec 19 11:44:55 2024: Falco version: 0.39.2 (aarch64)
Thu Dec 19 11:44:55 2024: Falco initialized with configuration files:
Thu Dec 19 11:44:55 2024: /etc/falco/config.d/engine-kind-falcoctl.yaml | schema validation: ok
Thu Dec 19 11:44:55 2024: /etc/falco/falco.yaml | schema validation: ok
Thu Dec 19 11:44:55 2024: System info: Linux version 6.8.0-51-generic (buildd@bos03-arm64-031) (aarch64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec 5 13:32:09 UTC 2024
Thu Dec 19 11:44:55 2024: Loading plugin 'k8saudit-aks' from file /usr/share/falco/plugins/libk8saudit-aks.so
Thu Dec 19 11:44:55 2024: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Thu Dec 19 11:44:55 2024: Loading rules from:
Thu Dec 19 11:44:55 2024: /etc/falco/falco_rules.yaml | schema validation: ok
Thu Dec 19 11:44:55 2024: /etc/falco/falco_rules.local.yaml | schema validation: none
Thu Dec 19 11:44:55 2024: /etc/falco/falco_aks_audit.yaml | schema validation: ok
Thu Dec 19 11:44:55 2024: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Thu Dec 19 11:44:55 2024: Starting health webserver with threadiness 4, listening on 0.0.0.0:8765
Thu Dec 19 11:44:55 2024: Loaded event sources: syscall, k8s_audit
Thu Dec 19 11:44:55 2024: Enabled event sources: k8s_audit, syscall
Thu Dec 19 11:44:55 2024: Opening 'k8s_audit' source with plugin 'k8saudit-aks'
2024/12/19 14:44:55 [k8saudit-aks] opened connection to blob storage
2024/12/19 14:44:55 [k8saudit-aks] opened blob checkpoint connection
2024/12/19 14:44:55 [k8saudit-aks] opened consumer client
2024/12/19 14:44:55 [k8saudit-aks] created eventhub processor
Thu Dec 19 11:44:55 2024: Opening 'syscall' source with modern BPF probe.
Thu Dec 19 11:44:55 2024: One ring buffer every '2' CPUs.

10:52:03.348668000: Debug K8s Audit Event Detected: verb=create, user=aksService, groups=(system:masters,system:authenticated), target=<NA>

Let's meet!

As always, we meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Enjoy 😎,

Igor

Blog: Introducing Falco 0.40.0

Tue, 28 Jan 2025 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.40.0!

This version brings several new features, performance improvements, and bug fixes that streamline Falco’s detection capabilities. During this release cycle, we merged more than 52 PRs on Falco and more than 150 PRs for libs and drivers, version 0.20.0 and version 8.0.0 respectively. Thank you to our maintainers and contributors, as this would not happen without your support and dedication!

To learn all about these changes, read on!

What’s new? TL;DR

Key features:

Streamlined Falco docker images;
Falco static build has been reintroduced for x86_64 binary using musl;
New process filters allow to filter events based on process metadata;
Added support for sendmmsg and recvmmsg syscalls parameters;
Plugins suggested output fields are now available in the Falco engine;

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.40.0 release contains a number of features and UX improvements. Here is a list of some of the key new capabilities.

Streamlined Falco docker images

In the Falco 0.40.0 release, the Falco team has streamlined the Docker images to improve usability and performance. The new images are designed to be more efficient and easier to use, providing a better experience for users deploying Falco in containerized environments.

Key improvements include:

Reduced Image Size: The new images are smaller, which reduces the time required to pull and deploy them.
Optimized Layers: The layers in the Docker images have been optimized to improve build times and caching efficiency.
Enhanced Security: The images have been hardened to enhance security, reducing potential vulnerabilities.

These changes make it easier to deploy and manage Falco in various environments.

Introducing new process filters

A new set of process filters are made available in this release: proc.pgid, proc.pgid.name, proc.pgid.exe, proc.pgid.exepath, proc.is_pgid_leader. These filters enable users to filter events based on process metadata, such as the process name, executable path, and arguments. The new filters introduce the pgid field, which is directly obtained from the kernel. This ID corresponds to the host pid namespace, aiding in the creation of more reliable rules.

Plugins suggested output fields

The Falco engine now supports plugins that can suggest output fields. This feature allows plugins to provide additional context and information about an event, enhancing its visibility and understanding. The suggested output fields are displayed in the Falco output, giving users valuable insights into the event and its context. By leveraging this feature, Falco makes it easier for users to take advantage of the metadata provided by plugins and improve their security monitoring and incident response capabilities. New output fields are added only if the option is enabled and the plugin supports this new feature.

Keep an eye on the existing plugins to be updated to support the new feature.

Breaking changes and deprecations ⚠️

This version comes with breaking changes, mostly in the configuration interface.

Removed command line options and equivalent configuration options

We removed the already deprecated options --cri, --disable-cri-async, and is now possible to achieve the same result through the Falco configuration. A new configuration options has been introduced to enable and configure the supported container engines in Falco:

container_engines:
 docker:
 enabled: true
 cri:
 enabled: true
 sockets: ["/run/containerd/containerd.sock", "/run/crio/crio.sock", "/run/k3s/containerd/containerd.sock"]
 disable_async: false
 podman:
 enabled: true
 lxc:
 enabled: true
 libvirt_lxc:
 enabled: true
 bpm:
 enabled: true

Please keep in mind that the new configuration options are tagged as incubating and may change in the future.

You can also use the -o command line option:

--cri <socket_path>: use -o container_engines.cri.enabled=true, -o container_engines.cri.sockets[]=<socket_path, -o container_engines.cri.disable_async=true instead to enable the CRI runtime and configure the socket path and disable the async mode.

You can find more information on breaking changes in the tracking issue.

New docker images

With the growing prominence of the modern eBPF probe, in Falco 0.38.0 we made the strategic decision to adopt it as the default driver for Falco. This shift brings key advantages to our distribution system by removing the need to bundle the full driver-building toolchain in the standard Falco distribution. As a result, we’re transitioning the default Falco image to a no-driver/distroless configuration, simplifying deployments and reducing system complexity. For users seeking alternative setups, a different container image will still be available.

In light of this change, we’ve re-evaluated all Docker images:

Image Name	Tag (aliases)	Description
`falcosecurity/falco`	`x.y.z` (`latest`)	Distroless image without driver building toolchain support, based on the latest released tar.gz of Falco. No tools or `falcoctl` included.
`falcosecurity/falco`	`x.y.z-debian` (`latest-debian`)	Debian-based image without driver building toolchain support, based on the latest released Deb of Falco. May include some tools (e.g., `jq`, `curl`), but not `falcoctl`.
`falcosecurity/falco-driver-loader`	`x.y.z` (`latest`)	Based on `falcosecurity/falco:x.y.z-debian`, plus driver building toolchain support and the latest version of `falcoctl`. Recommended only when modern eBPF is unsupported.
`falcosecurity/falco-driver-loader`	`x.y.z-buster` (`latest`)	Similar to `falcosecurity/falco-driver-loader`, but based on a legacy Debian image (i.e., `buster`). Recommended only for old kernel versions.

The following images have been deprecated and are not anymore available in the Falco 0.40.0 release:

Image Name	Reason
`falcosecurity/falco-distroless`	Deprecated in favor of `falcosecurity/falco:x.y.z`.
`falcosecurity/falco-no-driver`	Deprecated in favor of `falcosecurity/falco:x.y.z-debian` (essentially the same image with a new name).

Deprecations

In Falco 0.40.0, we have deprecated the following options:

-S / --snaplen cli flag has been deprecated in favor of the falco_libs.snaplen configuration option;
-A cli flag has been deprecated in favor of the base_syscalls.all configuration option;
-b cli flag has been deprecated in favor of the buffer_format_base64 configuration option;

Worthy of note

Release artifacts are now built with zig, using very recent versions of clang. This change alone has resulted in up to 10% speedup in userspace benchmarks.

The first graph shows the events processed by userspace per second:

The following one shows the average of multiple runs of Google Benchmark framework embedded in libsinsp:

Additionally, artifacts now use jemalloc as the allocator library. This should help mitigate some memory fragmentation-related issues.

Furthermore, Falco debug symbol files are now attached to GitHub releases. Falco is built in RelWithDebInfo mode, enabling users to download debug symbols and attach them to their debugging sessions.

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

Falco is more mature with each release. Following its graduation we have published the roadmap for version 1.0.0 which is guiding us in the next steps. For the next release, you can expect more stability, a new container plugin, refinements to our deployment methods with a k8s operator, and as always new detections and fixes.

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Introducing Falco 0.39.2

Thu, 21 Nov 2024 00:00:00 +0000

Today we announce the release of Falco 0.39.2 🦅!

Fixes

Falco's 0.39.2 is a small patch release that includes some important bugfixes for modern eBPF driver:

check cred field is not NULL before the access; this enables Falco back with modern eBPF driver to work on GKE
address verifier issues on kernel versions >=6.11.4: there was a kernel-breaking change in the tail call ebpf API merged into the 6.11.4 to fix a CVE. Adapt our code to work again on these new versions.

Thanks to everyone in the community for helping us spot these annoying bugs and improving Falco every day 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.39.2, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest, you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Enjoy 😎,

Federico

Blog: How to Deploy Falco with k8s-metacollector + k8smeta Plugin

Mon, 14 Oct 2024 00:00:00 +0000

In today's cloud-native world, securing Kubernetes environments has become increasingly critical as containerized workloads gain complexity. Falco is designed to monitor and detect anomalous activities in Kubernetes clusters and container environments. By continuously observing system calls and enriching event data with metadata, Falco ensures that any suspicious behavior is detected in real-time, protecting against threats like privilege escalations, file tampering, and network anomalies.

In this tutorial, we will guide you through deploying Falco with two powerful components: k8s-metacollector and the k8smeta plugin. These tools significantly enhance Falco’s security event detection by adding important Kubernetes context, such as pod names, namespaces, deployment details, to the alerts. Additionally, we will explore how to leverage the new append_output feature introduced in Falco version 0.39.0. This feature allows you to append extra metadata fields to Falco’s output, without the need to modify your rules.

By the end of this guide, you will have a Falco setup capable of detecting security issues in Kubernetes with enriched metadata output, ensuring you get a complete picture of your cluster’s security posture. Whether you're an experienced Kubernetes administrator or just starting to explore container security, this guide will help you make the most of Falco's capabilities in a Kubernetes environment.

What You'll Learn:

The purpose and benefits of using the k8s-metacollector and k8smeta plugin to enrich Falco alerts with Kubernetes-specific data.
How to deploy Falco with the k8smeta plugin on a Kubernetes cluster.
How to configure and use the append_output feature to enhance Falco alerts with additional metadata fields.

Prerequisites:

A working Kubernetes cluster and some familiarity with Kubernetes concepts.
Basic knowledge of Falco and how it works.
Helm installed on your system (for easy deployment of Falco).

Let’s dive in and set up a Falco deployment that will give you deeper security insights for your Kubernetes workloads.

Step 1: Understanding k8s-metacollector and k8smeta Plugin

As Kubernetes has become the de facto platform for orchestrating containerized applications, it’s important to gain full visibility into what's happening within your cluster, especially when it comes to security monitoring. Falco can detect suspicious activities based on system calls, but to make these alerts more actionable, additional context about your Kubernetes resources (such as pod names, namespaces, and labels) is invaluable.

That’s where the k8s-metacollectorand k8smeta plugin come in.

What is the k8s-metacollector?

The k8s-metacollector is responsible for gathering Kubernetes metadata for security events and sending that information to Falco. It collects key information for different resources from your Kubernetes cluster, such as:

Pods;
Namespaces;
ReplicaSets;
Services;
Deployments;

The collected metadata provides greater clarity about where and why certain events are happening, which is crucial for pinpointing and mitigating security incidents in large-scale Kubernetes environments. Without this context, security alerts may lack the detail needed for quick and effective response.

What is the k8smeta Plugin?

The k8smeta plugin is a source plugin for Falco that works in tandem with the k8s-metacollector. While Falco generates alerts based on detected anomalies, the k8smeta plugin enriches these alerts with Kubernetes-specific metadata, which allows you to understand exactly which Kubernetes entities (pods, deployments, namespaces) are involved in the detected event. This context is vital when you're trying to correlate security incidents with the resources they affect.

Key benefits of the k8smeta plugin include:

Enriched Alerts: Falco alerts become more informative with Kubernetes-specific data like pod names, namespaces, and deployment names.
Improved Debugging: Knowing exactly which pod or namespace is involved in an alert can significantly reduce the time spent debugging and fixing security issues.
Event Correlation: The plugin makes it easier to correlate low-level system events with higher-level Kubernetes concepts, providing a clearer view of what's happening in your cluster.

By using the k8s-metacollector and k8smeta plugin together, you transform Falco’s raw system call data into rich, actionable insights that give you full visibility into your Kubernetes environment.

Step 2: Installing Falco, k8s-metacollector, and k8smeta Plugin with Helm and Configuring append_output

Deploying Falco along with the k8s-metacollector and the k8smeta plugin using Helm is a seamless process. This step will guide you through adding the Falco Security Helm chart repository, installing Falco, enabling the k8s-metacollector, and configuring the append_output feature to append Kubernetes metadata to Falco alerts.

Step 2.1: Add the Falco Helm Chart Repository

Before you install Falco, you need to add the official Falco Security Helm chart repository to your Helm setup. Run the following command:

helm repo add falcosecurity https://falcosecurity.github.io/charts

Update your local Helm repositories to ensure you’re using the latest chart version:

helm repo update

Step 2.2: Install Falco with k8s-metacollector and append_output

With the repository added, use the following command which includes the additional settings to enable the collection of Kubernetes metadata and to append this metadata to Falco alerts:

helm install falco falcosecurity/falco \
 --version 4.11.1 \
 --namespace falco \
 --create-namespace \
 --set collectors.kubernetes.enabled=true \
 --set tty=true \
 --set-json 'falco.append_output=[{"match": {"source": "syscall"},"extra_output": "pod_uid=%k8smeta.pod.uid, pod_name=%k8smeta.pod.name, namespace_name=%k8smeta.ns.name"}]'

Breaking Down the Command:

helm install falco falcosercurity/falco: Installs Falco using the latest chart from the Falco Security repository.
--version 4.11.1: Uses the 4.11.1 version of the chart. At the writing time it's the latest version.
--namespace falco: Deploys Falco into the falco namespace. This helps keep Falco’s resources organized separately from other applications.
--create-namespace: Automatically creates the falco namespace if it doesn’t already exist.
--set collectors.kubernetes.enabled=true: Enables the k8s-metacollector and configures the k8smeta plugin.
--set tty=true: Ensures that Falco logs are emitted as soon as possible.
--set-json 'falco.append_output=...': Configures the append_output feature to append specific Kubernetes metadata fields to Falco’s alerts.

Why Use the append_output Feature?

The append_output feature allows you to enrich Falco alerts with additional metadata, providing a clearer view of which Kubernetes resources are involved in each security event. This context helps security teams quickly understand the severity and scope of an incident.

For example, an alert will now include:

pod_uid: To precisely identify the pod.
pod_name: To know which pod triggered the alert.
namespace_name: Namespace where the pod is running.

Step 2.3: Verifying the Installation

Once the installation is complete, you can verify that Falco and the k8s-metacollector are working as expected by checking the status of the Falco pod in the Falco namespace:

kubectl get pods -n falco

You should see the Falco pods running successfully.

Step 3: Testing the Setup

Now that everything is in place, it's time to test the setup by deploying a simple Nginx pod and triggering Falco to generate security alerts enriched with Kubernetes metadata.

Step 3.1: Deploy an Nginx Pod

To create some activity that Falco can monitor, start by deploying an Nginx pod in the falco namespace:

kubectl run nginx --image=nginx --namespace falco

This command will launch an Nginx container in the falco namespace.

Step 3.2: Wait for the Nginx Pod to Run

Confirm that the Nginx pod is up and running by checking its status:

kubectl get pods -n falco

Once the pod is in the Running state, you can proceed to the next step.

Step 3.3: Exec Into the Nginx Pod to Trigger Alerts

Exec into the running Nginx pod to simulate an interactive terminal session, which is something Falco is configured to detect:

kubectl exec -it nginx -n falco -- /bin/bash

This command opens a shell session inside the Nginx container. Inside the container, run some basic commands like ls or echo to generate system calls that Falco can monitor.

Step 3.4: Check Falco Logs for Alerts

After executing inside the Nginx pod, check the Falco logs to see if any alerts were triggered by the kubectl exec action:

kubectl logs -n falco -l app.kubernetes.io/name=falco

In the logs, you should see alerts related to the interactive terminal session such as:

13:18:57.434030270: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=root user_uid=0 user_loginuid=-1 process=bash proc_exepath=/usr/bin/bash parent=containerd-shim command=bash terminal=34816 exe_flags=EXE_WRITABLE|EXE_LOWER_LAYER container_id=7cff9da475c6 container_image=docker.io/library/nginx container_image_tag=latest container_name=nginx k8s_ns=falco k8s_pod_name=nginx) pod_uid=2f20370c-6e0b-44b8-8ea1-2aa786d80f13, pod_name=nginx, namespace_name=falco

This confirms that Falco is properly configured to detect activity inside the pod and append useful Kubernetes metadata to the alerts.

Key Takeaways:

In this tutorial, we explored how to deploy Falco with the k8s-metacollector and k8smeta plugin to enhance security monitoring in a Kubernetes environment. By enabling Falco’s append_output feature, we were able to enrich security alerts with vital Kubernetes metadata such as pod UID, pod name, and namespace, making the alerts more actionable and informative.

Enhanced Alert Context: By appending Kubernetes metadata, you get more contextualized and meaningful alerts, enabling better incident investigation and faster resolution.
Seamless Integration: Thanks to Helm, deploying Falco alongside the k8s-metacollector and k8smeta plugin is easy and efficient, requiring just a few simple commands.
Real-Time Threat Detection: Falco continuously monitors system calls and Kubernetes events in real-time, ensuring that you’re always aware of potentially suspicious or malicious activities within your cluster.

Blog: Introducing Falco 0.39.1

Wed, 09 Oct 2024 00:00:00 +0000

Today we announce the release of Falco 0.39.1 🦅!

Fixes

Falco's 0.39.1 is a small patch release that includes some important bugfixes:

Fixed a crash when using plugin with event parsing capabilities (eg: k8smeta plugin)
Fixed a bug while parsing -o key={object} command line arguments, when the object definition contains a comma
Improved config json schema to allow null init_config for plugin info

Thanks to everyone in the community for helping us with spotting these annoying bugs and improving Falco every day 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.39.1, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Enjoy 😎,

Federico

Blog: Introducing Falco 0.39.0

Tue, 01 Oct 2024 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.39.0!

This version brings several new features, performance improvements, and bug fixes that streamline Falco’s detection capabilities. During this release cycle, we merged more than 50 PRs on Falco and more than 100 PRs for libs and drivers, version 0. 18.0 and version 7.3.0 respectively. Thank you to our maintainers and contributors, as this would not happen without your support and dedication!

To learn all about these changes, read on!

What’s new? TL;DR

Key features:

Basename operator retrieves the basename of a given path;
New fields added in proc and fd classes #1916 #1936;
Regular expression operator can be used to match values in string fields;
Append output allows to add output text or fields to a subset of loaded rules;
Schema validation for config and rules files allows Falco to warn users when unknown keys are used;
Improved engine selection in Kubernetes environments driver loader will automatically pick the most compatible driver for each node in the cluster.

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.39.0 release contains a number of feature and UX improvements, here are list of some of the key new capabilities.

New Operators

The basename() transformer operator extracts the base name, i.e. the filename without directory, of the input field. Note that the behavior ofbasename() in Falco is slightly different from the Unix basename program. For instance, basename (proc.exepath) will evaluate to "cat" for /usr/bin/cat but will evaluate to an empty string ("") for /usr/bin/. This allows, for instance, to write expressions like basename(proc.exepath) = cat to match against the original executable name even if it has been symlinked without knowing the full path, or any other file name based detection.

The regex operator checks if a string field matches a regular expression. Please note that the regex operator is considerably slower (up to an order of magnitude) than the above operators that work with strings, which are highly recommended for simple comparisons. The supported regex flavor is from the Google RE2 library. Example: fd.name regex [a-z]*/proc/[0-9]+/cmdline.

Introducing the Append Output Feature

In response to long-standing community requests, Falco has introduced a new feature in version 0.39.0 that allows users to add custom outputs and fields to events generated by Falco. This new functionality, called append_output, gives users greater control over the data produced by Falco rules.

With the append_output option, you can now easily add extra output to rules based on source, tag, or rule name—or even apply it to all rules without conditions. This option is configurable in the falco.yaml file and works by specifying a list of append entries, which are applied in the order they appear.

Here’s an example configuration:

append_output:
 - match:
 source: syscall
 extra_output: "on CPU %evt.cpu"
 extra_fields:
 - home_directory: "${HOME}"
 - evt.hostname

In this example, any rule with the syscall source will have the string on CPU %evt.cpu appended to the end of the default output line. Additionally, extra fields such as home_directory and evt.hostname will be visible in the JSON output under the output_fieldskey but won’t appear in the regular text output. Notably, environment variables are also supported.

This option is also available on the command line using the -o flag. For example:

falco ... -o 'append_output[]={"match": {"source": "syscall"}, "extra_fields": ["evt.hostname"], "extra_output": "on CPU %evt.cpu"}'

The introduction of append_output offers Falco users a flexible way to enrich event outputs, providing deeper visibility and customization tailored to their monitoring needs.

Dynamic Driver Selection in Falco with Helm: Simplifying Multi-Node Deployments

Deploying across diverse Kubernetes environments just got easier! When using the official Falco Helm chart and setting driver.kind=auto, the driver loader now intelligently handles the heavy lifting for you.

Here's how it works: the driver loader will automatically generate a new Falco configuration file and select the correct engine driver based on the specific node Falco is deployed on. This means whether you're using eBPF, kmod, or a modern eBPF driver, Falco will configure itself dynamically depending on the environment.

In many Kubernetes clusters, nodes can differ in terms of kernel versions, capabilities, and driver compatibility. With this new auto-selection feature, you can seamlessly deploy different Falco drivers across various nodes within the same cluster. Here’s a simple illustration:

+-------------------------------------------------------+
| Kubernetes Cluster |
| |
| +-------------------+ +-------------------+ |
| | Node 1 | | Node 2 | |
| | | | | |
| | Falco (eBPF probe) | | Falco (kmod) | |
| +-------------------+ +-------------------+ |
| |
| +-------------------+ |
| | Node 3 | |
| | | |
| | Falco (modern eBPF)| |
| +-------------------+ |
+-------------------------------------------------------+

In this example:

Node 1 is configured with the eBPF probe driver.
Node 2 uses the kmod (kernel module) driver.
Node 3 leverages the modern eBPF driver for cutting-edge performance.

This setup gives you flexibility and ensures that each node in your Kubernetes cluster is running Falco in the most optimized way possible, without manual configuration. Simply set driver.kind=auto in the Helm chart and let Falco do the rest.

Breaking changes and deprecations ⚠️

This version comes with breaking changes, mostly in the configuration interface

Removed command line options and equivalent configuration options

We removed the already deprecated options -D, -t, -T and is now possible to achieve the same result through the Falco configuration You con still use the -o command line option:

-T : use -o rules[].disable.tag=<tag> instead. Turn off any rules with a tag=<tag>. This option can be passed multiple times. This option can not be mixed with -t;
-t : use -o rules[].disable.rule=* -o rules[].enable.tag=<tag> instead. Only enable those rules with a tag=<tag>. This option can be passed multiple times;
D : use -o rules[].disable.rule=<wildcard-pattern> instead. Turn off any rules with names having the substring . This option can be passed multiple times.

You can find more information on breaking changes in the tracking issue.

Notable Bug Fixes

Prometheus Compliant metrics: some metrics have been reworked to follow the prometheus best practices #3319;
Fixed ebpf drivers to use the correct memory barrier primitive for ARM64, preventing to read incomplete data from the ring buffers #2067;
Fixed an issue where stats messages were written to stdout and could mix with regular Falco event output #3338;
fs.path fields now account for dirfd, fixing discrepancies with fd.name #1993;

Deprecations

In Falco 0.39.0, the --cri and --disable-cri-async options were deprecated, and they will be completely removed in Falco 0.40.0. Moving forward, configuring container runtimes should be done through the falco.yaml file. Below is an example of the new configuration format:

container_engines:
 docker:
 enabled: true
 cri:
 enabled: true
 sockets: ["/run/containerd/containerd.sock", "/run/crio/crio.sock", "/run/k3s/containerd/containerd.sock"]
 disable_async: false
 podman:
 enabled: true
 lxc:
 enabled: true
 libvirt_lxc:
 enabled: true
 bpm:
 enabled: true

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

Falco is more mature with each release. Following its graduation we have published the roadmap for version 1.0.0 which is guiding us in the next steps. For the next release, you can expect more stability, streamlined container images, refinements to our rule syntax, new detections and plugins.

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Introducing Falco 0.38.2

Mon, 19 Aug 2024 00:00:00 +0000

Today we announce the release of Falco 0.38.2 🦅!

Fixes

Falco's 0.38.2 is a patch release that includes the most important bugfixes addressed this summer ☀️:

Fixed a crash when using transformer operators (e.g. tolower()) with a parameter that evaluates to an empty string
Fixed a bug and a regression that could result in incorrect comparison between ipv4 addresses and ipv6 subnets and vice versa
Fixed an issue that could result in missing exe_upper_layer flag
Fixed kernel module build for Linux 6.10
Fixed a bug that may result in kernel module crashes on recent versions of RHEL 9
Added additional logging to better troubleshoot hard to reproduce issues like "could not parse param ... for event ... of type ...: expected length X, found Y"

This patch also introduces a small change with the format of the new experimental Prometheus metrics to make them easier to use. Metrics are now distinguished by the file_name or rule_name labels, in line with Prometheus best practices and supporting groupBy queries.

Thanks to everyone in the community for helping us with spotting these annoying bugs and improving Falco every day 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.38.2, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Enjoy 😎,

Luca

Blog: Deploy Falco on a Talos cluster

Mon, 22 Jul 2024 00:00:00 +0000

Talos Linux is an OS designed for Kubernetes, with in mind to be secure, immutable and minimal. It offers a solution for having secure nodes for your Kubernetes cluster. Running Falco on them requires some configurations we'll see in this blog post. The good news is everything is available to collect the syscalls with eBPF and also the audit logs from the Kubernetes control plane.

In this tutorial we'll use a local Talos cluster created with Docker containers for convenience, adapt the configurations to your own context.

Requirements

For this tutorial, you'll need several tools installed:

Set up the Talos cluster

We'll start with a 2 workers cluster:

talosctl cluster create --workers 2 --wait-timeout 5m

After a few minutes, your containers and so your cluster should be up and running. You can check the status with:

talosctl cluster show --nodes 10.5.0.2,10.5.0.3,10.5.0.4

Output:

PROVISIONER docker
NAME talos-default
NETWORK NAME talos-default
NETWORK CIDR 10.5.0.0/24
NETWORK GATEWAY
NETWORK MTU 1500

NODES:

NAME TYPE IP CPU RAM DISK
talos-default-controlplane-1 controlplane 10.5.0.2 - - -
talos-default-worker-1 worker 10.5.0.3 - - -
talos-default-worker-2 worker 10.5.0.4 - - -

Get the kubeconfig

The talosctl CLI allows to easily set up your kubeconfig file for managing the apps in your fresh new cluster:

talosctl kubeconfig -n 10.5.0.2 -f

Check you have access to the cluster:

kubectl cluster-info

Output:

Kubernetes control plane is running at https://10.5.0.2:6443
CoreDNS is running at https://10.5.0.2:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Patch the cluster

When you deploy Falco with Helm in a Kubernetes cluster, an initContainer is bootstrapped to inject the eBPF probe into the kernel of each node. This behavior requires some privileges but Talos, designed to be secured, doesn't allow that by default. It's possible anyway by patching the nodes.

Create this patch.yaml file:

cluster:
 apiServer:
 admissionControl:
 - name: PodSecurity
 configuration:
 exemptions:
 namespaces:
 - falco

As you can see, we allow the pods in the namespace falco to use PodSecurity settings.

And now patch the cluster:

talosctl patch machineconfig --patch @patch.yaml --nodes 10.5.0.2

Output:

patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.2
Applied configuration without a reboot

Install Falco

We'll use Helm to deploy Falco.

Install the Helm registry for the Falco chart:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Create the values.yaml file:

cat << EOF > values.yaml
driver:
 kind: modern_ebpf
tty: true

falcosidekick:
 enabled: true
 replicaCount: 1
 webui:
 enabled: true
 replicaCount: 1
 redis:
 storageEnabled: false
 service:
 type: NodePort
 port: 2802
 targetPort: 2802
 nodePort: 30128

falcoctl:
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 artifact:
 install:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]
 follow:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]

services:
 - name: k8saudit-webhook
 type: ClusterIP
 ports:
 - port: 9765
 targetPort: 9765
 protocol: TCP
 name: http

falco:
 rules_files:
 - /etc/falco/falco_rules.yaml
 - /etc/falco/falco-incubating_rules.yaml
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit
 library_path: libk8saudit.so
 init_config: ""
 open_params: "http://:9765/k8s-audit"
 - name: json
 library_path: libjson.so
 init_config: ""
 load_plugins: [k8saudit, json]
EOF

Deploy Falco:

helm upgrade -i falco falcosecurity/falco -n falco --create-namespace -f values.yaml

Follow the deployment:

kubectl get pods -w -n falco

Before moving on, let's take time to explain why some of these values.

 driver:
 kind: modern_ebpf
 tty: true

We use the modern_epbf probe to collec the syscall events.
tty: true allows to get the alerts in the stdout immediatly, without any buffering.

 falcosidekick:
 enabled: true
 replicaCount: 1
 webui:
 enabled: true
 replicaCount: 1
 redis:
 storageEnabled: false
 service:
 type: NodePort
 port: 2802
 targetPort: 2802
 nodePort: 30128

We install Falcosidekick and its UI. All settings for the forwarding of the events between Falco and Falcosidekick are managed by the Helm chart.
As it's local cluster, we set the replicaCounts to 1, it loses the HA but save resources.
The UI will be exposed directly by the nodes on the port 30128, very convenient for a local cluster, prefer an ingress or just a port-forward for production.

 falcoctl:
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 artifact:
 install:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]
 follow:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]

falcoctl controls which plugins and rules to install and follow.
We install the stable and incubating rules for Falco
We install and follow the rules for the Kubernetes audit logs, the relevant plugins k8saudit and json will be automatically installed by falcoctl.

 services:
 - name: k8saudit-webhook
 type: ClusterIP
 ports:
 - port: 9765
 targetPort: 9765
 protocol: TCP
 name: http

The k8saudit plugin requires to create a Service listen the incoming events from the control plane.

 falco:
 rules_files:
 - /etc/falco/falco_rules.yaml
 - /etc/falco/falco-incubating_rules.yaml
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit
 library_path: libk8saudit.so
 init_config: ""
 open_params: "http://:9765/k8s-audit"
 - name: json
 library_path: libjson.so
 init_config: ""
 load_plugins: [k8saudit, json]

We load the rules for the syscalls and for the audit logs.
We load the plugins and their config. The k8saudit plugin will listen on the same port than configured in the services section.

Forward the audit logs to Falco

In a classic context, the control plane is configured to send its audit logs to an endpoint, like the k8saudit plugin. With Talos, it's not yet possible to configure this endpoint, but fortunately for us, these audit logs are written as files in the /var/log/audit/kube/ folder of the master nodes.

We'll use Fluent Bit to parse these files and forward them to the k8saudit plugin.

Install the Helm registry for the Fluent Bit chart:

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

Create the values.yaml file:

cat << EOF > values.yaml
podAnnotations:
 fluentbit.io/exclude: 'true'

daemonSetVolumes:
 - name: varlog
 hostPath:
 path: /var/log

daemonSetVolumeMounts:
 - name: varlog
 mountPath: /var/log

tolerations:
 - operator: Exists
 effect: NoSchedule

nodeSelector:
 node-role.kubernetes.io/control-plane: ""

config:
 service: |
 [SERVICE]
 Flush 5
 Daemon Off
 Log_Level warn
 HTTP_Server On
 HTTP_Listen 0.0.0.0
 HTTP_Port 2020
 Health_Check On
 Parsers_File /fluent-bit/etc/parsers.conf
 Parsers_File /fluent-bit/etc/conf/custom_parsers.conf

 inputs: |
 [INPUT]
 Name tail
 Alias audit
 Path /var/log/audit/kube/*.log
 Parser audit
 Tag audit.*
 Ignore_older true

 customParsers: |
 [PARSER]
 Name audit
 Format json
 Time_Key requestReceivedTimestamp
 Time_Format %Y-%m-%dT%H:%M:%S.%L%z

 outputs: |
 [OUTPUT]
 Name http
 Alias http
 Match *
 Host falco-k8saudit-webhook.falco.svc.cluster.local
 Port 9765
 URI /k8s-audit
 Format json
EOF

Deploy Fluent Bit:

helm upgrade -i fluent-bit fluent/fluent-bit -n kube-system -f values.yaml

To be allowed to mount the folder with the logs, we install Fluent Bit in the namespace kube-system.

Follow the deployment:

kubectl get pods -n kube-system -w -l app.kubernetes.io/name=fluent-bit

Some explanations of the values.yaml.

daemonSetVolumes:
 - name: varlog
 hostPath:
 path: /var/log

daemonSetVolumeMounts:
 - name: varlog
 mountPath: /var/log

The host folder with the logs is mounted inside the Fluent Bit pod.

tolerations:
 - operator: Exists
 effect: NoSchedule

nodeSelector:
 node-role.kubernetes.io/control-plane: ""

These settings are there to deploy Fluent Bit on the master nodes only.

config:
 inputs: |
 [INPUT]
 Name tail
 Alias audit
 Path /var/log/audit/kube/*.log
 Parser audit
 Tag audit.*
 Ignore_older true

Fluent Bit will parse the files *.logs from the folder /var/log/audit/kube/.

config:
 outputs: |
 [OUTPUT]
 Name http
 Alias http
 Match *
 Host falco-k8saudit-webhook.falco.svc.cluster.local
 Port 9765
 URI /k8s-audit
 Format json

The logs are forwarded to the endpoint falco-k8saudit-webhook.falco.svc.cluster.local:9765/k8s-audit, which is listened by the k8saudit plugin.

Visalize the alerts

Everything should be set up and running from now. You can access to the Falcosidekick-UI by the URL http://10.5.0.2:30128.

The default credentials are admin/admin.

Conclusion

Talos Linux is a more and more famous solution for creating resilient and secure Kubernetes clusters, but the trust doesn't exclude control. Mixing Talos and Falco makes you gain a step upper in term of security for your applications. Thanks to our modern eBPF probe and our k8saudit plugin, you can see how easy and quick it is to install Falco in Talos and start to observe what's happening.

Thanks to Quentin Joly for his blog post about Talos which helped me a lot to write this one.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falcosidekick UI project on GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Introducing Falco 0.38.1

Wed, 19 Jun 2024 00:00:00 +0000

Today we announce the release of Falco 0.38.1 🦅!

Fixes

Falco's 0.38.1 is a patch release aimed at addressing a few important bugs. It includes the following fixes:

A Falco crash while running with plugins and metrics enabled has been solved (https://github.com/falcosecurity/falco/issues/3229)
Falco -p output format option can now be passed to plugin events while -pc and -pk can only be used for syscall sources. Fixes an issue that could result in Falco exiting with LOAD_ERR_COMPILE_OUTPUT on startup with k8s clusters that had -pk and audit enabled (https://github.com/falcosecurity/falco/pull/3239)
Fixed an issue that could prevent the integer compare operators <, <=, >, >= in rules from working properly (https://github.com/falcosecurity/falco/issues/3245)
Ignore NSS user and group entries while loading users and groups (https://github.com/falcosecurity/libs/pull/1909)
Issues related to the new metric-related plugins API (https://github.com/falcosecurity/libs/pull/1885). Plugin API was also bumped to 3.6.0.
Plugin metrics are now enabled in Falco (https://github.com/falcosecurity/falco/pull/3228). Note that plugin must make use of the new metrics-related API to expose metrics.
Libs were updated to 0.17.2

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.38.1, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Introducing Falco 0.38.0

Thu, 30 May 2024 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.38.0! This is the first Falco release since its graduation within the CNCF, and, as usual, brings many improvements and features alongside some pretty big changes in its configuration mechanism.

This release brings an easier to use mechanism to install and configure your drivers, new rule language features, better support for Falco metrics and many more improvements.

During this release cycle, we merged more than 100 PRs on Falco and more than 180 PRs for libs and drivers, version 0.17.0 and version 7.2.0 respectively. Thank you to our maintainers and contributors, as this would not happen without your support and dedication!

To learn all about these changes, read on!

What’s new? TL;DR

Key features:

New capabilities in falcoctl to automatically select the best driver for your system and make it easier to install
The Falco configuration file can now be split into multiple files to make it more manageable
Rule selection from configuration file or command line
Field transformers and value comparison
Prometheus metrics support
Plugin API improvements

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.38.0 release contains a number of feature and UX improvements, here are list of some of the key new capabilities.

Driver loader magic ✨

If we could pick the most common issue that we've heard from adopters and we experienced first hand is the fact that sometimes we all struggle with installing and upgrading Falco drivers. The Falco team has been tirelessly working for years to improve the installation experience and Linux kernel compatibility with massive changes such as the introduction of the new CO-RE eBPF probe and most recently the complete rewrite of our driver loading component, integrated in falcoctl. With this new version of falcoctl, integrated in Falco 0.38.0, our loading tool will automatically detect your system and pick the most compatible driver without any intervention; on recent kernel versions this is likely the modern eBPF probe. As you probably know, the modern probe does not require any extra driver download or compilation, making it load almost instantly. Of course, the tool also allows to select the preferred driver if the automatic choice is not optimal for your use case. On top of that, our driver loader tool can now automatically download kernel headers for many distributions supported by driverkit so in many cases you will be able to install even the kernel module without having to install kernel headers first. Read more about how to configure this functionality in the installation documentation page.

Organize your Falco configuration files 🗃️

Our falco.yaml configuration file gains more options, fine tuning configuration flags and feature selection for every release; in fact, they are so many that some people would like to better organize them in separate configuration files which can also be kept across Falco upgrades. Starting from this release you can add list of files or directory to the config_files configuration entry, which comes populated with the /etc/falco/config.d/ directory by default. Any additional file is read in order and can override settings in falco.yaml. Read more in the configuration options section of the documentation.

Choose which rules to load at runtime 📝

We distribute several files that contain community contributed rules and you can always write your own. But how do you select which rules Falco will load at runtime? There are several ways, including using overrides or specifying command line options such as -D, -t and -T. However, those do not allow you to express something as simple as "I would like to exclude all rules except for this one" or "I would like to include a specific tag and disable some of its rules". Furthermore, you couldn't specify this configuration in your falco.yaml file. To make this possible, we introduced a new configuration option, rules, that can be specified both in the configuration file or the command line. For instance, you can now write:

rules:
 - disable:
 rule: "*"
 - enable:
 rule: Netcat Remote Code Execution in Container
 - enable:
 rule: Delete or rename shell history

To finely control your rule loading without modifying the rule files themselves. Read more in controlling rules.

Field transformers and value comparison in conditions

Up until now we couldn't write a condition that catches operations like "a process deleting its own executable" because you couldn't use a field value on the right hand side of the condition. Since this version we have added a syntax to do just that with the val() operator:

evt.type = unlink and proc.exepath = val(fs.path.name)

will trigger only if the process exepath is the same as the unlink argument target, meaning that the process is trying to delete its own executable!

In addition you can also apply simple transform operators to both sides of the comparison: toupper() and tolower() will convert casing as you'd expect and b64() can decode base64. Stay tuned for additional transformers to cover more use cases! Read more on transform operators in the documentation.

Prometheus Metrics support 🔥

If you have been following Falco development, you probably know we are constantly improving support for metrics that tell you how the Falco engine is doing. We now have introduced Prometheus support so you can better integrate Falco with your existing performance monitoring infrastructure, and paves the way for the community to create an official Grafana dashboard that can be integrated in our charts.

Plugin API improvements ⚙️

Plugins are getting more powerful at each version. We now have a set of experimental APIs to expose metrics and read more into the Falco internal state that our expert plugin authors have been asking about. Stay tuned for more in-depth documentation on those!

Breaking changes and deprecations ⚠️

This version comes with breaking changes, mostly in the configuration interface

Changed configuration options

The syscall_buf_size_preset Falco configuration option has been replaced by engine.<driver>.buf_size_preset (e.g. engine.kmod.buf_size_preset)
The syscall_drop_failed_exit Falco configuration option has been replaced by engine.<driver>.drop_failed_exit (e.g. engine.kmod.drop_failed_exit)
The modern_bpf.cpus_for_each_syscall_buffer Falco configuration option has been replaced by engine.modern_ebpf.cpus_for_each_buffer
The syscall_event_drops Falco configuration option has been replaced by the metrics config plus some automatic notification on drops.

Removed command line options and equivalent configuration options

The --modern_ebpf command line option has been replaced by engine.kind: modern_ebpf in falco.yaml (or, on the command line -o engine.kind=modern_ebpf). Likewise, --nodriver is now engine.kind: nodriver.

The environment variable FALCO_BPF_PROBE is replaced by engine.ebpf.probe configuration option. Example:

engine:
 kind: ebpf
 ebpf:
 # path to the elf file to load.
 probe: ${HOME}/.falco/falco-bpf.o

The -e option to load capture files is no longer available. In order to read a capture file use the configuration option engine.replay.capture_file. Since options can be specified on both the command line and the configuration file, an equivalent command line as falco -e <file.scap> is falco -o engine.kind=replay -o engine.replay.capture_file=<file.scap>

The gVisor command line options have been replaced by equivalent configuration options. -g/--gvisor-config is now engine.gvisor.config while --gvisor-root is now engine.gvisor.root. Example falco.yaml configuration file:

engine:
 kind: gvisor
 gvisor:
 # A Falco-compatible configuration file can be generated with
 # '--gvisor-generate-config' and utilized for both runsc and Falco.
 config: "/etc/docker/runsc_falco_config.json"
 # Set gVisor root directory for storage of container state when used
 # in conjunction with 'gvisor.config'. The 'gvisor.root' to be passed
 # is the one usually passed to 'runsc --root' flag.
 root: "/var/run/docker/runtime-runc/moby"

Or, equivalent writing on the command line:

falco -o engine.kind=gvisor -o engine.gvisor.config=/etc/docker/runsc_falco_config.json -o engine.gvisor.root=/var/run/docker/runtime-runc/moby

You can find more information on breaking changes in the tracking issue.

Deprecations

In Falco 0.39.0 we will remove the -D, -t, -T options, continuing our tradition of removing single-character options that nobody remembers what they do.

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

Falco is more mature with each release. Following its graduation we have published the roadmap for version 1.0.0 which is guiding us in the next steps. As you can see, this version is addressing some of the roadmap points with our changes to configuration and CLI options and adding rule constructs and drivers. For the next release, you can expect more stability, streamlined container images, refinements to our rule syntax, new detections and more.

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Track the Bitcoin transactions with Falco

Wed, 13 Mar 2024 00:00:00 +0000

The number of plugins available for Falco continues to grow thanks to our wonderful community. Thank you all for your help!

You can find the list of available plugins here.

The vast majority of plugins developed allow Falco to ingest logs from different sources and raise alerts when suspicious elements are identified by its rules. In order to show that any event stream can be a source if you have the right plugin, and to have something fun to show users during my talks, I developed a Falco plugin to track Bitcoin transactions.

How does it work?

I discovered the site https://www.blockchain.com/ exposes a public flux, accessible via a websocket, by subscribing to it you can retrieve transactions carried out on the blockchain in real time. This is perfect for a Falco plugin as it allows you to test the ingestion of events via a websocket, and serve as a basis for other plugins.

I am not going to describe the internal workings of the plugin here, nor how it was developed. If you are interested, you can look at the code here.

Alternatively, read our documentation explaining how to create a plugin from A to Z: https://falco.org/docs/concepts/plugins/developers-guide/how-to-develop/.

Default rules

The plugin comes with its default set of rules, we will use them as a working example. You are free to play with it for your own needs, such as monitoring suspicious movements of your wallet.

You can find the Falco rules file provided here.

Installation of the plugin

We will see the 3 classic ways to install the plugin:

via sources
with falcoctl
in kubernetes via Helm

Via sources

The prerequisites are:

Golang >= 1.19
make
Falco >= 0.36
Git

We will start by installing download the sources, build and install the plugin:

git clone https://github.com/Issif/bitcoin-plugin.git
cd bitcoin-plugin
sudo make install

We will create a falco.yaml file containing:

plugins:
 - name: bitcoin
 library_path: /usr/share/falco/plugins/libbitcoin.so
 init_config: ''
 open_params: ''

load_plugins: [bitcoin]

stdout_output:
 enabled: true

The plugin comes with a default set of rules which will be sufficient for testing. All that remains is to start Falco with this command:

sudo falco -c falco.yaml -r rules/bitcoin_rules.yaml

14:44:21.721357000: Notice The wallet bc1qfpeps3wcmzk422hvm5jeq5lelnqlzznjwyfy69 sent 96.78318104 BTC to (bc1q4hwcl377ereljtyn2t7ljdrh9umyxz5uuyl3qn,bc1qfpeps3wcmzk422hvm5jeq5lelnqlzznjwyfy69) in the transaction aab62fd0b529cd9da163508ba879d488ff64cce4c130caf6c8bd21ab1701ed46
14:44:27.020379000: Notice The wallet bc1qwk9hqnckv0ryhsnsdefcsmlpn3zx7uq3agdsw9 sent 68.68462728 BTC to (bc1qg0nkd5nckxvwlslf6lznukgat2vukrnrrcwjcv) in the transaction 734526413f6e3eefdf4adc4258e01375ccc145b9d02b7e0ab45517be0f57e7d9
14:44:29.393013000: Notice The wallet bc1qm34lsc65zpw79lxes69zkqmk6ee3ewf0j77s3h sent 14.94446421 BTC to (3F9e4JvPryCxC5A6TS4VHeT2EJSK2ivjBV,bc1qm34lsc65zpw79lxes69zkqmk6ee3ewf0j77s3h,bc1qaeq3z2edsuspt82qw7uflg0k860clxs7qjhrh0,bc1qnvhjvpa6gaglrg9lxg7w68ye8jdjcj2nk08y20,bc1q22hp7n28whk5h94z93vm05hfx2zxs8ca9gglk7,bc1qe9yxu2myqvt3kegknzj45u704dhtapwy7lxhnv,bc1qye5rp8pcqt4ej3nsz70c3lngacmew2fc4tfljd,bc1qcfqke8as8y08mkclcun9r3hlq4xl5za2vz3n2p,bc1qtqkjq4wq234netyucg247sm6nge9qu7m2fd28g,12Q4AHgzFmKWmY1Z2LEohMoxLVhvCAKsNV,3EyJiePQX4BUt8XXaAG3JmfhwB7cQ8ggp6,bc1q36ary7yaf2eeg6006h4m33drsgw4xa3pu6yvnn,13ybpB8kTgk8bCsnRrpyemNZdE2PJSHMEs,135dx8ncZzWSjhre8ecGG1yenmLwvNZPz4,3Nzr6LAJXstT8ET2CAGMH6h5vfgrh7Q94g,bc1qjhrhwpyc0z8zh6v22vhf5arzf6vcr47tgtkj5a,bc1qpxlsyrcmwuf2rk52emvfe0dvugphzzkxlyzvxv,bc1qf23j9ls2axtl6shpry40l4qat5c695x40vpfm8,bc1qsxsdunam68jkeuu7c3mplza4h74nrjhhu9w7dl,32e54ctKqWXfzKpdNKcCBBdsRoFHKoLijH,bc1qw2gafqcg2267xm2t0r4gfzu7ff392e2vl6s3zc,bc1qc0dwh27y56yajhz0k039j5p7xkwfjprhz7rfkq,3D493LGN6PchbRPtnJQo6dSUTLB8u5vN3i,3DhzjabzhAXTBU9vksNdBZFhZzMYzK7vix,18ex2LKyiLpjaSQStY1CLNbLbSToRkJAy6,bc1q3jvuvkvpukp0mnksfmpvnqq in the transaction 40c33db54610869c75b101431690e73b584b8cc77802eea76fa2d41bbb615852
14:44:29.395043000: Notice The wallet 3Hi5VHVgmYZYfAPc9aNvQoNXyEv5rYvJQN sent 50.00000000 BTC to (bc1q582qfqtlvfv038jf6k6s6xvd30we7x66katshx,bc1q6j3gxn68m5pkzhtytn3h464kgjnvce79x8nmwq,bc1qmxaaz6g07re55ekmtlmtrc5kj0kpj3lngy5y60,1CPjdsfkqiW6LB2ZNTDYczjKCzPpiJZ4Ci,1JtUKazSgYN6hCM7HPkvzL7JLVXwkL4stN,3GzfFtGVte95ZMFfQsrz3FFgFDHU8Zw6gS,bc1qcyl4sxkczex6gxldrfmfdctr2qsun4cgpufz8j,bc1q0realpv9h4zp3yhdwjeg78njqg97f9sm6ex3xrw8mkrz8g6qamsqua6tcw) in the transaction 3025c4566dc6cd6452c0c9ae6dc8cff9583df4530326b29e38e0a5e763a6c1c9
14:44:32.577196000: Notice The wallet bc1qfpeps3wcmzk422hvm5jeq5lelnqlzznjwyfy69 sent 96.43310490 BTC to (bc1qzrzhnlaru0pqmcxwm80vvvsqpdll9g6t39y686,bc1qfpeps3wcmzk422hvm5jeq5lelnqlzznjwyfy69) in the transaction 1083e02c554454db4dcff02f7418198aae5b563c4ec286b4c3ae4d30e649e8d5
14:44:32.577917000: Notice The wallet bc1qvruk6nhq5rz7whvx9cz6peqrp3nrutae59d63q sent 13.48137244 BTC to (1EtV3erwXxeKLhCvXq1BwKit7pMcB5BDvV,bc1qxgepulgdkjju7s8el6932m57svej5uzfvx7207) in the transaction 3e000a5745d7d5b6d2791bff75b9045696c2bea497363e845593ac249cc194b5

We can clearly see transactions (sending and receiving) for amounts exceeding 1 BTC appearing in real time.

With falcoctl

The prerequisites are:

Falco >= 0.36
Falcoctl >= 0.6
Git

Falcoctl is the CLI tool that we developed to facilitate the installation of artifacts around Falco, such as rules and plugins. To find out more, here is a blog article about it.

sudo falcoctl index add bitcoin https://raw.githubusercontent.com/Issif/bitcoin-plugin/main/index.yaml
sudo falcoctl artifact install bitcoin-rules:latest

Both the plugin and the rules will be downloaded thanks to the dependency:

 INFO Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO Resolving dependencies ...
 INFO Installing the following artifacts: [ghcr.io/issif/bitcoin-plugin/ruleset/bitcoin:latest bitcoin:0.2.0]
 INFO Preparing to pull "ghcr.io/issif/bitcoin-plugin/ruleset/bitcoin:latest"
 INFO Pulling 8758e31efdff: ############################################# 100%
 INFO Pulling 326b3ec82baf: ############################################# 100%
 INFO Pulling 8aec149e9934: ############################################# 100%
 INFO Artifact successfully installed in "/etc/falco"
 INFO Preparing to pull "ghcr.io/issif/bitcoin-plugin/plugin/bitcoin:0.2.0"
 INFO Pulling e7f990e1e4e6: ############################################# 100%
 INFO Pulling 0dfca1bb2434: ############################################# 100%
 INFO Pulling f269eb62cbf6: ############################################# 100%
 INFO Artifact successfully installed in "/usr/share/falco/plugins"

As with the installation via sources, the falco.org file should look like:

plugins:
 - name: bitcoin
 library_path: /usr/share/falco/plugins/libbitcoin.so
 init_config: ''
 open_params: ''

load_plugins: [bitcoin]

stdout_output:
 enabled: true

And Falco will be started by the command:

sudo falco -c falco.yaml -r /etc/falco/bitcoin_rules.yaml

In Kubernetes via Helm

The prerequisites are:

Helm

The installation will consist of just adapting the values in the values.yaml file. Everything will be automatically managed by the templates:

tty: true
kubernetes: false

falco:
 rules_files:
 - /etc/falco/bitcoin_rules.yaml
 plugins:
 - name: bitcoin
 library_path: libbitcoin.so
 load_plugins: [bitcoin]

falcosidekick:
 enabled: true
 webui:
 enabled: true

driver:
 enabled: false
collectors:
 enabled: false

controller:
 kind: deployment
 deployment:
 replicas: 1

falcoctl:
 config:
 indexes:
 - name: bitcoin
 url: https://raw.githubusercontent.com/Issif/bitcoin-plugin/main/index.yaml
 artifact:
 install:
 refs: ["bitcoin:0"]
 follow:
 refs: ["bitcoin-rules:0"]

And the classic Helm command for installation:

helm install falco-bitcoin -n falco falcosecurity/falco -f values.yaml --create-namespace

After a few seconds, you should have the pod running:

❯ kubectl get pods -n falco -l app.kubernetes.io/instance=falco-bitcoin
NAME READY STATUS RESTARTS AGE
falco-bitcoin-7474fbfcb5-srgsg 2/2 Running 110 (17m ago) 10d

And new events in falcosidekick-ui:

Conclusion

This plugin has no great purpose other than to dismantle the almost infinite possibilities that open up to Falco thanks to its plugin system. If you wish to be alerted on Telegram of a strange outgoing movement from your wallet, it is now possible with Falco!

Falco is no longer limited to securing Cloud environments. SaaS or others can also be used in a unified way. The Falco rules syntax has proven to benefit security practitioners in an ecosystem rich with numerous potential integration points.

Blog: Falco Weekly 10 - 2024

Fri, 08 Mar 2024 00:00:00 +0000

What happened in Falco this week?

First of all, you probably already heard it, Falco is now graduated!
If you missed this important news, go ahead and give our graduation blog post a read!

Let's go through the major changes that happened in various repositories under the falcosecurity organization during the last week.

Libs

We are approaching the 0.15.0 tag, therefore mostly bugfixes were merged, plus a great new feature and some refactors:

The so-called "kmod configure system" was finally merged: https://github.com/falcosecurity/libs/pull/1452. This helps us to ensure that our kernel module builds even when some features get backported from more recent kernels (ie: when checking for kernel release version in the code is not enough). Kudos to Angelo Puglisi for shipping such a feature! Also, keep an eye for the very same thing for bpf too: https://github.com/falcosecurity/libs/pull/1729! Thanks to the kmod configure system, our kernel-testing matrix is now fully green for kmod!
A big CRI API refactor finally landed: https://github.com/falcosecurity/libs/pull/1600
Proceeding with the journey around compiler sanitizers, we now have proper cmake options to enable ASAN and UBSAN: https://github.com/falcosecurity/libs/pull/1721
Fixed a crash when reading proclist from scap: https://github.com/falcosecurity/libs/pull/1726
Fixed some socketpair fds problems: https://github.com/falcosecurity/libs/pull/1733
Fixed and added some more tests: https://github.com/falcosecurity/libs/pull/1736, https://github.com/falcosecurity/libs/pull/1727

Falco

Some small changes happened too, in Falco main repository:

The proposal about features adoption and deprecation was merged: https://github.com/falcosecurity/falco/pull/2986!
Added a new configuration key falco_libs.thread_table_size to customize max thread table size in libsinsp: https://github.com/falcosecurity/falco/pull/3071
Throw an error when an invalid macro/list name is used: https://github.com/falcosecurity/falco/pull/3116
Fixed up directory iteration options while iterating over rule folder: https://github.com/falcosecurity/falco/pull/3127

Join relevant discussions

Our new discussion section: https://github.com/falcosecurity/falco/discussions
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840
Breaking changes in Falco 0.39.0: https://github.com/falcosecurity/falco/issues/3045

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list
Open a discussion in our discussion section

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Falco Graduates within the CNCF!

Thu, 29 Feb 2024 00:00:00 +0000

Today, the Falco project hit a big milestone: becoming a CNCF Graduated Project! Falco's graduation indicates the project's maturity and dependability, but most importantly, it is the culmination of a fantastic amount of work.

The journey for Falco started in 2016 when the first commit was made. Today, Falco has become synonymous with "runtime security" due to its comprehensive approach to securing the highly complex and dynamic environments of the modern cloud era.

“Falco approaches the security challenges associated with managing cloud native environments holistically,” says Loris Degioanni, the project’s founder. ”Runtime security is more than eBPF-based data collection, it requires enrichment, orchestrator integration, correlation of different data sources, and a rich, well-maintained policy library. All of the things that Falco provides to its users.”

This holistic, runtime-centric approach to security is what makes Falco unique. It enables any organization to secure their infrastructure — from scrappy startups to Fortune 500s. Since joining the CNCF, some of the largest enterprises in the world, including Amazon, Apple, IBM, and Red Hat, have contributed to Falco. The project also has a strong, rapidly growing community of adopters and has been downloaded over 100 million times!

To the thousands of people who have helped Falco fly over the past years, thank you — sincerely. Thank you to the Falco maintainers. Thank you to our CNCF Technical Oversight Committee sponsors, Emily Fox and Justin Cormack. Thank you to anyone who has ever raised an issue, submitted a pull request on GitHub, or just took part in our community.

Thank you, also, to the thousands of organizations who have entrusted Falco with the security of your runtime environments.

For us, Falco’s Graduation represents a calling to continue to improve the project in a way that serves its users. We believe that runtime protection is vital to security, and that Falco is well positioned to power that security as we move deeper into the cloud era.

In the future, Falco will have even stronger detections, richer signals, lower noise, and better performance. Its breadth of coverage will increase with more data sources, including cloud logs and key developer touchpoints like GitHub. Future versions of Falco will be even easier to deploy and manage in production.

Our mission is to make Falco a powerful companion that brings you peace of mind, knowing your cloud native apps are well looked after. We hope you will continue to be part of this journey with us.

Words from our community

Graduation means a lot to the members of our community. We asked them to share their thoughts and feelings with us on this occasion. Here is what they had to say:

"Since joining the Falco project in 2020, I've been inspired by our community's growth and commitment to open source values. Falco has unequivocally established itself as the quintessential tool for cloud native runtime security, leveraging key technologies like eBPF – notably, becoming one of the largest open source eBPF codebases. As we reach the CNCF graduation milestone, I'm immensely proud of our collective achievements and want to thank every contributor who has played a role in this journey." - Leonardo Grasso, Falco core maintainer

"Linux kernel security monitoring is undeniably mission-critical, in great demand, yet daunting to master. Working within the kernel can be intimidating due to its potential impact on application performance and the sheer volume of events on modern servers. Since joining in 2022, The Falco Project has adapted to meet new demands while staying true to its mission, and this journey continues to accelerate. Observing Falco's effectiveness and value in real-world production settings is truly beautiful." - Melissa Kilby, Falco core maintainer

"Back when I joined the Falco community, the project was a teenager that needed some love; here we are, it took a little while but it is now an adult! And I loved every little bit of its growth! What do I love more? The fact that we still have a lot of space for improvements, everywhere. This is good for users, the wider community and for us, developers and maintainers of the projects, to keep the fun with it." - Federico Di Pierro, Falco core maintainer

"It has been a long journey since my first try of Falco. It was its very first release, Kubernetes was a small thing and the containers started to become the game changer we know now. I'm very proud having been a modest piece of this achievement, developing tools for Falco made me a better DevOps, a better Go developer, an international speaker and it made me meet amazing users and contributors. It's a good thing to see it’s rising as a standard for the runtime security in the industry. I hope it will help even more SREs to peacefully sleep at night." - Thomas Labarussias, Falco core maintainer

"The project's rapid growth is evident with each new milestone, and I take great pride in being a member of this team, of this family. There's boundless potential for the project's expansion, and I see this milestone as the first step toward an even brighter future." - Andrea Terzolo, Falco core maintainer

"I have been following Falco for many years now and I am impressed by how far the project has come. I am personally proud and happy that I could be a part of the stellar team that drives Falco; thanks to maintainer's and contributor's efforts we were able to achieve incredible goals and it is great to see the project being recognized alongside the most successful in the CNCF. Thanks to everyone who has been with us in this journey and everyone who will join us in the future." - Luca Guerra, Falco core maintainer

"It's hard to believe that more than two years have already passed since I joined the amazing open source community of Falco. I can't express my deep gratitude for all our supporters and for the Falco family. This project gave me the privilege of connecting with incredibly skilled humans, and of witnessing the growth of a beautiful piece of technology that's now a fundamental security asset for countless organizations in the industry. Looking back, I feel immensely proud of all the collective efforts that led the project to this huge milestone, and I can't wait to see what the future holds from this point forward." - Jason Dellaluce, Falco core maintainer

"Today, I am immensely proud as Falco graduates within the CNCF. This achievement is a testament to the dedication of our maintainers and the broader community of adopters and contributors. Together, we’ve propelled Falco to become the industry’s de-facto standard for runtime threat detection in the cloud. I am deeply grateful for the collaborative efforts that have brought us to this moment. The tireless work, expertise, and passion shared by our team and community have elevated Falco’s capabilities, ensuring its effectiveness in safeguarding organizations against evolving security threats. As we enter this new chapter within the CNCF, I am confident in our collective ability to continue innovating and strengthening Falco’s position as a vital tool in the fight for cloud security." - Michele Zuccala, Falco core maintainer

"Since I joined, contributing to the ecosystem with passion and excitement to know more about and to contribute to this project, I've found a welcoming, vibrant, and healthy community as well as a strong maintainership. With time, the community grew and grew, and independence and diversity increased during these years. Nowadays, runtime security in the cloud native world is becoming more and more fundamental in our architectures, and Falco has become one of the de-facto standards for increasing observability in our Linux kernel-based cloud native systems. The project has evolved a lot during the last couple of years. I remember the design proposal for plugins back then! Thanks to the incredible work of the maintainers and the contributors, and now I can't explain how much this big step and acknowledgment from the CNCF matter! Congratulations to the Falco family!" - Massimiliano Giovagnoli, Falco core maintainer

Blog: Falco Weekly 7 - 2024

Fri, 16 Feb 2024 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

Multiple fixes and some cleanups happened in the libs repo:

newfstatat syscall is now configured with UF_ALWAYS_DROP: https://github.com/falcosecurity/libs/pull/1683
Fixed null destination address in sendto and sendmsg in modern bpf: https://github.com/falcosecurity/libs/pull/1687
Added a CT_UNKNOWN container type zero value and properly initialize uninitialized value: https://github.com/falcosecurity/libs/pull/1688
Fix in chisels: don't fail if a chisel directory does not exist: https://github.com/falcosecurity/libs/pull/1689
Cleaned up more memory reads/writes in filterchecks to avoid UBs: https://github.com/falcosecurity/libs/pull/1690
Properly initialize m_exe... fields in threadinfo: https://github.com/falcosecurity/libs/pull/1691
Fixed a small source of memleak in scap platform: https://github.com/falcosecurity/libs/pull/1692
Properly enforce the static CRT on Windows by default: https://github.com/falcosecurity/libs/pull/1695

Falco

Falco has seens quite a bit of C++ improvements, thanks to Samuel Gaist! Keep up the great job!

C++ cleanups: https://github.com/falcosecurity/falco/pull/3069, https://github.com/falcosecurity/falco/pull/3074, https://github.com/falcosecurity/falco/pull/3083, https://github.com/falcosecurity/falco/pull/3085
Consolidated Faclo engine and rule loader tests: https://github.com/falcosecurity/falco/pull/3066
Added http-headers option to Falco driver-loader images: https://github.com/falcosecurity/falco/pull/3075
Cleaned up an unused builder Dockerfile: https://github.com/falcosecurity/falco/pull/3088
Fixed some compiler warnings: https://github.com/falcosecurity/falco/pull/3089
Cleaned up falco_engine deps and include paths: https://github.com/falcosecurity/falco/pull/3090

Falcoctl

Falcoctl has seen a small yet important fix:

Correctly report artifact type: https://github.com/falcosecurity/falcoctl/pull/442

Kernel-testing

Even if the effort was part of last week, and since we skipped last "Weekly Recap", it is important to mention that the kernel-testing framework recently got a big update:

All images build is now tested in PR CI when they are modified
Images are now build and published on ghcr.io/falcosecurity/kernel-testing repo
They are published under main tag and under latest|$tag for releases
The image name is built as: $distro-{kernel,image}:$kernelrelease-$arch-$imagetag, eg: amazonlinux2-kernel:5.10-x86_64-v0.3.2
Ubuntu-6.3 images were bumped to 6.5 kernel
A new arch-6.7 image was added to the test matrix
A composite action was added and is now used by libs CI

As always, you can find detailed kernel-testing outputs against our drivers under https://falcosecurity.github.io/libs/matrix/.

Charts

Thanks to Aldo's continuous effort, we now have much better documentation all around the repo:

Updated docs for Falco exporter: https://github.com/falcosecurity/charts/pull/623,
Process all charts for changes in values.yaml: https://github.com/falcosecurity/charts/pull/624
Updated contributing section: https://github.com/falcosecurity/charts/pull/625
Fixed typos, formatting and dead links in Falco chart docs: https://github.com/falcosecurity/charts/pull/627
Fixed dead links for Falco exporter: https://github.com/falcosecurity/charts/pull/628
Fixed link tags in readme: https://github.com/falcosecurity/charts/pull/629

Join relevant discussions

Our new discussion section: https://github.com/falcosecurity/falco/discussions
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840
Breaking changes in Falco 0.39.0: https://github.com/falcosecurity/falco/issues/3045

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list
Open a discussion in our discussion section

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Introducing Falco 0.37.1

Tue, 13 Feb 2024 00:00:00 +0000

Today we announce the release of Falco 0.37.1 🦅!

Fixes

Falco's 0.37.1 release is a small patch aimed at addressing a few minor bugs. It includes the following:

Added --http-insecure flag to driver loader images
Added new env variable FALCOCTL_DRIVER_HTTP_HEADERS understood by driver loader images to pass a comma separated list of http headers for driver download, eg: FALCOCTL_DRIVER_HTTP_HEADERS='x-emc-namespace: default,Proxy-Authenticate: Basic'
Falcoctl was bumped to v0.7.2, fixing an issue building Flatcar drivers and a bug withing the kernel release fixup method to build drivers download URLs
Fixed a nasty bug that caused Falco to crash when a priority higher than debug was set in the config: https://github.com/falcosecurity/falco/pull/3060
Libs were updated to 0.14.3

Last, but not least, as recommended by the CNCF, we now link libelf dynamically instead of statically, so that the library remains separable from Falco at runtime.
This has multiple outcomes:

Falco static (musl) build is disabled for now; we are experimenting with some solutions and we will hopefully be able to bring it back up soon
Users of docker images won't notice anything since they already shipped libelf library
Users of deb and rpm packages won't notice anything since libelf was already a nested dependency
Users of the tar.gz package will need to manually install libelf where not present

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.37.1, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Introducing Falco 0.37.0

Tue, 30 Jan 2024 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.37.0!

This release brings an improved installation experience, a new way to modify Falco rules, and some great UX improvements. There are, as to be expected, a handful of breaking changes. But, rest assured, we've done all we can to help you with any changes you might need to make.

During this release cycle, we merged more than 100 PRs on Falco and more than 160 PRs for libs and drivers, version 0.14.2 and version 7.0.0 respectively. Thank you to our maintainers and contributors, as this would not happen without your support and dedication!

To learn all about these changes, read on!

What’s new? TL;DR

Key features:

A new way to enrich syscalls with Kubernetes metadata, replacing the old Kubernetes collector.
New capabilities in falcoctl to download and build our kernel drivers, replacing the old falco-driver-loader script.
Support for 32-bit syscall emulation on x86_64 in all kernel drivers (modern_ebpf, ebpf, kernel module).
A new override key to easily modify rules, lists, and macros.

Key UX improvements:

Introduction of a new engine key in falco.yaml to replace all other methods for opening engines such as FALCO_BPF_PROBE, --modern-bpf, -g, and -e.
Expansion of environment variables in falco.yaml even when they are part of a string.

This release also comes with breaking changes, we'd suggest to read them before upgrading. If you use helm, make sure to read the Helm chart breaking changes page as well.

Major features and improvements

The 0.37.0 release contains a number of feature and UX improvements, here are list of some of the key new capabilities.

A new way to enrich syscalls with K8s metadata

Falco 0.37.0 introduces a new method to enrich syscalls with Kubernetes metadata to help address scalability and other issues with the old collector. Falco always had Kubernetes support, but sometimes we need new approaches to keep up with the bigger and bigger scale that we see in production clusters today. You can find more technical details here.

While the collector was previously integrated into Falco, this feature uses a new architecture which leverages a plugin (k8smeta) and a remote collector (k8s-metacollector).

The plugin gathers details about Kubernetes resources from the remote collector. It stores this information and provides access to Falco upon request. The plugin specifically acquires data for the node where the associated Falco instance is deployed, resulting in node-level granularity. In contrast, the collector runs at the cluster level.

Within a given cluster there may be multiple k8smeta plugins (one per node), but only one collector exists per cluster.

More technical details about the architecture and design choices are here.

It’s important to note that both new components are considered experimental, which means although they are functional and tested, they are currently in active development. They may undergo changes in behavior as necessary without prioritizing backward compatibility.

Fields supported by the new `k8smeta` plugin

This section provides details on:

Kubernetes fields that are supported out-of-the-box by Falco through container runtime enrichment.
Fields the new k8smeta plugin supports
Fields have been deprecated.

The following fields are automatically populated with data from the container runtime, making them compatible with Falco without needing the old k8s collector or the new k8smeta plugin. These fields will continue to function as before, and no changes have been made:

k8s.pod.name
k8s.pod.id/k8s.pod.uid
k8s.pod.sandbox_id
k8s.pod.full_sandbox_id
k8s.pod.label
k8s.pod.labels
k8s.pod.ip
k8s.pod.cni.json
k8s.ns.name

All other fields with the k8s.* prefix previously supported by the old collector (e.g., k8s.deployment.name) are now deprecated and will return <NA> if used in rules.

These fields are now provided by the new plugin under the k8smeta.* prefix. A complete list of these fields can be found here

The new fields introduced by the k8smeta plugin are additive. They do not replace the fields provided by the container runtime. This means you can use both k8s.pod.name and k8smeta.pod.name simultaneously. While they may return the same value, the data is collected from different sources (container runtime for k8s fields, the Kubernetes API server for k8smeta). As a result, their availability and reliability may differ during the lifecycle of an application. While it may seem redundant, this approach should offer flexibility to users.

To wrap up:

If k8s.pod.* and k8s.ns.name fields meet your needs, you can use Falco without plugins. The default container runtime information in Falco should be enough.
If k8s.pod.* and k8s.ns.name fields are insufficient, you should evaluate the new k8smeta plugin.
The old k8s.* fields (excluding k8s.pod.* and k8s.ns.name) are now deprecated, and if used in Falco rules, they return <NA>

If you’d like to read more about this new feature check out the documentation for the k8smeta pluginand the k8s-metacollector, while if you want to deploy this solution with our helm chart check out the dedicated section.

New Falcoctl capabilities

Since falcoctl 0.7.0, users have been able to quickly download and compile Falco drivers using the falcoctl driver command. Starting with Falco 0.37.0 the falcoctl driver command will be used by the Falco installation process in place of the old falco-driver-loader script.

For example, to install the kernel module:.

Specify which driver we want to use

falcoctl driver config --type kmod

Install the driver

falcoctl driver install

By default, the falcoctl driver install command tries to download a prebuilt driver from the official Falco download s3 bucket. If a driver is found, then it is inserted into ${HOME}/.falco/. Otherwise, the script tries to compile the driver locally.

You can find more details on installing each driver type in our docs.

Finally, while the falcoctl driver command replaces the old falco-driver-loader script it’s important to note that, even though there is no change in terms of usage, the Docker images falco-driver-loader and falco-driver-loader-legacy no longer utilize the old falco-driver-loader script; instead, they now use falcoctl.

32-bit syscall emulation

The support for 32-bit syscalls has consistently been a highly requested feature for a long time. Until now, this support was only available in the kernel module, but starting from Falco 0.37.0, we have finally extended this support to the ebpf and modern_ebpf drivers. This feature is crucial as it addresses a security gap that has existed for some time.

It’s important to note that this feature is specifically for 32-bits syscalls emulated on the x86_64 architecture. Falco does not support pure 32-bit architectures.

Follow these steps to try out this new feature:

Create a C program ia32.c

#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <sys/types.h>

int main() {
 syscall(__NR_close, -1);
 return 0;
}

Compile it gcc ia32.c -o ia32 -m32
Start Falco with the following rule evt.type = close and proc.name contains ia32
Execute the binary

./ia32

You should see the rule triggered

New override key

Falco 0.37.0 replaces the append: true key-value pair with a new override section. The override section allows you to either replace or append keys to a rule, macro, or list value . It’s important to note that you cannot append and replace the same key; you must choose one or the other. Choosing both will result in an error.

The keys that can be modified vary according to the rules component being overridden. See the override documentation for the full list of keys that can be modified.

The override section can either be in a custom rules file or can be in the same file as the component being overridden. In either case, the override section needs to be specified after the rule that is being modified. When the override is in the same file, the override section needs to be below the original rule, list, or macro definition. If the override is in another file, that file needs to be loaded after the original rules file.

A quick example from the documentation illustrates how this new feature works.

In this example, the original rule is in falco_rules.yaml and the override is specified in falco_rules.local.yaml.

/etc/falco/falco_rules.yaml

- rule: program_accesses_file
 desc: track whenever a set of programs opens a file
 condition: proc.name in (cat, ls) and evt.type=open
 output: a tracked program opened a file (user=%user.name command=%proc.cmdline file=%fd.name)
 priority: INFO

/etc/falco/falco_rules.local.yaml

- rule: program_accesses_file
 condition: and not user.name=root
 output: A file (user=%user.name command=%proc.cmdline file=%fd.name) was opened by a monitored program
 override:
 condition: append
 output: replace

The modified program_accesses_filerule would trigger when ls or cat use open on a file, unless they were run by root.

The new output message would be A file (user=%user.name command=%proc.cmdline file=%fd.name) was opened by a monitored program

A final note, the old append: true method of modifying values has been deprecated and will be removed in Falco 1.0.0.

Additional UX improvements

Introduce a new unique engine key in falco.yaml to replace all the other methods of opening engines (FALCO_BPF_PROBE, --modern-bpf, -g, -e). See the deprecated features section for more info.
Falco now expands environment variables in falco.yaml even when they are part of a string. It is now possible to use syntax similar to this:

ebpf:
 probe: ${HOME}/.falco/falco-bpf.o

Our gVisor integration has also been improved by adding support for more events, including write, socketpair, timerfd_create and an updated configuration generator. In addition, we added support for any gVisor container ID format, making Falco more robust and compatible with gVisor sandboxed containers beyond Docker and Kubernetes.

Breaking changes

This is a list of breaking changes introduced in Falco 0.37.0

The Rate-limiter mechanism was removed as it is no longer used.
--userspace CLI option was removed as it’s no longer used.
The falco-driver-loader script is removed and embedded into falcoctl
The Helm chart 4.0.0 contains several modifications to work with the new k8s metadata collector. Please read its breaking change file for more information.
The new falcoctl driver implementation will drop:
- --source-only
- BPF_USE_LOCAL_KERNEL_SOURCES
- DRIVER_CURL_OPTIONS
- The FALCO_BPF_PROBE environment variable won't be used by the new falcoctl driver loader as it is already deprecated and scheduled to be removed in the next major version.
Various environment variables have been replaced as part of the new falcoctl driver feature:
- DRIVERS_REPO has been replaced by FALCOCTL_DRIVER_NAME or the --name command line argument.
- DRIVERS_NAME has been replaced by FALCOCTL_DRIVER_REPOS or the --repo command line argument.
- DRIVER_KERNEL_RELEASE has been replaced by --kernelrelease command line argument.
- DRIVER_KERNEL_VERSION has been replaced by --kernelversion command line argument.
- DRIVER_INSECURE_DOWNLOAD has been replaced by --http-insecure command line argument.
Remove -K/-k options from Falco in favor of the new k8s plugin
Dropped plugins shipped with Falco since plugins will now be managed by falcoctl. If you want to use a plugin like k8saudit be sure to install it at init time with falcoctl.
A new feature in Falco 0.37.0 allows environment variables to be expanded even if they are part of a string. This new functionality introduces a minor breaking change.

Previously, environment variables used in YAML that were empty or defined as “” would be expanded to the default value. This was inconsistent with how YAML was handled in other cases, where we only returned the default values if the node was not defined.

With Falco 0.37.0 we will return the default value for nodes that cannot be parsed to the chosen type. The program_output command will be environment-expanded at init time instead of letting popen; thus, the shell expands it.

This is technically a breaking change, even if no behavioral change is expected.

Note that you can avoid environment var expansion by using ${{FOO}} instead of ${FOO}. It will resolve to ${FOO} and won't be resolved to the environment var value.

You can find more information on breaking changes in the tracking issue.

Deprecated features

This is a list of features that will be removed in Falco 0.38.0

Modern probe Docker builder is no longer used.
syscall_buf_size_preset Falco config in favor of engine.kmod/ebpf/modern_ebpf.buf_size_preset.
syscall_drop_failed_exit Falco config in favor of engine.kmod/ebpf/modern_ebpf.drop_failed_exit.
modern_bpf.cpus_for_each_syscall_bufferFalco config in favor of engine.modern_ebpf.cpus_for_each_buffer.
FALCO_BPF_PROBE environment variable in favor of engine.ebpf.probe.
-e command line flag in favor of engine.replay.capture_file.
g,gvisor-config command line flag in favor of engine.gvisor.config.
gvisor-root command line flag in favor of engine.gvisor.root.
modern-bpf command line flag in favor of engine.kind=modern_ebpf.
nodriver command line flag in favor of engine.kind=nodriver.
syscall_event_drops falco config will be replaced by the metrics config plus some automatic notification on drops.

Be sure to check the tracker issue for more information.

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

The community is active on many fronts, and we plan on delivering more great features and stability fixes during the next release cycle!

Some of the things we are currently working on include:

Implement further improvements to our rule framework and rule syntax.
Add new features and enhancements to falcoctl to make it even more powerful.
Enhance the quantity, quality, and presentation of metrics in Falco.

And much much more

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Falco Weekly 4 - 2024

Fri, 26 Jan 2024 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

Libs will need a 0.14.2 tag for the Falco 0.37.0 release, with the revert of https://github.com/falcosecurity/libs/pull/1533 PR.
During our release process, we found out that the new std::filesystem based implementaton was up to 8x time slower than the old ones; that's because it supports much more cases and does many more checks.
Therefore, in https://github.com/falcosecurity/libs/pull/1645, we revert to the old sorcery implementation, plus some minor improvements and added tests.

Moreover, many more changes landed in libs, that won't be part of the upcoming Falco 0.37.0 release:

Modernized C++ struct/enum/union declarations: https://github.com/falcosecurity/libs/pull/1588
Added support for newfstatat syscall: https://github.com/falcosecurity/libs/pull/1628
Fixed a potential deadlock for kmod: https://github.com/falcosecurity/libs/pull/1629
Big effort by our hero, Jason, to cleanup some stale macros: https://github.com/falcosecurity/libs/pull/1633,https://github.com/falcosecurity/libs/pull/1634,https://github.com/falcosecurity/libs/pull/1635,https://github.com/falcosecurity/libs/pull/1637,https://github.com/falcosecurity/libs/pull/1638
A small fix for old ebpf driver to support some GKE envs: https://github.com/falcosecurity/libs/pull/1642
Solved a data race and segfault in logger: https://github.com/falcosecurity/libs/pull/1643
Allow to selectively disable bpf and kmod engines from cmake: https://github.com/falcosecurity/libs/pull/1644

Falco

Falco tag 0.37.0-rc2 is out! Try it!

Moreover:

syscall_event_drops was soft-deprecated to get ready for Falco 0.38.0 upcoming cleanups: https://github.com/falcosecurity/falco/pull/3015
Avoid storing escaped strings in engine: https://github.com/falcosecurity/falco/pull/3028
Bumped falcoctl to v0.7.1 and rules to 3.0.0: https://github.com/falcosecurity/falco/pull/3030,https://github.com/falcosecurity/falco/pull/3034
Fixed nlohmann_json library include paths when using system one: https://github.com/falcosecurity/falco/pull/3032
Fixes to new libsinsp state metrics handling: https://github.com/falcosecurity/falco/pull/3033

We are in the testing phase so any feedback would be appreciated! Moreover, we crafted a dedicated helm chart to test the new k8smeta plugin and the k8s-metacollector, you can read more about it here. Please note these 2 new components will be officially released with Falco 0.37.0 as EXPERIMENTAL features.

As a final reminder, please take a look at our polls if you have some spare seconds.

Falcoctl

Falcoctl 0.7.1 is out! Try it! and contains a small fix for the driver-loader on COS.

Moreover, we added dependabot configs, that then bumped lots of deps to their latest compatible versions: https://github.com/falcosecurity/falcoctl/pull/385.

Join relevant discussions

Our new discussion section: https://github.com/falcosecurity/falco/discussions
Breaking changes in Falco 0.37.0: https://github.com/falcosecurity/falco/issues/2763
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list
Open a discussion in our discussion section

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Andrea, Federico

Blog: Falco Weekly 3 - 2024

Fri, 19 Jan 2024 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

Libs tag 0.14.1 is out! Try it!

It fixes the following things:

fix(gvisor): gVisor engine crashes with non-hex container IDs: https://github.com/falcosecurity/libs/issues/1602
fix(gvisor): handle arbitrary sandbox IDs: https://github.com/falcosecurity/libs/pull/1612
fix(libsinsp): modify switch case: https://github.com/falcosecurity/libs/pull/1620
fix(libsinsp): Add new cgroup layout for podman: https://github.com/falcosecurity/libs/pull/1613
fix(libsinsp): consistent thread info filtering while dumping: https://github.com/falcosecurity/libs/pull/1606
fix(libsinsp): do not suppress zero ptids: https://github.com/falcosecurity/libs/pull/1598
fix(libsinsp): fix resolved PT_FSPATH and PT_FSRELPATH evt params: https://github.com/falcosecurity/libs/pull/1597

You can find a detailed summary on the release page.

Falco

Falco tag 0.37.0-rc1 is out! Try it!

Some final cleanup before the final tag:

cleanup(falco.yaml): rename none in nodriver: https://github.com/falcosecurity/falco/pull/3012
update(config): graduate outputs_queue to stable: https://github.com/falcosecurity/falco/pull/3016

As a final reminder, please take a look at our polls if you have some spare seconds.

Falcoctl

Falcoctl 0.7.0 is out! Try it!

These are some of the most relevant changes:

update(output): complete rework of the output system: https://github.com/falcosecurity/falcoctl/pull/335
update(cmd): remove redundant configuration for error handling: https://github.com/falcosecurity/falcoctl/pull/337
new(cmd): add artifact config command: https://github.com/falcosecurity/falcoctl/pull/340
feat(artifact/config): fetch config layer for a specific platform: https://github.com/falcosecurity/falcoctl/pull/349
new(artifact/manifest): add manifest command: https://github.com/falcosecurity/falcoctl/pull/351
new: driver command: https://github.com/falcosecurity/falcoctl/pull/343
new(pkg/driver): fixed some kernel version related issues: https://github.com/falcosecurity/falcoctl/pull/364
cleanup(cmd,internal,pkg): move driver config options to be common to all driver commands: https://github.com/falcosecurity/falcoctl/pull/365
fix(pkg/driver): do not call FixupKernel when building drivers: https://github.com/falcosecurity/falcoctl/pull/373
new: introduce asset artifact type: https://github.com/falcosecurity/falcoctl/pull/309

You can find a detailed summary on the release page.

Join relevant discussions

Our new discussion section: https://github.com/falcosecurity/falco/discussions
Breaking changes in Falco 0.37.0: https://github.com/falcosecurity/falco/issues/2763
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list
Open a discussion in our discussion section

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Andrea, Federico

Blog: Falco Weekly 50 - 2023

Fri, 15 Dec 2023 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

The anticipated 0.14.0 libs tag (and its driver counterpart) are going to be tagged soon, by the end of next week.
A xmas present for you all! :christmas_tree:

Mostly fixes were merged during this week:

Populate labels field for pod sandbox containers: https://github.com/falcosecurity/libs/pull/1564
Improved libscap modern bpf tests and CI checks: https://github.com/falcosecurity/libs/pull/1568
Avoid a double free when an exception is thrown during sinsp initialization: https://github.com/falcosecurity/libs/pull/1569
Made our pkg-config files paths-relative: https://github.com/falcosecurity/libs/pull/1570
Fixed some paths handling in fs.path: https://github.com/falcosecurity/libs/pull/1571
Do not include NULL terminator in enter event strings: https://github.com/falcosecurity/libs/pull/1574
Started a dedicated container engines test suite: https://github.com/falcosecurity/libs/pull/1544
Rewritten scary concatenate_paths function leveraging modern c++17 std::filesystem: https://github.com/falcosecurity/libs/pull/1533
Use a smart pointer for m_resolver in sinsp_dns_manager to avoid leaks: https://github.com/falcosecurity/libs/pull/1558

Also, thanks to actuated.dev for offering us arm64 github action runners, CI has been fully ported to github actions, except for a single CircleCI job! https://github.com/falcosecurity/libs/pull/1555

Rumors have it coming next:

Drivers build fix against linux 6.7-rc4+: https://github.com/falcosecurity/libs/pull/1566
Add k8s.pod.uid, k8s.pod.sandbox_id and mark k8s.pod.id as legacy: https://github.com/falcosecurity/libs/pull/1575

Falco

Falco has seen some big new features this week!

Env variables expansion was extended to all scalar values in Falco configuration file! https://github.com/falcosecurity/falco/pull/2918, https://github.com/falcosecurity/falco/pull/2972
Leveraging the above, engine.ebpf.probe path now defaults to ${HOME}/.falco/falco-bpf.o: https://github.com/falcosecurity/falco/pull/2971
CI has been ported to use actuated.dev github action arm64 runners! https://github.com/falcosecurity/falco/pull/2945, https://github.com/falcosecurity/falco/pull/2967
Monitor more types of events for Falco hot reload feature: https://github.com/falcosecurity/falco/pull/2965
libs and driver were bumped to latest master: https://github.com/falcosecurity/falco/pull/2970

Finally, the new falcoctl based driver-loader was finally merged in Falco: https://github.com/falcosecurity/falco/pull/2905.
If you can, please make sure to give it a spin and let us know any feedback, it is very valuable for us!
To try it out:

docker pull falcosecurity/falco-driver-loader:master
docker run --rm -i -t \
 --privileged \
 -v /root/.falco:/root/.falco \
 -v /proc:/host/proc:ro \
 -v /boot:/host/boot:ro \
 -v /lib/modules:/host/lib/modules \
 -v /usr:/host/usr:ro \
 -v /etc:/host/etc:ro \
 falcosecurity/falco-driver-loader:master

Falcoctl

Some fixes on top of the new driver-loader happened:

Cleanup eBPF probe symlink in Cleanup method: https://github.com/falcosecurity/falcoctl/pull/371
Do not call FixupKernel when building drivers: https://github.com/falcosecurity/falcoctl/pull/373

Moreover, we finally merged the new asset artifact type PR! https://github.com/falcosecurity/falcoctl/pull/309

Falcoctl is quite ready for v0.7.0 release; we only need more driver-loader testing!

Driverkit

Driverkit has seen a small bug fix release this week: https://github.com/falcosecurity/driverkit/releases/tag/v0.16.2.
It contains a fix to docker go package multiplexed output support: https://github.com/falcosecurity/driverkit/pull/310.

Moreover, we merged a PR that opens up the possibility for Driverkit to directly use cmake to configure and then build our drivers: https://github.com/falcosecurity/driverkit/pull/309.

What's next?
The cmake PR is opened and works super good; build times are as good as before, so no penalty! https://github.com/falcosecurity/driverkit/pull/302.
Moreover, we are going to make use of actuated.dev arm64 runners in driverkit too, porting its CI to github actions: https://github.com/falcosecurity/driverkit/pull/311.

Join relevant discussions!

Breaking changes in Falco 0.37.0: https://github.com/falcosecurity/falco/issues/2763
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Andrea, Federico

Blog: Falco Weekly 48 - 2023

Fri, 01 Dec 2023 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

The anticipated 0.14.0 libs tag (and its driver counterpart) are still a bit late, unfortunately.

Anyway, spring cleaning went on once again this week!

cleaned up dup3 flags param: https://github.com/falcosecurity/libs/pull/1469
cleaned up other params inconsistencies in the drivers: https://github.com/falcosecurity/libs/pull/1512
dropped b64 dep: https://github.com/falcosecurity/libs/pull/1518
dropped tinydir dep: https://github.com/falcosecurity/libs/pull/1516
removed some warning suppressions: https://github.com/falcosecurity/libs/pull/1519
cleaned up big unused function sinsp_evt::get_param_as_json: https://github.com/falcosecurity/libs/pull/1523

The big safer parameter handling PR was merged, making libs much more robust! Moreover, ppc64le support was extended to kmod and legacy ebpf probe, and we added CI jobs to test the build of drivers on it! Thanks to Afsan Hossain for his big contribution!

Finally, some more fixes:

build on s390x was fixed: https://github.com/falcosecurity/libs/pull/1522
some recently introduced regressions were fixed: https://github.com/falcosecurity/libs/pull/1524
fixed a memleak in sinsp_dns_manager: https://github.com/falcosecurity/libs/pull/1526

Rumors have it coming next week:

More fixes: https://github.com/falcosecurity/libs/pull/1530, https://github.com/falcosecurity/libs/pull/1528

Falco

We bumped libs and driver to latest master: https://github.com/falcosecurity/falco/pull/2929.
Moreover, Falco will now print system info during startup: https://github.com/falcosecurity/falco/pull/2927.
Falco does now expose a new config option to enable libsinsp state metrics: https://github.com/falcosecurity/falco/pull/2883 Finally, the new driver selection mechanism PR was merged!

Falcoctl

Some fixes on top of the new driver-loader happened:

fixed up naming for the new Falco driver selection in config: https://github.com/falcosecurity/falcoctl/pull/357
small fix for host-root driver-loader configuration: https://github.com/falcosecurity/falcoctl/pull/358
do not fail when /sys/kernel/debug fails to be mounted: https://github.com/falcosecurity/falcoctl/pull/361

Join relevant discussions!

Breaking change in Falco 0.37.0: https://github.com/falcosecurity/falco/issues/2763
Breaking change in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840
Falco metrics exposed to final users: https://github.com/falcosecurity/falco/issues/2928
Create a more coherent stats model for libs: https://github.com/falcosecurity/libs/issues/1463
Allow loading tracepoints other than the ones needed by Falco: https://github.com/falcosecurity/libs/issues/1376

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Federico

Blog: Falco Weekly 47 - 2023

Fri, 24 Nov 2023 00:00:00 +0000

Another week, another load of improvements everywhere in the falcosecurity!

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

The anticipated 0.14.0 libs tag (and its driver counterpart) are a bit late, unfortunately.

Anyway, spring cleaning went on this week!

removed stopwatch implementation, now unused: https://github.com/falcosecurity/libs/pull/1493
removed unused sinsp_test.cpp file: https://github.com/falcosecurity/libs/pull/1499
removed jq dep: https://github.com/falcosecurity/libs/pull/1500

Moreover, some fixes on the recently introduced async event queue class happened: https://github.com/falcosecurity/libs/pull/1490, https://github.com/falcosecurity/libs/pull/1504. Finally, some fixes around the stats code: https://github.com/falcosecurity/libs/pull/1505, https://github.com/falcosecurity/libs/pull/1506.

Rumors have it coming next week:

New big cleanup: deprecation of tracers: https://github.com/falcosecurity/libs/pull/1503
ppc64le support for bpf and kmod + CI build jobs: https://github.com/falcosecurity/libs/pull/1497
remove old metaevents implementation: https://github.com/falcosecurity/libs/pull/1495
Small fix on top of ia32 work: https://github.com/falcosecurity/libs/pull/1501

Second part of an effort by Luca Guerra to clean up libsinsp from potential undefined behavior: https://github.com/falcosecurity/libs/pull/1502.
This is so important that deserved to be left alone :)

Falco

We have a new official adopter! Welcome to Thought Machine: https://github.com/falcosecurity/falco/pull/2919 Small cleanup to avoid Falco configuratiom to be inited twice: https://github.com/falcosecurity/falco/pull/2917

Falcoctl

The new driver command was merged! https://github.com/falcosecurity/falcoctl/pull/343 We are now in the process of adding tests and eventually fixing spotted bugs :) Also, the new asset artifact type PR is being reviewed: https://github.com/falcosecurity/falcoctl/pull/309.

Others

Driverkit v0.16.0 was just released, and contains some fixes, a new local build processor and preliminary SLES support.

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Adding runtime threat detection to Google Kubernetes Engine with Falco

Mon, 20 Nov 2023 00:00:00 +0000

One of the big advantages of running your workloads on a managed Kubernetes service like Google Kubernetes Engine (GKE) is that Google ensures your clusters are being deployed and managed following industry best practices.

While GKE clusters are incredibly secure and reliable, there is always room for improvement.

In this blog, we’re going to describe how you can enhance GKE’s already great security by adding runtime threat detection with Falco.

What is Falco?

With Falco running on your GKE clusters you can be notified of a wide variety of events, such as:

Did someone start a container with high privileges?
Has someone shelled into a running container?
Has an executable been added to the container after it was deployed?

These are just a few examples. Falco has over 80 rules that can be used to make you aware of not only external threats but also when clusters aren’t being operated in accordance with industry best practices.

GKE Installation considerations

There are two different ways to install Falco on GKE. The first is using the prepackaged click-to-run offering in the Google Cloud Marketplace. The second is using Falco’s helm chart. The click-to-run offering is probably the simplest way to get up and running with Falco on GKE, but the drawback is that the version offered often lags behind the latest release.

It’s also important to note that as of this writing, you cannot run Falco on GKE clusters running in Autopilot mode. This is primarily because Falco uses an init container running with privileged access to install its driver, and Autopilot does not allow the execution of privileged containers.

Something else to be aware of is that Falco on GKE needs to use one of Falco’s eBPF drivers. Falco uses a driver to capture syscall events, and this driver is offered as a loadable kernel module or as an eBPF probe. There are actually two eBPF probes with Falco. One is called ‘eBPF’ (or classic eBPF) and the other is referred to as ‘modern eBPF’ - you can learn more about them in the Falco docs.

On the Google Cloud side, GKE uses Container-Optimized OS (COS) as the default operating system for its worker node pools. COS is a security-enhanced operating system that limits access to certain parts of the underlying OS. Because of this security constraint, Falco cannot insert its kernel module to process system calls. However, COS does support eBPF, so that’s the option we’ll use (more specifically we’ll use the classic eBPF probe)

Installing Falco via the Google Cloud Marketplace

Note: If you’d like to follow along, you’ll need to ensure your Google Cloud account has the appropriate permissions.

Installing Falco via the Google Cloud Marketplace is a pretty straightforward process.

Log into your Google Cloud account, and ensure you have the required permissions to deploy a new GKE cluster or operate an existing one.
Navigate to the Falco offering in the Google Cloud Marketplace.
Click the configure button.
From the next dialog you can choose the zone where your GKE cluster will run as well as the network and subnet on which it will run. For this walkthrough, the default values are fine.
You then choose whether or not you’d like to deploy Falco onto a new GKE cluster or use an existing one. Be aware that if you click Create New Cluster, Google Cloud will immediately start deploying a new cluster. Also, any Autopilot clusters that you have in your project will be grayed out and cannot be selected.
You can then choose which namespace Falco will run in. To keep things consistent with the rest of this blog, change it from default to falco.
Again, to keep things consistent with the rest of the blog, change the app instance name to falco.
Falco rules have different priority levels, you can choose the minimum priority level you’d like to run. The priority levels are ordered by severity, and typically the higher you make the minimum level, the fewer alerts you will receive (which helps to cut down on noise). For this example just leave it as debug.
Stackdriver is the old name for Google Cloud’s logging and monitoring suite. If you’d like to examine Falco’s metrics (not the actual alerts, but metrics on how Falco is performing) you can select that option. We won’t be covering that in this blog, so go ahead and leave it unchecked.
Click DEPLOY to deploy Falco onto the target cluster. (If you choose to deploy a new cluster, you will need to wait until that finishes to click the DEPLOY button.)

With that, Falco should be running on your GKE cluster. You can skip the next section, and continue with “Testing Falco”.

Installing Falco with Helm

Helm is the defacto way to install Falco on Kubernetes. Falco maintains an official Helm chart, and that chart is maintained as part of the overall Falco project.

If you’d like to follow along, you will need the following:

A Google Cloud account with appropriate permissions
A GKE cluster that you can operate
Helm and kubectl installed on your local computer or, alternatively, you can use Google Cloud Shell which has both Helm and kubectl already installed.

NOTE: Ensure that your kubectl context is set to the cluster on which you wish to install Falco.

With the pre-requisites out of the way, let's get started with the actual install.

Add the Falco chart to the Helm repository.

helm repo add falcosecurity \
https://falcosecurity.github.io/charts && \
helm repo update

Create the falco namespace for Falco to run in.
```
kubectl create namespace falco
```
Use Helm to deploy Falco. Notice that we use the driver.kind parameter to set the kernel driver to the eBPF probe.
```
helm install falco \
-n falco \
--set tty=true \
--set driver.kind=ebpf \
falcosecurity/falco
```
Wait for the Falco pods to come online.
```
kubectl get pods -n falco -w
```
Eventually you should see something similar to this:
```
falco-wfglg 2/2 Running 0 76s
falco-mdrlb 2/2 Running 0 91s
falco-7vxz6 2/2 Running 0 91s
```
Note: You will see one Falco entry for each of the nodes in your cluster. In this case, Falco is running on a 3-node cluster, so there are 3 entries.

Verify Falco is running correctly by examining the logs.

kubectl logs -n falco -c falco -l app.kubernetes.io/name=falco

You should see entries similar to this:

Fri Nov 3 15:48:07 2023: Falco version: 0.36.2 (x86_64)
Fri Nov 3 15:48:07 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Nov 3 15:48:07 2023: Loading rules from file /etc/falco/falco_rules.yaml
Fri Nov 3 15:48:07 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Fri Nov 3 15:48:07 2023: Starting health webserver with threadiness 2, listening on port 8765
Fri Nov 3 15:48:07 2023: Loaded event sources: syscall
Fri Nov 3 15:48:07 2023: Enabled event sources: syscall
Fri Nov 3 15:48:07 2023: Opening 'syscall' source with BPF probe. BPF probe path: /root/.falco/falco-bpf.o

Falco is now successfully running on your GKE cluster. The next step is to simulate some suspicious activity and verify that Falco detects it.

Testing Falco

One of Falco’s default rules fires an alert if someone shells into a running container. Follow the steps below to fire off that rule.

Start an Alpine container and have it sleep so it stays running.
```
kubectl run alpine –image alpine – sh -c "sleep infinity"
```
Execute a shell on the Alpine running container.
```
kubectl exec -it alpine -- sh -c "ls -al"
```

Now check the Falco logs to see the alert.

kubectl logs -c falco -n falco -l app.kubernetes.io/name=falco |\
grep "Notice"

You should see something like this:

18:52:06.630209324: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=root user_uid=0 user_loginuid=-1 process=sh proc_exepath=/bin/busybox parent=runc command=sh -c ls -al terminal=34816 exe_flags=EXE_WRITABLE container_id=e71eac85a570 container_image=docker.io/library/alpine container_image_tag=latest container_name=alpine k8s_ns=default k8s_pod_name=alpine)

Notice all the details the alert provides including the container ID, image, and name, as well as the executed command.

Conclusion

As mentioned at the outset, one of the big advantages of running a managed Kubernetes service is that a lot of the heavy lifting for hardening the cluster has been done for you. However, by using Falco to provide runtime insights into the activity on your cluster you can help ensure that the cluster is being operated responsibly or has not been compromised by any bad actors.

If you’d like to learn more about Falco, head on over to the docs or our GitHub repository. We also have our own channel (#Falco) on the Kubernetes Slack server.

Blog: Falco Weekly 46 - 2023

Fri, 17 Nov 2023 00:00:00 +0000

This is the first of a series of weekly blog post whose aim is to give a quick overview about the development of Falco and its related projects.

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

Lots of cleanups happened in the libs repo; the most outstanding ones being:

udig engine removal (https://github.com/falcosecurity/libs/pull/1485)
dropped legacy metadata clients for k8s and mesos (https://github.com/falcosecurity/libs/pull/1478)
cleaned up proc callback handling code (https://github.com/falcosecurity/libs/pull/1471)

Please, note that the removal of the legacy k8s client is part of a bigger effort to entirely rewrite it as a plugin, with a more future proof architecture and language.
See the tracking issue: https://github.com/falcosecurity/libs/issues/987.

All of these cleanups account for ~26k loc removed!! :rocket:

Moreover, some fixes landed:

removed some more Undefined Behavior warnings from integer copies (https://github.com/falcosecurity/libs/pull/1481)
solved win32 linking issues with zlib (https://github.com/falcosecurity/libs/pull/1484)
prevent libbpf stats from being collected with no bpf stats (https://github.com/falcosecurity/libs/pull/1487)

Finally, some new features were merged:

libraries will now be properly installed under CMAKE_INSTALL_LIBDIR (https://github.com/falcosecurity/libs/pull/1101)
added ppc64le experimental support for modern bpf driver (https://github.com/falcosecurity/libs/pull/1475)
upgraded openssl to 3.1.4 (https://github.com/falcosecurity/libs/pull/1488)

Also, we now have a target release date and a tracking issue for libs 0.14 and next driver release: https://github.com/falcosecurity/libs/issues/1482.

Falco

Now Falco builds and runs on win32 and osx too! https://github.com/falcosecurity/falco/pull/2889 While Falco won't ship for these platforms, we will now have proper CI for them.

Following the huge round of cleanups in libs, k8s and mesos related configs and options were removed: https://github.com/falcosecurity/falco/pull/2914. Also, another small cleanup relative to the legacy k8saudit implementantion (not the plugin one!) was merged: https://github.com/falcosecurity/falco/pull/2913.

Falcoctl

While the code for the new driver-loader feature for falcoctl is being reviewed (part of the effort to drop falco-driver-loader script (https://github.com/falcosecurity/falcoctl/issues/327 and https://github.com/falcosecurity/falco/issues/2675), some features landed too:

fetch config layer for a specific platform (https://github.com/falcosecurity/falcoctl/pull/349)
added a new artifact manifest command (https://github.com/falcosecurity/falcoctl/pull/351)

Others

A new repo, k8s-metacollector, was donated to the falcosecurity.
It is a self-contained module that fetched metadata from kubernetes API server and dispatches them to Falco instances via gRPC.
A new plugin is being developed to receive those metadata from gRPC, and will be shipped with Falco 0.37.

Driverkit gained support for SUSE Linux Enterprise: https://github.com/falcosecurity/driverkit/pull/304.

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Andrea, Federico

Blog: Introducing the new Falco training course, by CNCF, Linux Foundation, and Sysdig

Mon, 06 Nov 2023 00:00:00 +0000

Detecting Cloud Runtime Threats with Falco (LFS254) is the new Falco training course created by CNCF, Linux Foundation, and Sysdig. We're very excited about this new immersive course designed to enhance your expertise in securing cloud-native applications through hands-on learning.

Detecting Cloud Runtime Threats with Falco (LFS254) is a 20-hour course focused on runtime security. It covers what is runtime security and how Falco is a powerful tool designed to detect anomalous activity in applications. From Falco's history and design principles to its architecture, to how it addresses cloud security challenges.

This course is designed for IT professionals, security analysts, DevOps engineers, and anyone interested in cloud security.

Why?

In a rapidly evolving digital landscape with a surge in cloud adoption, the importance of comprehending and deploying robust security solutions, such as Falco, cannot be overstated. Regrettably, cloud-native technologies, particularly cloud-native security, are relatively novel, and there exists a gap in knowledge and expertise for addressing these emerging challenges. Our mission is to bridge this knowledge gap and empower individuals to tackle cloud and container security complexities effectively. Through accessible training, we aspire to contribute to narrowing the talent deficit in these pivotal domains.

How?

In this course, you'll embark on a journey of securing cloud-native environments. The course breaks down complex concepts, making them accessible and actionable. Its self-paced nature provides the flexibility to learn at your own rhythm, accommodating your personal and professional commitments. This structure allows you to digest intricate concepts and apply them bit by bit, ensuring a deeper and more lasting comprehension.

Course Structure

The course begins with an introduction to Falco, encompassing its history, design principles, and its broader role in cloud security. It then delves into the core components of Falco, explaining its architectural design and walking you through the setup and operation of Falco. Moving forward, the course explores the significance of the system call data source in host security, offering insights into the nature of system calls, observation techniques, and best practices for efficient data collection. It further showcases Falco's versatility by examining its utilization of diverse data sources such as Github, Cloudtrail, and Kubernetes Audit logs through its Plugin Framework.

The course also thoroughly covers conditions and fields, delving into the realm of Falco default rules and their integration with security frameworks. It then provides comprehensive guidance on customizing Falco rules to align with specific requirements. The course also addresses Falco outputs and introduces Falcosidekick as a valuable output management and customization tool.

Finally, the course guides you through Falco's configuration process and fine-tuning strategies. It concludes by streamlining the process of writing new Falco rules, presenting a development methodology, along with key considerations to bear in mind when crafting rules.

Sneek Peek

In one of the course exercises, we explain the Log4j vulnerability and the Log4Shell exploit. We detail each step of the attack, allowing you to simulate it in the lab environment.

Then, we walk you through how to write a new rule to detect this type of attack in Falco.

Enroll Now

Ready to embark on this transformative journey? Visit the course page to enroll and step into the world of cloud-native security mastery with Falco.

Blog: Introducing Falco 0.36.2

Fri, 27 Oct 2023 00:00:00 +0000

Today we announce the release of Falco 0.36.2 🦅!

Fixes

Falco's 0.36.2 release is a small patch addressing a few bugs. It includes the following:

Fixed a possible segfault caused by uninitialized variable in libsinsp::next() method call. (https://github.com/falcosecurity/falco/issues/2878)
Improved supported program type detection for modern BPF; this ensures we can actually be sure that our BPF program type is unsupported when returning an error to the user. (https://github.com/falcosecurity/libs/pull/1404)
Fixed a subtle bug in rawarg filtercheck for non-string types. (https://github.com/falcosecurity/libs/pull/1428)
Fixed an uninitialized variable in the libscap bpf engine that lead to stdin getting closed while Falco soft restarted. (https://github.com/falcosecurity/libs/issues/1448)

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.36.2, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

What next? 🔮

The community is active on many topics and we hope to deliver great features and many stability fixes once again during the next release cycle!

The old falco-driver-loader script is showing its age and it's time to work on a more maintainable solution. falcoctl is a great candidate to host everything driver related, implement new features and make our lives easier when we need to install Falco drivers on a new machine.
Our rule framework is brand new and we forsee many improvements and active development work on it.
The latest Falco versions brought many improvements to the plugin framework; we wish to use those to create a more scalable Kubernetes client plugin that will be able to withstand much heavier loads and will be easier to maintain.

And many, many, more enhancements!

To get a weekly reminder of all the great stuff happening in the Falco lands, make sure to join the #falco channel on the Kubernetes Slack!

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions:

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Introducing Falco 0.36.1

Mon, 16 Oct 2023 00:00:00 +0000

Today we announce the release of Falco 0.36.1 🦅!

Fixes

Falco's 0.36.1 release is a small patch aimed at protecting our uses by addressing a few minor bugs. It includes the following:

Address a HIGH severity vulnerability in libcurl CVE-2023-38545, bumping the library to the patched version 8.4.0. You can find more details in the section below.
The legacy eBPF probe can now handle systems with CPU hotplug enabled, opening the right number of kernel buffers. (https://github.com/falcosecurity/falco/issues/2843)
Remove a no longer useful experimental Falco config outputs_queue.recovery. This was introduced in Falco 0.36.0 as an experiment.
Fix a possible segfault caused by a faulty implementation of timer_delete. (https://github.com/falcosecurity/falco/issues/2850)

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Vulnerability in libcurl

A HIGH severity vulnerability in libcurl, CVE-2023-38545, was disclosed alongside a patched version (8.4.0). We would like to answer the main question you might have about it: Does it affect Falco?

According to the excellent in-depth description of the bug, this can only be triggered if both conditions below are true:

A SOCKS5 HTTP(S) proxy has been configured. This happens if you have set the standard environment variables that control proxy connections, such as http_proxy/https_proxy/no_proxy or libcurl-specific ones as indicated in the advisory or the libcurl documentation.
An attacker controls the server that Falco is connecting to, namely the server configured to receive http_output or a custom prebuilt driver repository server, and the SOCKS5 proxy is "slow enough" to allow the attack to happen.

While it may be rare that users have an exploitable environment, it's still a possibility. For this reason, Falco maintainers decided to ship this patch release 🦅

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.36.1, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

What next? 🔮

The community is active on many topics and we hope to deliver great features and many stability fixes once again during the next release cycle!

The old falco-driver-loader script is showing its age and it's time to work on a more maintainable solution. falcoctl is a great candidate to host everything driver related, implement new features and make our lives easier when we need to install Falco drivers on a new machine.
Lately we have expanded the syscall coverage that Falco can provide. We wish to improve these efforts across all drivers with even more 32 bit syscalls.
Our rule framework is brand new and we forsee many improvements and active development work on it.
The latest Falco versions brought many improvements to the plugin framework; we wish to use those to create a more scalable Kubernetes client plugin that will be able to withstand much heavier loads and will be easier to maintain.

And many, many, more enhancements!

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Andrea, Luca

Blog: Linux Introspection - From BPF to Wireshark to Falco

Wed, 11 Oct 2023 00:00:00 +0000

Falco, an open source innovation, was conceived with the vision of crafting a flexible and robust rules engine atop the Sysdig libraries. This initiative aimed to furnish a potent tool for the detection of aberrant behaviors and intrusions within modern applications, akin to the Snort paradigm but tailored to the realm of system calls and finely tuned for cloud environments.

Nevertheless, it's important to recognize that Falco and Wireshark represent distinct facets of this evolutionary process. Falco offers ongoing surveillance akin to Snort, while Wireshark specializes in interactive endpoint network traffic analysis.

The Need for Modern System Introspection

Part of this journey has been the emergence of cloud native apps. From the early days of BPF (Berkley Packet Filter) and libpcap (a portable C/C++ library for network traffic capture), which laid the foundation for network packet analysis, to the familiar graphical user interface of Wireshark, our understanding of network data has undergone profound changes. This article embarks on a journey through this transformation, shedding light on how tcpdump and libpcap sparked an explosion of packet-based analysis and runtime security tools exemplified by Wireshark and Snort.

Wireshark, Snort, Nmap, Kismet, ngrep, and a bunch of other tools started at around the same time and are all evolutionary branches of tcpdump and libpcap.

However, as cloud computing continues to reshape the technological landscape, traditional network packet analysis tools have found themselves grappling with an evolving challenge: the cloud itself. Cloud native applications have ushered in a new era of complexity and dynamism, rendering many existing visibility solutions obsolete. This shift necessitated a fresh perspective on network monitoring, leading to the birth of Falco, a tool poised to be the Snort of the cloud.

Starting the story with Network Packet Analysis

During the late 1990s Internet boom, the demand for computer networks skyrocketed, leading to an increased need for monitoring, troubleshooting, and securing these networks. Regrettably, the available network visibility tools of that era were prohibitively expensive for many operators, leaving them grappling with a lack of insights.

Consequently, teams worldwide embarked on a mission to address this predicament. Their efforts revolved around expanding existing operating systems to incorporate packet capture capabilities, essentially transforming off-the-shelf computer workstations into devices capable of residing on a network and capturing all inbound and outbound data packets from other workstations. One such solution was the Berkeley Packet Filter (BPF), crafted to extend the functionality of the BSD (Berkeley Software Distribution) operating system kernel.

For Linux users, the term 'eBPF' may ring a bell – a virtual machine renowned for securely executing arbitrary code within the Linux kernel. Remarkably, eBPF has evolved into a powerful and flexible technology over the years. However, its origins trace back to a modest programmable packet capture and filtering module designed for BSD Unix.

The BPF team introduced a game-changing library known as 'libpcap,' which enabled any program to capture raw network packets. It was developed in order to make tcpdump more useful. For instance, it gave the ability to filter packets. Since then, a bunch of spin-off networking projects would emerge on the scene. In 1998, a GUI-based open source protocol analyzer named 'Ethereal' (later renamed Wireshark) was introduced, eventually becoming the gold standard in packet analysis that persists to this day. \

What unites 'tcpdump,' Wireshark, and numerous other popular networking tools is their ability to access a data source that is rich, accurate, and reliable, all collected in a nonintrusive manner: raw network packets. This fundamental concept will be central to our discussion moving forward.

The evolution of Packet-Based Intrusion Detection Systems

Introspection tools, such as tcpdump and Wireshark, naturally emerged as the initial applications harnessing the capabilities of the BPF packet capture stack. However, as time progressed, innovative applications for packet data began to surface. Enter Snort, an open source, packet-based runtime security tool that shares common ground with Falco. Much like Falco, Snort operates as a rule engine, processing packets acquired from network traffic. Like its cloud native counterpart, Snort boasts an extensive library of pre-configured rules designed to identify threats and unwarranted activities by scrutinizing packet content, protocols, and payload data. The success of Snort served as a catalyst for the development of similar tools, including Suricata and Zeek.

What truly empowers tools like Snort is their proficiency in assessing the security of networks and applications in real time, even as these applications run. This real-time focus proves invaluable by delivering immediate protection with a unique emphasis on runtime behavior, enabling the detection of threats rooted in vulnerabilities that may remain undisclosed.

The issue with Network Packet Capture in the Cloud

The utilization of network packets as a foundational data source has spawned a thriving ecosystem. Nonetheless, several emerging trends have gradually eroded the viability of packets as an unequivocal source of information.

First, the task of comprehensively collecting packets has grown increasingly complex, especially within environments such as the cloud, where access to routers and network infrastructure is constrained. Second, the proliferation of encryption and network virtualization has posed formidable challenges in extracting valuable insights from network traffic. Lastly, the ascent of containerization and orchestrators like Kubernetes has rendered infrastructures more elastic while concurrently complicating the reliable collection of network data.

Once again, a dynamic new ecosystem was unfolding, yet the means to effectively troubleshoot and secure it remained elusive.

System Calls are the New Network Packets

Before the emergence of Falco, an open source tool known as 'Sysdig Inspect' was crafted with a primary focus on the collection of packet data within cloud native ecosystems. This was achieved through the capture of system calls, often referred to as syscalls, originating from the kernel of the operating system.

Syscalls, as a data source, offer a richness that surpasses that of mere network packets. They encompass a wide spectrum of activities, extending beyond network data to encompass file I/O operations, command executions, interprocess communication, and more. Syscalls stand out as an ideal data source for cloud native environments as they can be harnessed from the kernel, catering to both containerized environments and cloud instances. Moreover, the process of collecting syscalls is characterized by its simplicity, efficiency, and non-invasiveness.

The architecture of Sysdig comprised a kernel capture probe, making use of either the default, loadable kernel module or leveraging eBPF. To facilitate the development of capture programs, Sysdig offered a suite of libraries, enabling seamless integration with modern cloud native technologies such as Kubernetes and various orchestrators. This versatility addressed the shortcomings observed in environments where traditional solutions like Snort and Wireshark fell short. Additionally, Sysdig provided a command-line tool replete with decoding and filtering functionalities, tailored to accommodate the prevalent network packet workflows essential in cloud environments, where the ease of filtering and scriptability of trace files is paramount.

Falco - the evolution of Wireshark to the Cloud

Drawing from our comprehension of how Snort introduced a rule-based engine for scrutinizing network traffic to identify suspicious activity, an evolution that implemented Wireshark's network introspection, and how Sysdig expanded the scope of visibility within cloud native environments by delving into system calls, effectively departing from sole reliance on Wireshark's libpcap framework. It logically followed that an Intrusion Detection System (IDS) solution would emerge, featuring a sophisticated rule-based engine tailored for cloud native workloads while harnessing the capabilities of eBPF and the kernel's system call architecture.

Falco's rule engine drew inspiration from Snort's design but operated within a far more expansive and versatile dataset, seamlessly integrated with the Sysdig libraries. While its default ruleset may be more concise than Snort's, Falco empowers users to craft intricate rules that trigger in real-time based on arbitrary contextual factors. These factors encompass a wide array of scenarios, including access to sensitive data, mode transitions, unexpected network connections, socket alterations, compliance breaches, and more. Given its capacity to monitor all activities on a server or node through system calls, Falco functions as a real-time intrusion detection tool, mirroring Wireshark's role in providing real-time network analysis for endpoints.

Falco for Cloud Native Security

In the journey from the early days of BPF to the widespread adoption of Wireshark, we've witnessed the remarkable evolution of system introspection tools, each one contributing to the ever-expanding landscape of cybersecurity. However, as cloud native computing and microservices architectures become the new norm, a new champion has emerged: Falco. Falco represents the cutting edge of intrusion detection, specifically designed to tackle the intricacies and challenges posed by cloud native hosts and workloads. With its real-time behavioral monitoring, container awareness, and comprehensive rule sets, Falco stands as a testament to the adaptability and innovation in the world of cybersecurity. As the digital landscape continues to evolve, Falco is the tool of choice for those who prioritize the security and integrity of their cloud native environments. It's not just a system introspection tool; it's the future of protecting what matters most in this rapidly changing world of technology.

If you want to try out Falco, check out our Getting Started documentation. Join our community at #falco channel within Kubernetes Slack.

Blog: Tracing System Calls Using eBPF - Part 2

Fri, 06 Oct 2023 00:00:00 +0000

Introduction

In Tracing System Calls Using eBPF Part 1, we lay the groundwork, introducing you to the fundamentals of eBPF and its predecessor, BPF (Berkeley Packet Filter). We delve into the evolution of this technology, its safety, performance, and observability advantages over traditional kernel modules, and its pivotal role in securing modern cloud native environments. We guide you through the intricate process of working with eBPF programs, from compilation to execution, highlighting its power in tracing system calls.

In the second installment, Tracing System Calls Using eBPF Part 2, we elevate our understanding of eBPF's capabilities. We unravel the world of Uprobes and Uretprobes, demonstrating how these features empower developers to instrument and monitor user-space applications seamlessly. We then venture into Kprobes and Kretprobes, unlocking the potential to dynamically trace and debug kernel functions, offering insights into system behavior and performance analysis.

Uprobes

Uprobes, short for user probes, are a feature in the Linux kernel that enables developers to instrument and monitor user-space applications without modifying their code directly. They allow for the insertion of breakpoints at specific points of interest within an application, facilitating the collection of data, tracing of function calls, debugging, and performance analysis.

Uretprobes

Uretprobes, short for User Return Probes, are a feature in the Linux kernel that allows developers to trace and monitor the return paths of functions in user-space applications. While uprobes are used to instrument and intercept the entry points of functions, URETprobes specifically focus on the exit points or return paths. They enable developers to set up probes that are triggered when a specific function returns to its caller.

Here is an eBPF program that uses user probes to trace the printf function present in glibc (the standard GNU C Library).

In accordance with the instructions outlined in our Tracing System Calls Using eBPF Part 1 blog, we can create a loader to effectively load this eBPF program and read the logs from the file /sys/kernel/tracing/trace_pipe .

Kprobes

Kprobes, short for Kernel Probes, are a feature in the Linux kernel that allow dynamic tracing and debugging of kernel functions. They are particularly useful for tasks like performance analysis, bug diagnosis, and system monitoring. They provide a non-intrusive way to gather runtime information from the kernel without requiring modifications to the kernel code itself. Additionally, they can be used to trace specific function calls, track parameters and return values, and gather statistical data on function execution

Kretprobes

Kretprobes, short for Kernel Return Probes, are a feature in the Linux kernel that complements Kprobes by allowing dynamic tracing and debugging of kernel function return points. While Kprobes focus on probing the entry points of kernel functions, kretprobes specifically target the return points of these functions.Similar to Kprobes, kretprobes work by inserting a probe handler function that gets executed when a specific kernel function is about to return. This allows developers and system administrators to gather information, modify return values, or perform additional actions at the point of function return.

Here is an eBPF program that uses kernel probes to trace a kernel function named prepare_kernel_cred. This function is used to create a new struct cred object that represents the credentials or privileges associated with a kernel task. It is commonly used in privilege escalation exploits for gaining root access. By tracing this function, we can identify all processes that invoke it, providing valuable insight for analyzing potential malicious activity.

SEC(“kprobe/prepare_kernel_cred”) indicates that an eBPF program is associated with the kprobe event for the "prepare_kernel_cred" kernel function. This event allows dynamic tracing and debugging by intercepting the entry point of the function.

struct pt_regs is a data structure that provides access to the register state of the program when it is executed. It contains information about the CPU registers at the time of the eBPF program invocation. It is defined as :

To facilitate the loading of the aforementioned eBPF program, we’ll use the following program.

Here is a Makefile for compiling the eBPF program and the loader

Conclusion

In this two-part exploration of Tracing System Calls Using eBPF, we've embarked on a fascinating journey through the inner workings of this powerful technology. Part 1 laid the foundation by introducing us to the fundamentals of eBPF and its predecessor, BPF, shedding light on their evolution and significance in modern cloud native environments. We uncovered how eBPF's safety, performance, and observability advantages empower us to trace system calls with unmatched efficiency.

In Part 2, we took our understanding to new heights. We delved into the world of Uprobes and Uretprobes, showcasing how they enable seamless instrumentation and monitoring of user-space applications. We then ventured into Kprobes and Kretprobes, unlocking the ability to dynamically trace and debug kernel functions. Armed with these advanced techniques, we gained valuable insights into system behavior, performance analysis, and even the detection of potential malicious activity.

As we conclude this journey into the heart of eBPF, we stand equipped with a powerful set of tools and knowledge. Whether you're a seasoned sysadmin, a curious developer, or a vigilant security enthusiast, the capabilities of eBPF open new doors to real-time monitoring and analysis.

Stay tuned for further insights and practical guidance in the world of eBPF, where innovation meets security, and the future of system monitoring becomes a reality.

Blog: Falco 0.36.0

Tue, 26 Sep 2023 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.36.0!

This releases comes as usual with many new features and improvements. Thanks to everyone that worked on all the features, bugfixes and improvements! To read a detailed account of the release, see v0.36.0 in the changelog.

During this release cycle, we merged more than 100 PRs on Falco and more than 150 PRs for libs and drivers, version 0.13.1 and version 6.0.1 respectively. Thank you to our maintainers and contributors, as this would not happen without your support and dedication!

This release comes with many brand new features, some long awaited UX improvements and configuration and also beware of some breaking changes! Don't worry, everything is explained below!

What's new? TL;DR

In release v0.36.0, we focused on the following features:

Brand new Falco rule framework and ruleset
More robust executable file path detection, symlink resolution and ancestors detection
Falco is no longer limited to one rule firing per event!
Signatures are now automatically verified in Falcoctl for plugins and rules
Upgrade of the default Falco images

We have also some massive experimental upgrades that the community has spent incredible amounts of effort on:

WASM support
Kernel driver testing at scale!
Falco now has an experimental distroless container image based on Wolfi

Breaking changes ⚠️

We have seen many requests from the community in the form of questions and issues. Those are the ones that shape the evolution of Falco, so we can hopefully make the user experience better at every release. Sometimes, in order to do this we need to implement changes that may impact some workflows. In this release we have important breaking changes you should be aware of:

The default rules file that is shipped in the Falco image and/or can be downloaded via falcoctl as falco-rules is now a stable rule file. This file contains a much smaller number of rules that are less noisy and have been vetted by the community. This serves as a much requested "starter" Falco rule set that covers many common use case. The rest of that file has been expanded and split into falco-incubating-rules and falco-sandbox-rules. Read more below to learn about the difference.
The main falcosecurity/falco container image and its falco-driver-loader counterpart have been upgraded. Now they are able to compile the kernel module or classic eBPF probe for relatively newer version of the kernel (5.x and above) while we no longer ship toolchains to compile the kernel module for older versions in the default images. Downloading of prebuilt drivers and the modern eBPF will work exactly like before. The older image, meant for compatibility with older kernels (4.x and below), is currently retained as falcosecurity/falco-driver-loader-legacy.
The Falco HTTP output no longer logs to stdout by default for performance reasons. You can set stdout logging preferences and restore the previous behavior with the configuration option http_output.echo in falco.yaml.
The --list-syscall-events command line option has been replaced by --list-events which prints all supported system events (syscall, tracepoints, metaevents, internal plugin events) in addition to extra information about flags.
The semantics of proc.exepath have changed. Now that field contains the executable path on disk even if the binary was launched from a symbolic link.
The -d daemonize option has been removed.
The -p option is now changed:
- when only -pc is set Falco will print container_id=%container.id container_image=%container.image.repository container_image_tag=%container.image.tag container_name=%container.name
- when -pk is set it will print as above, but with k8s_ns=%k8s.ns.name k8s_pod_name=%k8s.pod.name appended
Command line options s and stats-interval have been removed in favor of metrics config in falco.yaml.

Major features and improvements

New Falco rules framework 🛡️

This project is the result of a discussions that started a long time ago and required a massive amount of work from the community. Following this proposal we have decided to split the rules that the Falco community maintains into three main groups, described in the maturity levels section of the contributing guide:

Stable Falco rules. Those are the only ones that are bundled in the Falco by default. It is very important to have a set of stable rules vetted by the community. To learn more about the criterias that are required for a rule to become stable, see the contributing guide
Incubating rules, which provide a certain level of robustness guarantee but have been identified by experts as catering to more specific use cases, which may or may not be relevant for each adopter.
Sandbox rules, which are more experimental.

It is important to keep in mind that the stable ruleset is significantly changed since the last release! Not only the rules are a much smaller subset but they have been refined and they may have been renamed according to the style guide.

Thanks to Melissa Kilby for driving this effort 🚀!

The list of releases for each type of rule is present in the repository, where you can download each file. They can also be downloaded from the download page and are also available as signed OCI artifacts for download via falcoctl!

Want to contribute to the rules? You can find more information in the contribution guide and the style guide.

Process executable and lineage 🪪

We have achieved a higher level of accuracy and data quality regarding the existing proc.exepath field and the process tree reconstruction in general. This step forward reinforces our commitment to refining Falco and providing you with an even better user experience.

In more detail:

The proc.exepath process executable path field now contains a resolved version of the executable path, meaning that even if an executable was launched from a symlink, the field will show the original location of the binary. In the past, we resolved the exe argument in userspace by utilizing the process's cwd when the path was not absolute. Conversely, if exe was absolute, the exepath was equivalent to exe. The new implementation ensures the extraction of the authentic and accurate disk path of the executable when it resides on the disk.
As it turns out, it's not that simple to reconstruct the complete process tree in a Linux system. The Linux kernel presents intriguing edge case behaviors, where the direct parent process might genuinely have already exited. In the past, Falco encountered difficulties in continuing to reconstruct the parent process lineage in such situations. To address this, we've enhanced Falco's logging capabilities. Now, even in scenarios where the parent process has exited, Falco can continue reconstructing the process tree.

Container image changes 📦

We have two big changes to our default container images:

The falco-driver-loader image is now based on Debian Bookworm with a more modern version of compilers, meaning that it will be much easier to build on contemporary systems but you might see compilation issues for older kernels (4.x and below). For that, the falco-driver-loader-legacy image is provided! Also, this means that vulnerability scanners will not report so many false positive vulnerabilities in the new version of the image since it does not contain legacy versions of compilers.
We have a falco-distroless image based on Wolfi, thanks to contributions from Adrian Mouat and the Falco Supply Chain Security WG! This is for all of you that are fans of minimal images! You can try it out by replacing falco-no-driver with falco-distroless.

Falcoctl ❤️ cosign

Since Falco 0.35.0 we started providing signed official container images signed with cosign in keyless mode. But how about our other OCI artifacts, which are rules and plugins? Starting from Falcoctl 0.6.1, shipped with this release, all of the official rules and plugins are signed and automatically verified at installation time thanks to the magic of cosign in keyless mode!

Thanks to Massimiliano Giovagnoli for his help along with the Falco Supply Chain Security WG! Stay tuned for an in-depth explanation of the security architecture of this feature.

Multiple rules can be matched on each event

Pro Falco users know that we could only match one rule for each event. This is not true anymore, and since this version we have a rule_matching option in the configuration file. rule_matching: all will remove this limitation and match everything. See the documentation in falco.yaml for more information!

Big experimental contributions

Last but not least, we have several big projects that we have started with the community and are very proud of.

Falco Kernel Testing Framework

Falco supports a large number of Linux kernels. And the truth is, in order to test this kind of functionality you have to start an (ideally) equally large number of live Linux systems and load the driver there. This is absolutely not easy to do and just taking a look at the task list for such an endeavor gives you an idea of the complexity required. The results are awesome: you can find a matrix of kernels that are continuously tested for x86_64 and ARM as well! See the in-depth blog post to learn much more about this!

Falco WASM

Flaco is excited to introduce its latest addition: the WebAssembly target. This new target has been developed exclusively for the Falco Playground using Emscripten, where it brings essential core functionalities to the forefront. These functionalities include a rule compiler and the ability to reproduce events from capture files. It’s worth noting that certain features, such as kernel modules and Kubernetes support, have been intentionally omitted from this wasm target. This omission is due to the inherent limitations of running these features within a web browser environment. falco.wasm can be found as a github artifact in the latest workflow.

Falco Playground

Falco playground is simple web application where you can create, edit and validate falco rules. This is a quick solution for users wanting to easily check the accuracy of their custom rules. This application is completely client side and doesn’t make calls to any backend server. It leverages the power of WebAssembly to test your rules. You can try it live and find the code in the falco-playground repository!

Additional UX improvements

With each release, Falco gets more quality-of-life improvements, such as:

Environment variables resolution in configuration files
A new outputs_queue configuration option to better fine tune Falco's output performance

Deprecated features

It's sad to see features go, but sometimes we need to remove something in order to focus on what matters for our adopters. This is what maintainers are proposing for deprecation in this release and removal in the next Falco version 0.37.0:

The optional rate-limiter mechanism, since it seems to be no longer used and it also can discard events including potentially critical alerts
The --userspace option, since the corresponding feature and the associated projects in the Falco organization have not been maintained for years
The falco-driver-loader bash script. The driver loading functionality is going to be implemented in falcoctl to improve Falco's driver loading capabilities and make it easier to maintain and contribute to.

Try it out

It’s time to try out the new release! Here are some pointers for getting started with Falco:

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What next? 🔮

The community is active on many topics and we hope to deliver great features and many stability fixes once again during the next release cycle!

The old falco-driver-loader script is showing its age and it's time to work on a more maintainable solution. falcoctl is a great candidate to host everything driver related, implement new features and make our lives easier when we need to install Falco drivers on a new machine.
Lately we have expanded the syscall coverage that Falco can provide. We wish to improve these efforts across all drivers with even more 32 bit syscalls.
Our rule framework is brand new and we forsee many improvements and active development work on it.
The latest Falco versions brought many improvements to the plugin framework; we wish to use those to create a more scalable Kubernetes client plugin that will be able to withstand much heavier loads and will be easier to maintain.

And many, many, more enhancements!

Stay tuned 🤗

Join us in our communication channels and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to having your feedback and hearing your ideas.

You can find all the most up to date information at https://falco.org/community/.

See you for the next release!

Enjoy,

Luca, Andrea, Rohith

Blog: Tracing System Calls Using eBPF - Part 1

Mon, 11 Sep 2023 00:00:00 +0000

Introduction:

In this article, we will delve into the details of eBPF (extended Berkeley Packet Filter) and explore its significance in tracing system calls. This particular blog will be in two parts; in the first blog, we will discuss eBPF, and in the subsequent section, we will delve into probes. eBPF is a powerful technology that allows for the dynamic and efficient tracing of events within the kernel space of an operating system. You have probably heard of the acronyms BPF and eBPF being used interchangeably. That's why we will aim to address both BPF and eBPF before discussing how and why Falco uses this technology.

BPF (Berkeley Packet Filter)

BPF is a technology used for network packet filtering and analysis. It is a powerful tool for implementing network security features, such as firewalls and intrusion detection systems. It can also be used to examine network traffic in real-time, detect suspicious patterns, and take appropriate actions to protect the network.

eBPF (Extended Berkeley Packet Filter)

The Extended Berkeley Filter (eBPF) is an evolution of the original BPF technology. It extends the capabilities of BPF by providing a more powerful and flexible way to perform dynamic tracing, network analysis, and performance monitoring. It allows developers to write and load programs into the kernel which can be attached to various hooks and events in the system. These programs can provide real-time insights and control over system activities.

Working on an eBPF program

The process of compiling and running an eBPF program involves several steps:

The eBPF program is converted into bytecode by using a compiler, ready to be loaded by a loader program.
The eBPF verifier checks the program for safety, correctness and adherence to specific rules and constraints. First of all, it performs a depth-first search on all possible execution paths to ensure that the program always proceeds to completion. Next, it performs a static analysis on the bytecode and ensures that the program doesn't violate memory access rules, and doesn't cause instability.
Once the eBPF program passes the verification process, it can be loaded into the kernel. The loader ensures that the program is securely loaded and attached to the desired hooks or targets in the system.
At runtime, the eBPF bytecode is further optimised through JIT (Just-in-time) compilation. This step converts the eBPF bytecode into machine code that can be executed by the CPU.

Kernel Modules

Apart from eBPF, the other approach we previously discussed for process tracing in Linux is the use of kernel modules. Kernel modules allow developers to write custom code that can be loaded into the kernel to extend its functionality.

By leveraging kernel modules, it is possible to hook into various points of the kernel's process management code and capture detailed information about process execution. This includes events such as process creation, termination, and context switches.

By accessing the kernel's internal data structures and functions, the module can gather valuable insights such as process IDs, parent-child relationships, execution time, system calls, and more.

So why does Falco use eBPF?

The integration of eBPF brings significant advantages to projects like Falco, empowering them to securely and efficiently monitor and analyze system calls in real-time. You might be wondering why eBPF is necessary when Falco already has real-time detection capabilities through its kprobe (kernel probe) that handles syscall events.

One compelling reason for incorporating eBPF support is to enable Falco to seamlessly operate in modern cloud native environments, where the traditional kernel probe may encounter limitations or face restrictions imposed by the control plane nodes.

By embracing eBPF, Falco ensures the continuity of its real-time detection capabilities in a secure manner, allowing for the prompt and accurate identification of security incidents, regardless of the underlying environment.

Later in the article, we will delve into the various considerations surrounding the adoption of an eBPF probe for Falco, providing valuable insights for determining when it becomes advantageous to leverage this functionality.

eBPF programs vs kernel modules

Safety and Isolation

eBPF programs are subjected to a thorough verification process before they are loaded into the kernel. This step provides an extra layer of protection and helps prevent security vulnerabilities. In contrast, kernel modules have direct access to the kernel code, which can pose a threat to the system if not implemented correctly.

Performance

eBPF programs are JIT compiled into machine code, which significantly improves the performance. JIT compilation optimizes the program for the specific CPU architecture, enabling efficient execution. Despite all these efforts, an eBPF instrumentation will always cause a greater overhead in the system than a kernel module one, the reason is that in the kernel module instrumentation there are no calls to the BPF subsystem.

Observability and Debugging

eBPF provides powerful tracing and observability capabilities. eBPF programs can be attached to various events, such as network packets, system calls, or kernel functions, allowing detailed visibility into the system behaviour. This makes eBPF a valuable tool for debugging, performance analysis, and security monitoring. Kernel modules typically require more invasive and complex mechanisms for achieving similar observability.

Attaching eBPF programs to hooks and events

There are various instrumentation points defined in the Linux kernel. An instrumentation is a specific point in a computer program where additional code, known as instrumentation code, is inserted to gather information about the program's execution. Instrumentation code can be injected at runtime using JIT compilation. Kernel probes, tracepoints, user-space probes, kretprobes are examples of instrumentation points.

Here is an eBPF program that runs when the execve system call is made.

In the eBPF programming context, the macro SEC() from the bpf/bpf_helper.h header file plays a crucial role. It allows the programmer to specify the section in which a function or variable will be placed within the eBPF object file. This becomes essential when loading eBPF programs into the kernel using mechanisms like the bpf() system call.
By organizing functions and variables into named sections, the eBPF loader can efficiently locate and load the required code and data. Specifically, when dealing with tracepoint events, the SEC format follows the pattern SEC("tp/<category>/<name>"), where <category> and <name> represent the respective tracepoint category and event name.
tp/syscalls/sys_enter_execve refers to a tracepoint that records when a process spawns the execve system call.
A list of all the available tracepoints is present in the /sys/kernel/debug/tracing/available_events file. The format for each line in the file is <category>:<name>. For example, syscalls:sys_enter_execve.

Before compiling the program, we need to do some basic configuration:

Let's compile the program. The following command can be used to do this task:

Now, we need to write a loader program that loads and attaches this program. This loader program is used to load and attach an eBPF program to the Linux kernel. It opens and loads the eBPF object file, checks for errors during the process, finds a specific eBPF program within the loaded object, and attaches it to the kernel. Once attached, the eBPF program will be executed when certain events occur. The program enters an infinite loop at the end, indicating that it will continue running until it is manually terminated.

Let's compile and run this program

To get the logs generated by the function bpf_printk, we can read the file: /sys/kernel/tracing/trace_pipe

Manually reading messages from the tracepipe doesn't seem to be an efficient approach. It would be advantageous to establish a mechanism for the eBPF program to send messages to the loader program. One viable solution is to utilize ring buffers. Let’s review more details about ring buffers.

Ring buffers

eBPF ring buffer, also known as bpf_ringbuf, is a mechanism provided by the Linux kernel for efficient communication between eBPF programs and user-space programs.

It allows the exchange of data and events between eBPF programs running in the kernel and user-space applications. It is a MPSC (Multi Producer Single Consumer) queue and can be safely shared across multiple CPUs simultaneously.

The eBPF ring buffer, being shared across all CPUs, offers a unified and efficient solution for managing memory utilisation, mitigating issues of overuse or under-allocation that commonly occur with perfbuf.

Let's have a look at a few functions that we'll be using to write an eBPF program that sends data to userspace.

bpf_ringbuf_reserve

This function is used to reserve size bytes of space in a BPF ring buffer.

bpf_probe_read_user_str

This function is used to read a null terminated string from user-space memory into the destination dst. The dst parameter is a pointer to the destination buffer in the kernel space. unsafe_ptr is a pointer to the source string in the user-space.

bpf_ringbuf_submit

This function is used to submit data that had previously been reserved in a ringbuf.

bpf_object__find_map_fd_by_name

This function is used to find the file descriptor of a named map.

bpf_program__attach_tracepoint

This function is used to attach an eBPF program to a kernel tracepoint.

ring_buffer__new

This function is used for creating and opening a new ringbuf manager.

ring_buf__consume

Used to remove or consume data from a ring buffer.

BTF (BPF Type Format)

It provides a way to describe the types of data structures used by eBPF programs, allowing for improved type safety, debugging, and introspection.

Now, let's write a program that sends data to userspace using ringbuf.

Having created the program, we can write a loader to load this eBPF program.

The infinite loop is necessary to ensure that the program continuously checks for new events in the ring buffer. Without the loop, the program would only consume events that were already in the buffer at the time of the initial ring_buffer__consume() call. By looping and calling ring_buffer__consume() repeatedly, the program can retrieve events as soon as they become available and process them in real-time. The sleep(1) call within the loop serves to reduce the CPU usage of the program by introducing a one-second delay between each call to ring_buffer__consume().

Great, we were able to recover the process name as well as the PID!

Conclusion

In conclusion, this article has provided a comprehensive overview of eBPF (extended Berkeley Packet Filter) and its significance in tracing system calls. We have explored the evolution from BPF to eBPF, discussed why Falco uses this technology, and delved into the process of working with eBPF programs and ring buffers for efficient data communication between the kernel and user-space applications.

As we journeyed through the capabilities of eBPF in this first part, we uncovered its benefits in terms of safety, performance, and observability when compared to traditional kernel modules. eBPF empowers us to securely and efficiently monitor and analyze system calls in real-time, making it a valuable tool in modern cloud native environments.

In the upcoming second part of this blog series, we will further expand our exploration by delving into the realm of probes and additional advanced topics. We will dive deeper into how eBPF probes can be leveraged for enhanced system tracing, performance analysis, and security monitoring. Stay tuned for more insights and practical guidance on harnessing the power of eBPF.

Keep an eye out for Part 2, where we'll continue our journey into the world of eBPF and system call tracing.

Blog: Crafting Falco Rules With MITRE ATT&CK

Sun, 16 Jul 2023 00:00:00 +0000

Introduction:

The landscape of cybersecurity attacks has witnessed a notable rise in sophistication and complexity over the last decade, posing significant challenges to organizations in their efforts to identify and counter such threats effectively. It was within this context that the MITRE ATT&CK® Framework emerged as a valuable resource for security practitioners. In this blog, we will explore the benefits of using ATT&CK as a baseline to comprehensively understand threats, and Falco to detect and respond to these threats.

The ATT&CK Framework serves as an extensive repository of documented tactics, techniques, and procedures (TTPs) commonly employed by cyber adversaries. By gaining a comprehensive understanding of these TTPs, organizations can enhance their defensive capabilities and fortify their cybersecurity posture.

Falco is a valuable open source tool that provides runtime security for containers, virtual machines, and standalone Linux hosts. Organizations use Falco to monitor, detect, identify, and respond to suspicious activity. Falco detects suspicious activities and alerts security teams in real-time based on static rules provided in the rules file.

Whether you are a security analyst, a DevOps engineer, or an avid container enthusiast, this blog offers invaluable insights on utilizing MITRE ATT&CK-focused Falco rules to bolster your environment against advanced adversarial attacks.

Step 1: Gather Necessary Details

Consider the attacker’s perspective

To create a rule that identifies a specific ATT&CK technique, you need to get inside the mind of the attacker. Imagine watching an attacker and thinking about how they might try to harm or take advantage of a victim’s environment. There are an endless number of things to consider, such as: changes to important files or folders; collection of user and network information; and the creation and upload of malicious scripts. Conducting an in-depth assessment of MITRE’s ATT&CK TTPs will provide you with enough information to understand an attacker’s perspective.

Know where to look

In order to detect malicious activities using static rules, Falco relies heavily on system events (syscalls) generated within the user’s environment. To effectively observe these system behaviors, we must carefully consider the relevant system calls that occur during an attack. If network activity is involved, we must pay attention to the corresponding network traffic and alert on known malicious IP addresses. Additionally, we need to monitor the events that occur for files and directories during an attack. Armed with this knowledge, we can proceed to write rules that detect malicious activities as they unfold, ultimately triggering relevant events for further analysis and response.

Bring in Falco

Falco uses a rule-based system to monitor application and container behavior in real-time. With predefined rules, Falco detects security threats like privilege escalation, file system manipulation, abnormal process execution, and many more. It continuously compares system activities against these rules, and either generates alerts or takes action when a match occurs. Since Falco is open source, its flexibility allows customization of rules to fit an organization’s specific security requirements. By integrating with container orchestration platforms, Falco collects data from various sources and applies the rules in real-time, enabling proactive threat detection and prevention for cloud-native applications.

It is important to note that Falco will not identify a type of attack or malware. Rather, its strength lies in efficiently detecting common malicious system behaviors. Falco acts as a notifier, bringing your attention to specific system activities that have occurred. Once alerted, it becomes your responsibility to investigate the activity and take the appropriate steps to mitigate and prevent further malicious activities.

Step 2: Write the Falco Rule

Rule writing is an iterative process that typically begins with crafting a basic rule, as best you can, and gradually refining its conditions to be more specific. It entails continually incorporating exceptions into the rule to prevent false positives, ensuring that events are not erroneously flagged as malicious. This process evolves over time as we identify activities related to an attack and incorporate them into the rule. Exceptions are crucial as they allow for the inclusion of benign activities that may be associated with malicious behavior. Avoiding false positives is vital, as it reduces noise by ensuring a rule does not incorrectly flag benign system changes or events as malicious activities.

Identify the MITRE technique

Building upon our theoretical understanding of rule creation, we will now focus on the specific MITRE ATT&CK technique of Inhibit System Service and develop a Falco rule that can effectively identify this technique.

This technique is related to the recovery service of a system. Every system provides recovery services which are used in case of system failure. To make the system inaccessible an adversary may damage the system in such a way that the system becomes inaccessible to the user.

However, operating systems, including Linux, provide built-in mechanisms to assist administrators in recovering from such situations. These mechanisms include features like backup catalogs, volume shadow copies, and automatic repair functionalities. These tools help administrators restore the system to a functional state by reverting to previous configurations or repairing damaged components.

To impede system recovery efforts, attackers may specifically target and disable these built-in mechanisms. They can employ various native Linux utilities to accomplish this task. For instance, they might use commands like "rm" and "systemctl" to delete important system files, or employ tools like "dd" to overwrite the hard drive with random data. By doing so, they hinder the ability of system administrators to leverage these recovery mechanisms effectively.

Understand the technical details

The following points provide an overview of the recovery features present in operating systems and the adversarial strategies employed to compromise them:

Recovery feature for creating archives of data. These recovery features are given to store data on our system in the compressed form so that it can be used when it's needed by uncompressing it. Following are the tools which help us achieve this activity.
- tar: create a backup of the /home directory: tar -czvf home_backup.tar.gz /home
- rsync: synchronize files and directories: rsync -av /src /dest
- dd: create a disk image backup: dd if=/dev/sda of=/mnt/backup.img Adversary actions to make the data inaccessible to users. Now, an adversary may try to completely remove these archives by making all the data inaccessible.
- Delete backup files: rm /path/to/backupfile.tar.gz
- Overwrite backup data: dd if=/dev/zero of=/path/to/backupfile.img
Recovery feature for creating snapshots of file systems. Creating snapshots of a filesystem is a common practice in data management and backup strategies. Snapshots are essentially point-in-time copies or representations of a filesystem, capturing its state at a specific moment. Following are some snapshot creation techniques.
- Btrfs: create a snapshot of the / filesystem: btrfs subvolume snapshot / /snapshot
- ZFS: create a snapshot of a dataset: zfs snapshot pool/dataset@snapshot Adversary actions to make file system broken: To remove the backup of filesystem an adversary might do following activities. In which it contains deleting the snapshots , modifying the snapshots to make it unusable.
- Delete snapshots: btrfs subvolume delete /path/to/snapshot
- Modify snapshots: vi /path/to/snapshot
Boot Loader settings for banning of disabling recovery features The ability to ban or disable recovery features. The reason such settings exist is to provide system administrators or advanced users with the ability to control and secure the boot process according to their specific requirements.
- GRUB: modify the /etc/default/grub file to disable recovery mode
- GRUB_DISABLE_RECOVERY=true Adversary actions to try disabling all recovery features by modifying GRUB menu. An adversary may try to edit the grub configuration such that it disables the banning of certain recovery features.
- Modify GRUB configuration: vi /etc/default/grub
Systemd services for system recovery In a Linux system, systemd is a widely used init system and service manager that plays a crucial role in managing the overall system and its services. It includes various features and components to ensure system recovery and maintain system availability. Here are the few recovery services present in the linux operating system.
- Enter rescue mode: add systemd.unit=rescue.target to the end of the linux line in the GRUB configuration file
- Switch to a different root filesystem: use the systemd-nspawn command Also, there are other important services like:
- recovery-mode.service
- emergency.service
- rescue.service
- apport.serviceAA Adversaries will try to disable the above system recovery services.

Considering all these technical aspects and system changes that occur in a system when an adversary tries to inhibit system recovery, we can write a falco rule to identify these system events.

Follow Falco guidelines

The official Falco documentation provides all the rule fields, priorities, event, filenames, directory names, and covered syscalls that can be used in falco.

Decide on a Rule Name: We need to first decide name of our rule, since we wrote a rule specific to an ATT&CK technique, let’s keep it similar:
- rule: Disable recovery features.
Write a description: Define what is the purpose and intention for the rule that you intend to trigger. In our case, this can point to the definition of the ATT&CK technique we are capturing.
- desc: Detects disabling system recovery features by deleting or disabling services and commands.
Define a Conditional Set: To trigger an event, we need to define conditions that encompass various system activities such as relevant system calls, file or directory modifications, involved commands, spawned process names, and connections to prohibited IP addresses. These technical details should come from the attack analysis done prior to rule writing.

It is essential to adhere to the specified Falco format while crafting these conditions. Additionally, we must include exceptions to prevent false positives. Now, let us proceed to outline some fundamental conditions that will activate the desired event. - condition: > (spawned_process and proc.name = "rm" and proc.args contains "-rf") or (spawned_process and proc.name = "chattr" and proc.args contains "+i") or (spawned_process and proc.name = "mkfs.ext4" and proc.args contains "-E nodiscard") or (spawned_process and proc.name = "mount" and proc.args contains "remount,ro") or (spawned_process and proc.name = "systemctl" and proc.args contains "disable systemd-backlight@.service") or (spawned_process and proc.name = "systemctl" and proc.args contains "disable apport.service") or (spawned_process and proc.name = "systemctl" and proc.args contains "disable rescue.service") or (spawned_process and proc.name = "systemctl" and proc.args contains "disable emergency.service") or (spawned_process and proc.name = "systemctl" and proc.args contains "disable recovery-mode.service")

In order to accommodate system administrators with the appropriate privileges to perform different tasks, such as the root user, we can introduce an exception in our conditions (e.g., user.name != "root").

Likewise, we can incorporate exceptions for specific processes that engage in these activities for legitimate reasons. These exceptions help fine-tune our conditions and reduce the occurrence of false positives. 4. Create an enabled field: This way you can optionally disable the rule as per your requirements. If this field is not set, the rule is automatically enabled by default. - enabled: true/false 5. Write an output field: This is the text that Falco sends you when alerting on a suspicious activity. Here we can use all fields that can be used in rule conditions to give output in a more descriptive way like process name, username, container name or id etc. - output: "Disabling recovery features so that system becomes non recoverable in case of failure." 6. Specify a priority: The Falco team explains the concept of priorities within rules on their official documentation. It is important to clarify that the event is not triggered based on its assigned priority.

Rather, the event is triggered by specifying the priority level of a rule, which indicates the urgency in addressing it. Considering the paramount importance of this rule, we will assign it the highest priority level, known as CRITICAL: - priority: CRITICAL 7. Add Appropriate Tagging: In the final step, we will include tags for the rule. Tags serve as metadata providing additional information about the rules, although they are not mandatory fields. - tags:[mitre_impact, inhibit_system_recovery, T1490] When we bring all these elements together, our rule takes the following form:

The team at CloudDefense.AI developed a Python Script that simulates an attack scenario by emulating system changes. 'Link To The Script'. To observe the corresponding output in the falco logs, you can try incorporating this rule into your falco_rules.local.yaml file and running the script.

It's important to note that certain system calls are not instantiated by default in alco, so you will need to execute Falco with all syscalls enabled. This can be achieved by running falco -A.

Conclusion

To wrap up, we have delved into the pivotal role of Falco rules in securing our environments against ATT&CK-aligned attacks. By leveraging the synergies between the MITRE ATT&CK Framework and Falco's capabilities, we can greatly enhance our ability to detect and respond to potential threats.

Throughout this blog, we have learned how to closely examine a MITRE ATT&CK technique and write a corresponding Falco rule to aid us and other Falco users in identifying suspicious activities. Equipped with this knowledge, we can better recognize and mitigate potential security threats.

However, it is crucial to reiterate that this is an ongoing process and not a one-time solution. We have emphasized the significance of continuously refining our Falco rules to minimize false alarms and improve accuracy. By remaining vigilant and staying abreast of the latest attack vectors, we can proactively stay ahead of the curve and effectively protect our environments.

Ultimately, this blog has provided invaluable insights for security professionals and DevOps teams striving to bolster their defenses. By embracing the MITRE ATT&CK Framework and implementing targeted Falco rules, we can actively detect and respond to threats as they emerge, ensuring a robust shield against attackers.

Blog: Adaptive Syscalls Selection in Falco

Tue, 04 Jul 2023 00:00:00 +0000

The release of Falco 0.35.0 is a significant milestone, introducing a groundbreaking feature: the ability to select which syscalls to monitor. This empowers users with granular control, optimizing system performance by reducing CPU load through selective syscall monitoring.

Why stop at just the one groundbreaking feature, such as selecting which syscalls to monitor? Previously, Falco was limited to monitoring a narrower set of syscalls, which was a drawback since its underlying libraries and kernel drivers were capable of monitoring a wider range of syscalls. We addressed this gap, and Falco now has the enhanced capability to monitor every syscall supported by its libraries. This milestone, allowing access to a notable range of syscalls, represents another significant advancement in threat detection.

Kudos to the remarkable teamwork of Jason Dellaluce, Federico Di Pierro, Andrea Terzolo, and Melissa Kilby for making the adaptive syscalls selection feature a reality. We would also like to express our gratitude to Stanley Chan for providing invaluable feedback to ensure a clear and user-friendly experience.

Key Terms

First, let's define key terms that are crucial for understanding the complexity and high-level nature of the refactoring that has been performed.

syscall: In Linux, system calls serve as the interface for requesting permission from the kernel to interact with hardware resources, such as accessing memory or reading files. These system calls are defined in the Linux headers, and each syscall is associated with a specific number, such as __NR_close. Falco, being designed to support multiple architectures internally, employs a mapping mechanism to track each system call using a custom invented code called PPM_SC_*. This mapping allows Falco's libraries to uniquely identify and handle each supported syscall in a uniform manner. (e.g. PPM_SC_CLOSE represents the close syscall).
event: Syscalls consist of an enter event and an exit event. This is why Falco introduces an additional mapping from the PPM_SC_* code to another enumeration called PPME_*, which is specific to Falco's libraries. This mapping is crucial for organizing the parsing process and ensuring a structured approach, especially because Falco not only handles syscall events but also deals with non-syscall events. For syscalls, specific codes are assigned to events like the enter event of the syscall (e.g. PPME_SYSCALL_CLOSE_E) and the exit event (e.g. PPME_SYSCALL_CLOSE_X). However, not every syscall has its own PPME_* code. Falco introduces the concept of generic syscalls, which brings the advantage of a shared schema and parsing mechanism for multiple syscalls using a generic extractor, optimizing efficiency by avoiding the need for separate parsers in those cases.

monitoring/tracing: Refers to passively observing and analyzing events within the Linux kernel by hooking into tracepoints and subsequently serving information up as structured Falco alerts. Falco's monitoring process does not influence or modify any syscalls.
processing/parsing: Extracting meaningful information from the events captured by Falco and converting them into a structured format. We extract data fields in the kernel and perform the necessary parsing in userspace.
filtering: Refers to stopping the processing / parsing or ignoring events. No modifications are made to the kernel during this process.
rules matching: Refers to evaluating an event in userspace against the Abstract Syntax Tree (AST) generated from a Falco rule.

Background

Before the 0.35.0 release, Falco would monitor a predefined set of commonly used syscalls in its kernel driver for threat detection, regardless of the specific rules being applied. However, this approach had limitations as it would monitor a large number of syscalls. In certain user configurations, Falco would needlessly monitor syscalls not relevant to the loaded rules, consuming system resources without effectively contributing to the intended purpose of threat detection.

You may ask yourself why Falco has been monitoring a predetermined set of commonly used syscalls until now. Falco relies on a set of syscalls to establish and maintain its state in userspace. For example, when a new process is spawned or a network connection is created, multiple syscalls are involved. Additionally, Falco maintains a process cache table in userspace, which requires tracking certain syscalls to ensure the accuracy and currency of the cache table. The process table is crucial for retrieving real-time process tree lineages and other functions.

Initially, tracing a predefined set of syscalls provided a solid foundation for Falco's functionality. However, with the growing computational workload on servers and systems, it became necessary to adopt a new and more efficient approach to optimize performance.

Adaptive Syscall Selection

Adaptive syscall selection is a new feature that adds the ability to select which syscalls to monitor. This empowers users with granular control, optimizing system performance by reducing CPU load through selective syscall monitoring. Adaptive syscall selection was added to Falco on version 0.35, and, by default, it doesn't change Falco behavior from a high-level view. In other words, by default, Falco continues to operate as usual.

The changes primarily impact the handling of syscall events and the selection of specific syscalls to be monitored and analyzed. The current changes only affect live syscall events. The sycall selection is now done as follows:

Falco determines a base set of syscalls to monitor. This can be either the default minimum set known as the "sinsp state set" (automatically determined by the underlying libsinsp library to maintain state consistency and stability), or a customized set of syscalls defined by the user via the new base_syscalls configuration (discussed later).
The final set of syscalls selected by Falco is determined as the union of two components: the base set of syscalls computed in the previous step, and the syscalls specified in the loaded rules.
If the -A flag is not enabled, performance-heavy syscalls, such as I/O-intensive syscalls, are excluded from the set of syscalls, and a warning is shown to the user.
Falco configures the kernel driver with the chosen set of syscalls and only monitors syscalls that match the selected syscalls on the kernel side.

Adaptive syscall selection does not apply to capture files and only affects the behavior of live capture.

New Configuration Options

As discussed above, Falco 0.35 allows users to define a customized base set of syscalls to monitor. This is done via the base_syscalls setting, which provides two configuration options:

The custom_set option enables users to define a custom list of syscalls to monitor in Falco in addition to the syscalls from each Falco rule. It supports both positive notation, where a syscall is specified to be activated, and negative notation, indicated by ! followed by the syscall name, to deactivate a syscall even if it is used in the ruleset. This flexibility allows users to have precise control over which syscalls are included or excluded in the sys_enter and sys_exit tracepoints, ensuring a tailored configuration that aligns with their specific requirements, use cases and cost budget.

To maintain a streamlined and efficient configuration, it is recommended to remove unwanted syscalls directly from the Falco rules instead of excluding them in the custom_set configuration. This approach ensures that the rules accurately reflect the desired behavior and reduces unnecessary complexity in the configuration.
When the repair option is set to true, it automatically adjusts the custom syscall set to ensure the accurate creation of its state engine, including necessary syscalls such as close or procexit. However, it is designed to be the most system resource-friendly by activating the least number of additional syscalls (outside of those enabled for enabled rules) as alternative to Falco's default libsinsp state engine enforcement. It dynamically selects necessary syscalls based on the active rules, making it truly adaptive and efficient. The repair_state feature helps mitigate issues that may arise from incorrect usage of custom_set.

Here are some helpful suggestions:

For process-related rules include syscalls such as clone, clone3, fork, vfork, execve, execveat, and close in the base_syscalls.custom_set; these syscalls are essential for retrieving process information and managing file descriptors.
For networking-related rules include syscalls like socket, bind, and getsockopt in the base_syscalls.custom_set; these syscalls ensure that network-related events are properly logged, including IP tuples and relevant information.
For tracking process information accurately consider adding syscalls such as setresuid, setsid, setuid, setgid, setpgid, setresgid, setsid, capset, chdir, chroot, prctl and fchdir; these syscalls help track the correct UID, GID, SID, and PGID etc of a process when it interacts with files or makes network connections.

The provided suggestions serve as a starting point for configuring the base_syscalls.custom_set according to users' specific monitoring needs. It is crucial to assess these suggestions within the context of individual use cases and make appropriate adjustments.

By adhering to these recommendations and carefully fine-tuning the syscall selection, users can achieve optimal performance, minimize resource utilization and possible kernel side event drops, and maintain precise monitoring and detection of potential security threats with Falco. Tailoring the syscall selection aligns Falco with the unique requirements of the environment, enhancing its effectiveness in threat detection.

For further information, please refer to the Falco configuration file and navigate to the new base_syscalls option: https://github.com/falcosecurity/falco/blob/master/falco.yaml

If all of this sounds confusing or unclear, the best way to gain a deeper understanding of this new feature is through hands-on experimentation. By actively exploring and testing the feature, you can observe firsthand how it can enhance the performance of your deployment.

Also, you can test it live in this interactive environment that we have prepared for you.

Click on it to start playing with it 🎮

If you wish to examine the final set of syscalls that will be used by Falco on your own envinronment, you can print them to the STDOUT by including the -o "log_level=debug" -o "log_stderr=true" --dry-run args during a dry run of Falco. By utilizing this option, you can gain valuable insights into the selected syscalls, aiding in troubleshooting and verifying the configuration.

❯ falco -o "log_level=debug" -o "log_stderr=true" --dry-run
Tue May 30 14:01:27 2023: Falco version: 0.35.0-alpha5 (x86_64)
Tue May 30 14:01:27 2023: CLI args: falco -o log_level=debug -o log_stderr=true --dry-run
Tue May 30 14:01:27 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Tue May 30 14:01:27 2023: Configured rules filenames:
Tue May 30 14:01:27 2023: /etc/falco/falco_rules.yaml
Tue May 30 14:01:27 2023: /etc/falco/falco_rules.local.yaml
Tue May 30 14:01:27 2023: /etc/falco/rules.d
Tue May 30 14:01:27 2023: Loading rules from file /etc/falco/falco_rules.yaml
Tue May 30 14:01:27 2023: Loading rules from file /etc/falco/falco_rules.local.yaml
Tue May 30 14:01:27 2023: Skipping daemonizing in dry-run
Tue May 30 14:01:27 2023: Skipping signal handlers creation in dry-run
Tue May 30 14:01:27 2023: Skipping daemonizing in dry-run
Tue May 30 14:01:27 2023: Setting metadata download max size to 100 MB
Tue May 30 14:01:27 2023: Setting metadata download chunk wait time to 1000 μs
Tue May 30 14:01:27 2023: Setting metadata download watch frequency to 1 seconds
Tue May 30 14:01:27 2023: Skipping clients initialization in dry-run
Tue May 30 14:01:27 2023: (32) syscalls in rules: accept, accept4, connect, creat, dup, dup2, dup3, execve, execveat, link, linkat, listen, mkdir, mkdirat, open, openat, openat2, ptrace, recvfrom, rename, renameat, renameat2, rmdir, sendmsg, sendto, setuid, socket, symlink, symlinkat, unlink, unlinkat, userfaultfd
Tue May 30 14:01:27 2023: +(40) syscalls (Falco's state engine set of syscalls): bind, capset, chdir, chroot, clone, clone3, close, epoll_create, epoll_create1, eventfd, eventfd2, fchdir, fcntl, fork, getsockopt, inotify_init, inotify_init1, io_uring_setup, mount, open_by_handle_at, pipe, pipe2, prctl, prlimit, procexit, recvmsg, setgid, setpgid, setresgid, setresuid, setrlimit, setsid, shutdown, signalfd, signalfd4, socketpair, timerfd_create, umount, umount2, vfork
Tue May 30 14:01:27 2023: (72) syscalls selected in total (final set): accept, accept4, bind, capset, chdir, chroot, clone, clone3, close, connect, creat, dup, dup2, dup3, epoll_create, epoll_create1, eventfd, eventfd2, execve, execveat, fchdir, fcntl, fork, getsockopt, inotify_init, inotify_init1, io_uring_setup, link, linkat, listen, mkdir, mkdirat, mount, open, open_by_handle_at, openat, openat2, pipe, pipe2, prctl, prlimit, procexit, ptrace, recvfrom, recvmsg, rename, renameat, renameat2, rmdir, sendmsg, sendto, setgid, setpgid, setresgid, setresuid, setrlimit, setsid, setuid, shutdown, signalfd, signalfd4, socket, socketpair, symlink, symlinkat, timerfd_create, umount, umount2, unlink, unlinkat, userfaultfd, vfork
Tue May 30 14:01:27 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Tue May 30 14:01:27 2023: Skipping starting webserver in dry-run
Tue May 30 14:01:27 2023: Skipping event processing in dry-run
Tue May 30 14:01:27 2023: Skipping unregistering signal handlers in dry-run
Tue May 30 14:01:27 2023: Skipping stopping webserver in dry-run

Refactors Involved

This section provides an overview of the underlying refactors that have enabled the implementation of the innovative adaptive syscall selection mechanisms in Falco.

Falco's ppm sc API

A new event set class has been introduced to support efficient set operations natively in the ppm sc API in Falco's libs.
Additional refinements in the ppm sc API have resulted in robust mechanisms to accurately map syscall and other event strings from the loaded Falco rules to the PPM_SC_* or PPME_* enumerations.
This enhancement was essential to overcome the challenges inherited from previous mapping processes and improve the efficiency and structure of event parsing.
The initial separate enumeration for kernel tracepoints has been merged with the PPM_SC_* codes (sc now reflecting scap codes instead of syscall codes only), resulting in a single enumeration.

This consolidation sets the groundwork for integrating future LSM (Linux Security Modules) hooks into Falco. By combining these codes, Falco achieves a seamless integration of tracepoint activations and syscall event handling within a unified framework.

Syscall Event Type Extraction from Falco's Rules Expression Language

In order to extract the corresponding event types, Falco traverses the filter Abstract Syntax Tree (AST) of each rule. The traversal process has been improved for robustness and integrated with the modernized ppm sc API by moving it to the underlying Falco libs.
Additionally, traversals now include support for mapping the rules event strings to both the PPM_SC_* and PPME_* codes. These changes have allowed for the resolution of technical debt concerning the mapping of event strings to their respective syscall codes.
Just to emphasize this once more: Falco now has the capability to support any syscall that is supported by its underlying libraries. This expansion of support allows Falco to monitor and analyze a wider range of syscalls for threat detection purposes (see Falco's supported syscalls).

Driver Syscall Push Down Filters

After mapping the event strings from the rules to their corresponding syscall IDs, we utilize a dedicated eBPF map (in the case of *bpf* drivers) or an internal bitmask using the ioctl API (in the case of kernel module) to inject this information into the sys_enter and sys_exit tracepoints within the driver.
Due to the triggering of the sys_enter and sys_exit kernel tracepoints for every syscall, our pushdown filter is designed to efficiently ignore unnecessary syscalls before any data field extraction takes place in our kernel drivers. Once again, Falco operates as a passive monitor of syscalls and does not exert any influence or modify the behavior of the syscalls being monitored.
Furthermore, the objective of kernel-side filtering is to minimize the number of events that need to be transferred to userspace via the buffer between kernel and userspace, as well as reduce the number of events that are processed and evaluated against Falco rules in userspace.
This filtering allows us to achieve these efficiencies without sacrificing visibility, as the ignored syscalls are not utilized in Falco rules.

Spoiler alert: Imagine a monitoring experience where Falco adapts in real-time, intelligently adjusting its capabilities as needed. The ppm sc API already allows you to dynamically enable or disable syscalls and tracepoints at runtime.
This opens up exciting possibilities for the future of Falco. One day, we envision a truly adaptive monitoring system where Falco can supervise itself and automatically adjust the level of logging verbosity on the fly based on the system's needs.

Userspace libsinsp State Engine

By efficiently extracting syscall codes from Falco rules strings, we no longer need a large hard-coded predetermined set of syscalls.
As mentioned previously, the remaining task involved enabling all the syscalls required for the libsinsp state that were not explicitly included in the Falco rules. This is where the new base_syscalls configuration comes into play, providing end users with complete control over this process.
With the base_syscalls configuration, users can define and activate the necessary syscalls for the libsinsp state, ensuring comprehensive monitoring and threat detection capabilities.

Action Items and Recommendations for Adopters

Review the updated falco.yaml file for performance tuning configurations.
Assess if performance optimizations are needed based on your threat model and budget. Adjust the base_syscalls configuration accordingly.
Gradually tailor the base_syscalls configuration as needed, aiming to optimize resource utilization without compromising threat detection, if feasible.
Experiment and measure changes in resource usage by utilizing the native Falco metrics option.
Enhance Falco rules with specific syscalls for increased robustness [see note 1].
Official syscall string names in rules are now required.
Explore new threat detection approaches with Falco's expanded syscall support.
Stay updated on future releases for enhanced kernel monitoring capabilities.

One concrete example is the rule named "Linux Kernel Module Injection Detected." Previously, this rule relied on monitoring spawned processes. However, with the broader syscall coverage, it is now possible to enhance the rule by focusing on specific syscalls such as init_module and finit_module, which are directly related to kernel module injection.

Blog: Falco 0.35.1

Thu, 29 Jun 2023 00:00:00 +0000

Today we announce the release of Falco 0.35.1 🦅!

Novelties 🆕 and Fixes

Here is a tiny patch release! It addresses some small bugs that will not bother us and our users anymore:

Bug fix in the plugin framework, we can now associate a thread ID also to async events so that we can access related juicy information when writing rules! We suggest updating to this version to be able to use all the new capabilities that the new Plugin API has to offer!
Modern BPF can now be used in least privileged mode without any trouble in COS
Driver loader now correctly parses the kernel version of Ubuntu’s kernel flavors, and also supports Debian rt and cloud
Solved a rule ordering problem on our default ruleset that caused some rules to be shadowed
Updated falcoctl to the latest version, which fixes a corner cases that cause the tool to freeze

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.35.1, you can install its packages following the process outlined in the docs:

Do you rather prefer using the container images? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

What's next 🔮

We are in the working to let new big things happen in Falco, stay tuned!

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Jason and Lorenzo

Blog: Falco 0.35.0

Wed, 07 Jun 2023 00:00:00 +0000

Dear Community, today we are delighted to announce the release of Falco 0.35.0!

A big thank you to all our contributors for helping get the latest release out, we are thrilled to share this release and its goodies with the community. To read a detailed account of the release, see v0.35.0 in the changelog.

During this release cycle, we had 90+ PRs on Falco and a grand total of 170+ PRs for libs 0.11.0 and 60+ for drivers 5.0.1. Thank you to our maintainers and contributors, as this would not happen without your support, dedication, and contribution!

What's new? TL;DR 🩳

In release v0.35.0, we focused on addressing the following key features:

Moving the modern eBPF probe out of experimental status
Improving Falco performance, allowing tailoring syscall detection to one's needs
New Falco metrics
Falco images signing
Improving plugins SDK
Test infra revamp

For more information check out the 0.35 overview video on YouTube

Modern eBPF probe 👨‍🚀

The new, modern eBPF probe was released as experimental during the 0.34.0 release cycle. Since then we worked hard to implement all the remaining syscalls and behaviors, and now the same eBPF probe has left experimental status.

The new eBPF probe is a CO-RE probe, which means it is already built into Falco, and you don't need any downloads. Moreover, it sports better performance compared to the old eBPF probe.

Finally, while delivering the new eBPF probe, Andrea Terzolo also shipped a brand new driver testing framework, now used in libs CI to test consistency between all three drivers. This addition alone was worth the effort: on behalf of the whole community, thank you Andrea!

The new probe has stricter kernel release requirements: for more info, check out our blog post.

Improved Falco performance

Thanks to the collaborative effort from Melissa Kilby, Jason Dellaluce, Andrea Terzolo and Federico Di Pierro, we were able to completely revamp the way that Falco detects syscalls that needs to be captured. With the new adaptive syscalls feature, Falco will only enable syscalls that are needed to detect the ruleset it is being run with. It will also enable a bunch of syscalls that are needed for libsinsp internal state parsers, and that's it.

Consequently, the -A flag semantics have changed. By default, ie. without -A, heavy syscalls (like I/O ones) won't be captured, even if the ruleset ships with them, and a warning is shown to the user. Using -A will now allow Falco to capture even heavy syscalls, without showing a warning. A couple of new config keys are now available to further tailor Falco adaptive syscalls: a related blog post will be published soon, so stay tuned!

One of the neatest things about this work is that the huge libs refactor it required lays the groundwork for another highly requested feature: LSM and kprobes support.

Falco metrics

Thanks to yet another collaborative effort led by Melissa, Falco has a new experimental metrics feature. This introduces a redesigned stats/metrics system, emitted as monotonic counters at predefined intervals (Prometheus-like).

There are multiple options available: one can enable the output of these metrics as internal metric snapshot rule, allowing them to be emitted as outputs. Or you can choose to output metrics to a file, that is not rotated by Falco. Moreover, there are options to enable CPU and memory usage metrics, internal kernel event counters and libbpf stats.

This is a great first step to improve Falco resource observability!

Falco images signing

Starting from 0.35.0, all Falco images that you can deploy in your cluster are now signed with cosign 2.0 in keyless mode.
This means that you can always verify that the Falco image you downloaded is an official Falco image, regardless of which registry you downloaded it from. Moreover, you don't have to install or explicitly trust any public key for it to work. This is the magic of cosign in action!

So, how do you verify our brand new images? Install cosign 2 and run:

cosign verify docker.io/falcosecurity/falco:0.35.0 \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com \
--certificate-identity-regexp=https://github.com/falcosecurity/falco/ \
--certificate-github-workflow-ref=refs/tags/0.35.0

Of course, you can do the same for all the deployable images including falco, falco-driver-loader, falco-no-driver and falcoctl (see its repo for more details).

This wouldn't have been possible without a big effort from Luca Guerra and Federico Di Pierro to migrate our entire release pipeline from CircleCI to GitHub Actions. The work is part of a larger effort from the Falco Supply Chain Working Group to bring all the Falco official artifacts up to date with the latest supply chain security standards. Special thanks to Massimiliano Giovagnoli, Batuhan Apaydın and Carlos Panato for your help and expertise in this area!

Plugins workstream

The Plugin API has seen quite a few big improvements, mainly from Jason.

The first big change is that the plugin framework is now totally compatible with all the events supported by the Falco libraries, including all system calls and kernel events. The plugin API now shares all the event definitions of libscap and allows plugins to both produce syscall events and extract fields from them. This feature has been in big demand since the first plugin system release (#410, #992), and opens the door to many new opportunities for Falco extensions.

Second, plugins now have a standard way for managing and maintaining internal state. Up until now, plugins were only able to extract fields from the information available in the payloads of each event, thus being stateless components by definition. Now, plugins have a defined protocol (#991) for hooking into the event stream, reconstructing an internal state, and using it for extracting fields for Falco rules. Also, plugins can inject asynchronous metadata events in open data streams to notify about state transitions and make them reproduceable when replaying capture files, just like has always happened with container-related events in the Falco libraries.

Lastly, plugins are now able to communicate bidirectionally with the Falco libraries and access their internal state, both in read and write modes. For example, this enables creating plugins that extract metadata fields from syscall event streams, and that have access to all the thread information reconstructed by libsinsp, with the opportunity of enriching it dynamically at runtime. The API surface also allows cross-plugin state access. We hope the developer community will appreciate the new power this offers plugin authors.

This big feature package required altering the plugin API in a way that is incompatible with the previous versions (the API major version has been bumped). As such, plugins released after Falco version 0.35 will not be compatible with Falco versions <= 0.34.1, and plugins released before version 0.35 will not be compatible with Falco from version 0.35 onwards. So, the action required for you is to remember to also update all your plugins to the latest versions when updating Falco to v0.35!

Test-infra revamp

Massimiliamo Giovagnoli and Samuele Cappellin have contributed tremendous work on improving our infra. Prow is now lighter, quicker and less issue-prone. Multiple prow jobs were moved to GitHub Actions to improve cluster efficiency; moreover, driver-building jobs are now much less frequently killed (basically never).

Also, arm64 drivers are now built on arm64 nodes, without using qemu, speeding up the build time. At the same time, resources allocated to the cluster were enlarged, with autoscaling limits now set to 20 ARM nodes and 20 x86 nodes. We can now deliver weekly new driver artifacts much quicker than before!

Finally, the cluster now exposes Grafana dashboards for monitoring purposes: https://monitoring.prow.falco.org/.

Try it out

It’s time to try out the new release! Here are some pointers for getting started with Falco:

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What next? 🔮

The community is active on many topics and we hope to deliver great features and many stability fixes once again during the next release cycle!

We will revisit and improve libsinsp API, for a more coherent developer experience.
Finally, the long-awaited LSM and kprobes will be implemented.
As the plugin API has seen huge improvements, we expect new plugins using the new features very soon.
Fixes, fixes and also fixes everywhere
Above all, we will work to improve thread tables and process trees inconsistencies; that's a huge topic and we plan to tackle it in multiple ways!

Stay tuned 🤗

You can find all the most up to date information at https://falco.org/community/.

See you for the next release!

Federico, Andrea and Lorenzo

Blog: Falco 0.34.1

Mon, 20 Feb 2023 00:00:00 +0000

Today we announce the release of Falco 0.34.1 🦅!

Novelties 🆕 and Fixes

Here's a minor update! This patch release addresses small but persistent issues that have been causing inconvenience for users:

http_output not working as expected when the remote endpoint was using the HTTPS protocol;
FALCO_ENGINE_VERSION was bumped since in Falco 0.34.0 new event fields were added for the process events;
cleanups and fixes related to memory management were introduced in libs;
avoid file descriptor leakage when checking for online CPUs in libpman.

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.34.1, you can install its packages following the process outlined in the docs:

Do you rather prefer using the container images? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

What's next 🔮

It's an exciting time for Falco as we see so many great improvements and features. What's more exciting is the fact that many great ideas and awesome work are going on to make the next big things happen.

The upcoming release will include complete syscall support in the modern BPF probe (feature parity with kernel module and current BPF probe) and introduce adaptive syscall selection for the Falco ruleset.

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Aldo

Blog: Falco 0.34.0 a.k.a. "The Honeybee 🍯"

Tue, 07 Feb 2023 00:00:00 +0000

Dear community, today we are delighted to announce the release of Falco 0.34.0 🎉!

A big thank you to the community for helping get the latest release out. The Falco community is thrilled about this release and cannot wait to share the latest goodies. Check out the newest features from this most recent launch to learn more ⬇️. To read a more detailed account of the release, see v0.34.0 in the changelog.

What’s new? 🆕

In this release we saw more than 190 pull requests across the repos of Falco and its libraries. Thank you to our maintainers and contributors, as this would not happen without your support, dedication, and contribution.

Updates - TL;DR 🩳

In release v0.34.0 the community focused on addressing the following key features:

Downloading and dynamically updating Falco rules at runtime
Shipping the brand new experimental modern eBPF probe
Designing more ways to catch suspicious executions

Automatic rules update 🔄

A few questions that often come up when using Falco is how can we update rules once Falco is installed in the cluster and, how do we get updated rules from the Falco organization without having to wait for the next release? This is the first release to include falcoctl with an out of the box solution to do exactly that!

When using the new Falco Helm Chart 3.0.0 rules will automatically be updated from the official repository. To learn more about this feature and how to configure it read the Helm chart documentation. Likewise, when using a SystemD based install you can configure Falco to automatically update rules.

Want to upgrade to the new Helm chart? Read all you need to know before you do so!

Modern eBPF probe 👨‍🚀

Last quarter, Andrea published the blog, “Getting started with modern BPF probe in Falco,” and announced that the new experimental eBPF probe had landed among us bringing with it a few key features: CO-RE paradigm, BPF Ring Buffer map, BTF-enabled program, BPF global variables, BPF skeleton, and finally Multi-arch support.

Why a new probe? 👽

The old probe supported old kernels (>=4.14) that can not take advantage of the new shiny eBPF features. While it would be great to have only one probe that works for every kernel version, recent features change (and simplify!) the way we write, maintain and deploy the code so deeply that a new fresh probe is the most reasonable solution. In order to leverage these recent eBPF improvements and use the new probe you will need a kernel version >= 5.8.

Modern eBPF in action 🎬

Try it now!

Shiny new eBPF features ✨

Why are Falco maintainers and community members excited about the modern eBPF probe? There are quite a few features that you might be interested in! Some of our favorites are:

CO-RE paradigm - stands for "Compile-once-run-everywhere", so as you may imagine, this paradigm allows compiling the eBPF probe just once for all kernels! You understood well: NO MORE MISSING DRIVERS, and no more painful local builds requiring the much-loved KERNEL HEADERS.
Multi-arch support - the modern BPF probe also supports multiple architectures by design. The actual targets for Falco are x86_64 and arm64 but new ones can be added at any time. If you have a project that needs BPF instrumentation for one of these architectures you could simply link the Falco libraries (libsinsp, libscap) to obtain a working solution out of the box.
Performance Improvements - the modern eBPF probe leverages features recently introduced in the Linux kernel such as BPF global variables and ring buffers to be faster and more efficient than the traditional eBPF probe!

Even more ways of catching suspicious executions 🕵️‍♀️

Detecting when a suspicious new executable is spawned is often considered a crucial baseline detection. Generally speaking, detecting this kind of behavior and understanding when it is malicious is not an easy task. For this reason Falco has not one, but several features that can help defenders craft appropriate rules for their workflows.

Thanks to great contributions from Lorenzo Susini and Melissa Kilby (thanks for both the code contributions and the image above!) we have two more ways to check for suspicious executions in our Falco rules as we have the following new fields tied to process spawn (execve) events:

proc.is_exe_upper_layer: which is true if the process’ executable is in the upper layer of the overlayfs. In practice, that means that the executable that is being launched has been introduced or modified in the container after it was started. While some applications might do this legitimately, in many cases it is a thing to watch out for because it might be signaling an attack in progress! Note that you can use this only on kernel versions greater or equal than 3.18.0, since overlayfs did not exist before then, and of course with container runtimes that make use of it as a union mount filesystem 😉. This flag complements proc.is_exe_writable, which is similar but only checks if the executable file is also writable by the same user that spawned it.

Don’t think this is enough? Do you think you need more flags to get more accurate detections? Here’s the second group of fields:

proc.exe_ino.ctime and proc.exe_ino.mtime: they show the last change time and modification time of the process’ executable file, respectively.
proc.exe_ino.ctime_duration_proc_start and proc.exe_ino.ctime_duration_pidns_start: demonstrates the time difference, in nanoseconds, between the process ctime and when the process was actually spawned or when the PID namespace was created, respectively. I’m sure you can see why you could be interested in that. Launching executables that were just created could be something that you want to know about 😁.

While the above signals won't replace the need to monitor file operation events, they can help reduce the search space for tracking spawned processes where for example chmod +x was run against the executable file on disk prior to execution (this causes ctime of inode to change, but we don't know if it was chmod related or a different status change operation). In addition, users could use these fields for selected rules to augment information available for incident response.

Artifact distribution 📜

Automatic rules updates and other upcoming features would not be possible without a proposal aimed to create a unified management of the distribution of artifacts. The overall goals for this are:

Allow users to consume artifacts in a consistent way
Define official artifacts
Unify distribution mechanism, infrastructure, and tooling
Provide generic guidelines applicable to any artifact that is distributed

The officially supported artifacts are a set of artifacts published by Falcosecurity and now are part of Falco and its ecosystem. Prior to release 0.34.0 the Falcosecurity organization distributed several kinds of artifacts in the form of files or container images, which included:

Installation packages
Helm charts
Drivers (eg, kmod, eBPF)
Rule files
Plugins
Other kinds may be added in the future.

Now, the new distribution channels include HTTP Distribution and OCI Distribution.

What we accomplished ✅

Falco rules have their own repo now 🏠

The benefits of having rules living in their repository are:

Dedicated versioning
Rules release will not be tied anymore to a Falco release (e.g., no need to wait for the scheduled Falco release to publish a new rule aiming to detect the latest published CVE)
Consistent installation and update mechanism for other rulesets (plugins rules are already published in their repository and can be consumed by falcoctl)
Rules are published as plain files as well as OCI artifacts at each release

Check it out: https://github.com/falcosecurity/rules

Falcoctl is official 😎

The falcoctl project was promoted to "Official" status, and its repository is now part of the core.

What's Next? 🔮

It’s time to try out the new release! Here are some pointers for getting started with Falco:

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

The community is active on many things and there is no shortage of great ideas for next releases!

Thanks to all the people who wrote and tried plugins, we have great feedback for the next version. If you are a plugin developer or user, stay tuned for more APIs and functionality!

The modern eBPF probe is awesome and we want to keep improving it to get it out of the experimental stage 🚀

Falco maintainers also care a lot about the project’s own security. We’re exploring security-related considerations in the Falco Supply Chain Security Working Group. Join us if you can't wait to know more about this.

Stay Tuned 🤗

Join us in our communication channels and in our weekly community calls! It’s always great to have new members in the community and we’re looking forward to having your feedback and hearing your ideas.

You can find all the most up to date information at https://falco.org/community/.

Till the next release! 👋

Luca, Andrea, Teryl and Jacque

Blog: Falco on AWS Cloud

Wed, 30 Nov 2022 00:00:00 +0000

It's Amazon Web Services' largest user conference this week, re:Invent, which is a good time to highlight the ways you can use Falco in the AWS Cloud for runtime security. In this article we'll review what's new, and take a look at installation, plugins, and integrations for AWS.

Support for Amazon Security Lake

We're pleased to announce that Falcosidekick will shortly be available with preview integration for Amazon Security Lake, a new service that optimizes and centralizes security data from cloud, on-premises, and custom sources into a purpose-built data lake.

Falcosidekick is designed to forward Falco events into other services: the new integration exports security events using the Open Cybersecurity Schema Framework (OCSF) format, an open industry standard, and sends them directly to Amazon Security Lake. This makes it easier to normalize and combine Falco events with other security data sources. You can check out the integration in the next version of Falcosidekick, 2.27.0.

Installation and drivers

You can find Falco and Falcosidekick as container images through the Amazon ECR Registry:

Falco image.
Falcosidekick image.

Additionally, the Falco project publishes pre-built driver modules for AWS kernels, whether you are using the kernel module driver or the eBPF probe. These can be fetched using falco-driver-loader.

Review the available drivers:

The prebuilt modules are available for both x86_64 and aarch64 architectures.

Plugins

Falco plugins let you use event sources other than kernel syscalls. Falco has two plugins specific to AWS.

The CloudTrail plugin can read AWS CloudTrail logs and emit events for each CloudTrail log entry. It includes out-of-the-box rules that can be used to identify potential threats in CloudTrail logs, including:
- Console logins that do not use multi-factor authentication.
- Disabling multi-factor authentication for users.
- Disabling encryption for S3 buckets.
Falco has extended its capability to read Kubernetes audit logs through a plugin for CloudWatch, where it can read the EKS audit logs. Read more about configuration and usage.

Falcosidekick integrations

Falcosidekick lets you forward events from Falco into a variety of different services, including many on AWS.

Cloudwatch Logs: emit events into a CloudWatch log stream.
S3: add events in JSON format to an S3 bucket.
Lambda: invoke a Lambda function in response to a Falco event.
SQS: send a message into an SQS queue.
SNS: create a push notification to apps or people.
Kinesis: send Falco events as streaming data.
Amazon Security Lake: add Falco events to a security data lake.

Conclusion

Falco offers a wide variety of support for runtime security on the AWS cloud. As we are an open source project, we welcome contributions and feedback! Read more about running Falco on AWS from this AWS Security blog post, Continuous runtime security monitoring with AWS Security Hub and Falco.

You can find us in the Falco community. Please feel free to reach out to us for any questions, suggestions, or even for a friendly chat!

If you would like to find out more about Falco:

Get started in Falco.org
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter

Blog: Falco 0.33.1

Thu, 24 Nov 2022 00:00:00 +0000

Today we announce the release of Falco 0.33.1 🦅!

Novelties 🆕 and Fixes 🐛

Here's a tiny patch release! It only fixes two bugs reported by the community:

CrashLoopBackOff in some cases when the gVisor integration is enabled on Kubernetes (reported on Minikube and some versions of GKE)
Crash when the eBPF probe is used and one or more CPUs are switched off. Thanks FedeDP!

Thanks to everyone who reported and worked on issues!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.33.1, you can install its packages following the process outlined in the docs:

Do you rather prefer using the container images? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

What's next 🔮

So many great things are happening in the Falco community right now. After meeting our friends at KubeCon NA, we're back at work with new features for the upcoming 0.34.0 release coming early 2023 with an unbelievable amount of work being done in the new eBPF probe, enhancements to falcoctl to make management of rules and plugins easier and much more!

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Enjoy 🎉

Luca

Blog: Falco 0.33.0 a.k.a. "the pumpkin release 🎃"

Wed, 19 Oct 2022 00:00:00 +0000

Dear community, today we are happy to announce the release of Falco 0.33.0 🎉!

A big thank you to the community for helping get the latest release over the finish line. The Falco community rallied behind this release and we wanted to share some of the latest novelties you’ll find in this most recent launch. To read a more detailed account of the release, check out v0.33.0 in the changelog.

What’s New? 🗞️

In this release we saw more than 160 pull requests across the repos of Falco and the libraries. We had a total of 20+ individual contributors. We’d like to give a special shout-out to Andrea Terzolo and Melissa Kilby for standing out as two of the most high-impact contributors for this release.

The project really seems to be more alive than ever! Thank you to our maintainers and contributors, as this would not happen without your support.

Updates - TLDR; 🚀

In release v0.33.0 the community focused on addressing the following updates & changes:

Libs now allow individual selection of which syscalls to collect during live captures, which helps Falco improve performance and reduce dropped events
Introduced the Kernel Crawler, a new tool that automatically identifies the most up to date kernel versions supported by popular distros
Syscall kernel ring-buffer size is now customizable for your environment needs
Mitigations for libsinsp’s Kubernetes metadata client to address recent issues that caused Falco to crash
Support for multiple simultaneous event sources, which means that you can now run multiple event sources in the same Falco instance
Added minikube as a supported platform in the driver loader and included it in our driver build matrix
Rule alert rate limiter is now optional and disabled at default
Support for two new syscalls and many improvements to the default Falco security ruleset

Selecting Interesting Syscalls ⚙️

A historical challenge when using Falco with a large system was to keep up with large amounts of kernel events. In the past, this was mitigated by what used to be called “simple consumer mode”, through which Falco discarded kernel events that were not useful for runtime security purposes. However, we lacked support for individually selecting which syscalls had to be collected and which to discard. This feature has been requested by the community for a while, as it is a great bonus point for both Falco and all other projects based on top of the Falco libraries. In this release, we refactored the whole system and introduced new libsinsp APIs that allow to individually select which syscalls and tracepoint events need to be instrumented for collection in the kernel. Now, Falco has higher control over collected security events, and is able to improve performance and reduce the amount of dropped events. At the same time, other projects can easily consume only the events they need without any additional instrumentation overhead.

Kernel Crawler 🔍

When deploying Falco, one of the biggest challenges has been to compile its drivers (kernel module or eBPF probe) for the specific kernel versions and customization you wish to instrument. To help our community, the Falco project has created prebuilt kernel modules and eBPF probes for widely-adopted distros and kernel versions. We have also provided a "driver loader" script that takes care of downloading and installing them before attempting local compilation. The build matrix has so far been constructed manually depending on the community demand and contributions, which makes it very hard to keep up with the most recent kernel versions.

Recently, the Kernel Crawler joined the Falco ecosystem as a tool that automatically searches for the most up to date kernels supported by multiple Linux distros (huge thanks to Federico Di Pierro for leading the effort). This helped us to dramatically expand our driver build matrix, and keeps it up to date with the latest kernel versions supported by the most popular distros without the need of manual intervention. This is a major step forward for Falco’s adoption, which we now expect to grow even further. Moreover, the Kernel Crawler populates an open database with all the information it collects. This is both a reference of the kernel versions and the distros supported by Falco, and a useful source of information for communities working in the space of kernel instrumentation like we couldn’t find on the internet so far.

Customizing the Syscall Kernel Ring-Buffer Size 💍

The ring-buffer is the shared piece of memory between Falco and the drivers in which all kernel events are pushed upon collection for Falco to consume them. When Falco is not able to keep up with the high throughput of events pushed, the buffer becomes full and some events are inevitably dropped.

Thanks to the great effort driven by Andrea Terzolo and Melissa Kilby, the syscall kernel ring-buffer size is now variable and configurable. In some cases, tuning this size may lead to better performance and less event drops on certain machines and environments. If you’re interested, check out the discussion at falcosecurity/libs#584.

Mitigations for Kubernetes Metadata Client ☸️

Starting from June’s Falco release, we included minor fixes for the Kubernetes client bundled inside libsinsp. This is the piece of code responsible for downloading metadata from your API server and populating fields in your security rules such as k8s.deployment.name, k8s.rc.name, etc. However, this causes Falco to receive too much data in certain situations, and to eventually crash. You can find more details in the following issue: falcosecurity/falco#1909.

Finding a stable and permanent solution is still being researched, as the problem of data overload has some intrinsic complexity. In this release, we introduced some short term solutions that prevent Falco from crashing in those scenarios by discarding useless information and handling errors gracefully. However, the big problem identified is that the Kubernetes cluster provides too much data, and we will keep looking for optimal solutions to this challenge in the future.

Running Multiple Simultaneous Event Sources 🚴

Wouldn’t it be nice if Falco could multi-task? Well, now it sorta can! We are delighted to announce that in this release Falco can now run multiple event sources in parallel. What does this mean? Well, it means that you can run plugins and syscall collections on the same Falco instance.

Historically, Falco supported consuming events from one source only. The only exception was the legacy support of the Kubernetes Audit Events, which allowed receiving those events and kernel events simultaneously. However, it was non-standard and has been substituted in favor of a plugin-based solution starting from Falco 0.32.0. Up until now, this meant that to consume events from more than one event source, users needed to deploy many instances of Falco, each configured with a different source.

This is a huge improvement and also brings back support for running syscall and k8s audit logs in the same Falco instance, for all the folks who were interested in doing so. For insights about the principles and rationale behind this release, follow the discussion at falcosecurity/falco#2074.

Please note that this feature introduces few user-facing changes to be aware of when updating. The primary one is that the syscall event sources will always be enabled by default if not explicitly disabled. So, please make sure you pass --disable-source=syscall to the Falco CLI if you’re interested in a plugin-only deployment! You can find more details in the documentation.

Supporting minikube in the Driver Loader 📥

We now offer new prebuilt drivers for the three most recent major version releases of minikube, which is a newly-supported platform for the Falco driver loader.

In general, it’s not possible to compile the Falco drivers locally when deploying on minikube, so in the past we needed to wait for a new minikube release to bundle the most recent Falco drivers. Thanks to the new Kernel Crawler, and great work carried out by Aldo Lacuku, our driver build grid now supports and auto-discovers the driver configurations for minikube and provides users with pre-built drivers to download with the driver loader. This reduces release delays to the bare minimum, and running Falco on minikube has never been easier!

Disabling Alert Rate Limiter at Default ❗

Falco provides a throttling mechanism for reducing the number of rule alerts, with the purpose of reducing noise in some environments. However, some users found concerns in this approach, as in the discussion at falcosecurity/falco#1333.

Falco v0.33.0 makes the rate limiter optional, and disables it in the default configuration, so that there is never a risk of discarding important alerts. At the same time, the feature is still present and configurable for everyone who needs to reduce Falco’s noise in their environment.

Updates on Syscall Coverage and Security Rules 🛡️

Call and you shall receive! Okay, that’s not exactly how that saying goes, but we acknowledged the importance of instrumentation coverage and critical updates to syscalls. After all, the power of Falco’s runtime security lies in the visibility it has over the system it gets deployed into. With this new release, Falco supports the collection of two new syscalls to ensure we keep those pesky hackers away: fsconfig and mlock2.

On top of that, there have been major updates to the default set of security rules bundled in Falco.

Since the last release, three new security rules have been added. Special thanks go to hi120ki for having been very active in maintaining the security rules over the past few months, and much of his work will be part of the next Falco releases as well. For v0.33.0, the new rules are:

Directory traversal monitored file read: detects attacks based on directory traversal
Modify Container Entrypoint: detects attacks based on CVE-2019-5736
Read environment variable from /proc files: detects attempts to read process environment variables

Additionally, existing rules have been updated to become less noisy and more optimized. Huge thanks to Melissa Kilby for taking the initiative to clean up the ruleset by disabling by default all the rules that were proved to never be triggered by Falco. This is a great step forward helping Falco be more performant by having fewer rules to evaluate at runtime.

What's Next? 🔮

It’s time to try out the new release! Here are some pointers for getting started with Falco:

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

But the party is far from being over! The community is preparing lots of exciting updates for the near future. Special mention goes to the modern eBPF probe work led by Andrea Terzolo, which is under active development and should be rolled out by the next Falco release! Moreover, there has been plenty of work on falcoctl, and we can expect a new release of the tool to come soon and bring plenty of exciting novelties in the ecosystem!

Stay Tuned 🤗

You can find all the most up to date information at https://falco.org/community/.

See ya! 👋

Jason and Jacque

Blog: Falco 0.32.2

Tue, 09 Aug 2022 00:00:00 +0000

Today we announce the release of Falco 0.32.2 🦅!

Novelties 🆕

This release is really small, like a little 🐦, it only fixes the URL to download the falco BPF probe from Falco download page. A big thank you goes to eric-engberg, who proposed the fix, and as usual to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Fixes 🐛

This release fixes just one bothersome bug:

The url from which Falco tryes to download the BPF probe was wrong, eric-engberg proposed the solution in this PR. Thank you again! 🙏

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.32.2, you can install its packages following the process outlined in the docs:

Do you rather prefer using the container images? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

What's next 🔮

It's an exciting time for Falco as we see so many great improvements and features. What's more exciting is the fact that there are many great ideas and awesome work going on to make the next big things happen.

Recently, there has been a lot of interest on the shiny new eBPF probe, making use of modern eBPF features like CO-RE, ringbuffer API and new tracing program.

In addition, many people in the community are interested in using Falco to read syscall events and plugin events simultaneously. If you are, I would suggest to take a look at the in-depth design for this new feature!

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Andrea

Blog: Manage Falco easier with Giant Swarm App Platform

Tue, 09 Aug 2022 00:00:00 +0000

In this article, you will learn how Giant Swarm simplifies the maintenance of the software stack within Kubernetes clusters by using its App Platform technology. Additionally, we will show how customers can leverage this to easily deploy Falco, either individually or as part of Giant Swarm's Security Pack, to secure their managed Kubernetes service.

Giant Swarm

Having CoreOS, Fleet, and Docker as base technologies, Giant Swarm was founded in 2014. In 2016, it chose Kubernetes to reinvent itself. And just a year later, in 2017, it became part of the founding members of the Kubernetes Certified Service Providers. Customers like Adidas or Vodafone backup a company that, supported by a fully remote team, has been able to foresee the trends of technology and working lifestyle.

As a managed Kubernetes company, its services and infrastructure enable enterprises to run resilient distributed systems at scale while removing the burden of Day 2 operations. Giant Swarm takes pride in delivering a fully open source platform that's carefully curated and opinionated.

Security and simplicity

Giant Swarm takes security as seriously as ease of management. Hence, when using a managed Kubernetes platform, everything that happens on the management cluster is as important as the performance of the workload cluster itself.

That's why, leveraging the concept of operators to control all resources that clusters need as 'Custom Resources', Giant Swarm can deploy and update its management clusters in the quickest possible way. Needless to say, this is exactly what Giant Swarm offers to its customers to manage their applications.

Falco, the Runtime Security Project

Falco is the de facto Kubernetes threat detection engine, and also extends its reach to cloud and Linux hosts. It monitors the behavior of every process in the node and can alert us when something fishy happens.

How does Falco do that? Based on a set of rules that Falco interprets at startup time, it waits for events and syscalls that would trigger one of those rules. When a rule is triggered, Falco raises an alert and, thanks to applications like Falco Sidekick, allows teams to react accordingly.

But with great power comes great responsibility. What happens when we start getting false positives our Falco rules haven't been updated for some months, or our Falco daemon is a few versions behind? The answer is as simple as updating. Well, maybe not that simple if we are responsible for tens of clusters with hundreds of nodes.

Giant Swarm App Platform

Giant Swarm describes App Platform as a set of features that allow you to browse, install, and manage the configurations of managed apps from a single place: The management cluster.

The technology behind it is simple: Apps are packaged as Helm charts, can be configured with values, overridden with a different app configuration, etc. - whatever meets your needs. To deploy, a CRD (Custom Resource Definition) resource is created, interpreted by the App Operator (running on the managed cluster), assigned to the Chart Operator (running on the workload cluster), and in a few seconds, our application will be deployed on as many clusters as desired.

The App Platform offers its repertoire of applications from the App Catalog. Giant Swarm offers two App Catalogs out of the box: The Giant Swarm Catalog and the Giant Swarm Playground. But what we love the most from the App Platform is that we can have our additional catalogs, storing our applications and configurations.

What does it look like on the CLI?

It's now time to look at App Platform running. Let's walk through its deployment on a minikube cluster. Following these instructions, it shouldn't take too long until we are ready to deploy our first managed app, Falco, using a single CRD.

To keep this as standard as possible, we'll even go through some steps to compile some interesting Giant Swarm tools, like the plugin kubectl-gs.

Do you already have a Kubernetes cluster nearby?

If not, we can spin up a minikube instance pretty quickly.

$ minikube start --driver virtualbox
😄 minikube v1.25.1 on Darwin 11.6.6
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
▪ kubelet.housekeeping-interval=5m
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

If you don't have kubectl installed or your system, the easiest way to access it would be through an alias to minikube kubectl, like this:

alias kubectl="minikube kubectl --"

Don't forget the -- at the end. That tells the command prompt not to pass any added parameters to minikube, since we need them to be understood by kubectl.

One disadvantage of this method, in comparison to having a local kubectl binary, is that the kubectl-gs plugin might not work when called as kubectl gs (explained later during this tutorial) so you might need to call it directly.

To ensure our cluster is up and running, execute the following command and verify that all nodes, pods, and containers are up and ready:

$ kubectl get nodes,ns,pods -A
NAME STATUS ROLES AGE VERSION
node/minikube Ready control-plane,master 4m16s v1.23.1
NAME STATUS AGE
namespace/default Active 4m14s
namespace/kube-node-lease Active 4m15s
namespace/kube-public Active 4m15s
namespace/kube-system Active 4m16s
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-64897985d-qbf4n 1/1 Running 0 4m
kube-system pod/etcd-minikube 1/1 Running 0 4m12s
kube-system pod/kube-apiserver-minikube 1/1 Running 0 4m12s
kube-system pod/kube-controller-manager-minikube 1/1 Running 0 4m12s
kube-system pod/kube-proxy-6ds89 1/1 Running 0 4m
kube-system pod/kube-scheduler-minikube 1/1 Running 0 4m14s
kube-system pod/storage-provisioner 1/1 Running 1 (3m29s ago) 4m10s

Prerequisites: Compiling `apptestctl` and `kubectl-gs`

As mentioned above, we'll compile a couple of tools. The first one will be apptestctl. This tool will help us bootstrap App Platform on a cluster not managed by Giant Swarm.

To do this, we'll use the docker.io/golang:1.17 image.

The following command will prepare an available instance of a Golang compiler for us to compile both of these tools:

$ kubectl run golang --image docker.io/golang:1.17 -- sleep infinity
pod/golang created

Compiling `apptestctl`

These steps are quite simple: clone the apptestctl repository and compile it as indicated.

We'll do this inside the container we created in the previous step so we don't pollute our system.

$ kubectl exec -it golang -- git clone https://github.com/giantswarm/apptestctl src/apptestctl
Cloning into 'apptestctl'...
... output omitted ...
Resolving deltas: 100% (791/791), done.
$ kubectl exec -it golang -- make -C src/apptestctl
make: Entering directory '/go/src/apptestctl'
... output omitted ...
====> apptestctl-v-linux-amd64
... output omitted ...
cp -a apptestctl-v-linux-amd64 apptestctl
====> build
make: Leaving directory '/go/src/apptestctl'

Alternatively, you can build a Darwin client.

$ kubectl exec -it golang -- make build-darwin -C src/apptestctl
make: Entering directory '/go/src/apptestctl'
... output omitted ...
====> apptestctl-v-darwin-amd64
... output omitted ...
cp -a apptestctl-v-darwin-amd64 apptestctl-darwin
====> build-darwin
make: Leaving directory '/go/src/apptestctl'

Either way, you can copy the apptestctl binary to your system and use it from wherever you prefer.

$ kubectl cp golang:/go/src/apptestctl/apptestctl-darwin ./apptestctl
$ kubectl chmod u+x ./apptestctl

Compiling `kubectl-gs`

Use the same steps to compile the kubectl-gs plugin this time, which will allow us to interact with App Platform. Pay attention to the fact that we'll compile it just for Darwin in this instance.

$ kubectl exec -it golang -- git clone https://github.com/giantswarm/kubectl-gs src/kubectl-gs
Cloning into 'kubectl-gs'...
... output omitted ...
Resolving deltas: 100% (4427/4427), done.
$ kubectl exec -it golang -- make build-darwin -C src/kubectl-gs
make: Entering directory '/go/src/kubectl-gs'
... output omitted ...
====> kubectl-gs-v-darwin-amd64
... output omitted ...
cp -a kubectl-gs-v-darwin-amd64 kubectl-gs-darwin
====> build-darwin
make: Leaving directory '/go/src/kubectl-gs'
$ kubectl cp golang:/go/src/kubectl-gs/kubectl-gs-darwin ./kubectl-gs
$ kubectl chmod u+x ./kubectl-gs

Deploying App Platform via `apptestctl`

Once we have both tools, apptestctl and kubectl-gs, it's time to bootstrap App Platform. To do that, we'll use the apptestctl bootstrap command.

The command apptestctl bootstrap needs the KUBECONFIG information to access our minikube cluster, so in this case, we will use the command kubectl config view --flatten --minify -o json to obtain it.

Alternatively, we would need to look for the .kube/config file and pass it with the --kubeconfig-path option.

$ ./apptestctl bootstrap --kubeconfig "$(kubectl config view --flatten --minify -o json)"
bootstrapping app platform components
... output omitted ...
app platform components are ready

Once deployed, we can run a few commands to observe the resources created in our cluster.

$ kubectl get deployments -n giantswarm
NAME READY UP-TO-DATE AVAILABLE AGE
app-operator 1/1 1 1 1m20s
chart-operator 1/1 1 1 1m20s
chartmuseum-chartmuseum 1/1 1 1 1m20s
# kubectl get catalog -A
NAMESPACE NAME CATALOG URL AGE
default chartmuseum http://chartmuseum-chartmuseum:8080/charts/ 1m25s
default helm-stable https://charts.helm.sh/stable/packages/ 1m25s

Wait a moment... Where does this Catalog resource come from? The bootstrap process of App Platform creates some CRDs that will support the operators to manage our applications.

$ kubectl get crd
NAME CREATED AT
appcatalogentries.application.giantswarm.io 2022-06-10T15:30:12Z
appcatalogs.application.giantswarm.io 2022-06-10T15:30:12Z
apps.application.giantswarm.io 2022-06-10T15:30:12Z
catalogs.application.giantswarm.io 2022-06-10T15:30:12Z
charts.application.giantswarm.io 2022-06-10T15:30:12Z

In short, once we register a Catalog, several AppCatalogEntries resources will be created. There will be at least one per application and version.

Registering a `Catalog`

Now, it looks like a great time to see what the kubectl-gs plugin can do for us.

$ kubectl-gs get catalogs
NAME NAMESPACE CATALOG URL AGE
chartmuseum default http://chartmuseum-chartmuseum:8080/charts/ 25m
helm-stable default https://charts.helm.sh/stable/packages/ 25m

All right, that was maybe not so impressive, but it'll become much more useful when we register our first catalog. Why is that? Because kubectl gs will help us generate the definition of a Catalog resource through its template subcommand.

$ kubectl-gs template catalog --name giantswarm --namespace default \
--description "Giant Swarm Catalog" --logo http://logo-url \
--url https://giantswarm.github.io/giantswarm-catalog
---
apiVersion: application.giantswarm.io/v1alpha1
kind: Catalog
metadata:
name: giantswarm
labels:
application.giantswarm.io/catalog-visibility: public
namespace: default
spec:
title: giantswarm
description: Giant Swarm Catalog
logoURL: http://logo-url
storage:
URL: https://giantswarm.github.io/giantswarm-catalog
type: helm

Et voilà, our Catalog CRD pointing to a Giant Swarm collection of applications is ready to be deployed into our cluster.

You might have figured out already what each parameter represents. kubectl gs will complain if any of those parameters are missing. Also, pay attention that we didn't use a real logo URL, but if you were using happa, the Giant Swarm Web-UI, would't you like to see a logo identifying your application?

Finally, the URL is the location of the Helm repository from which App Platform will download the applications.

Once we understand what the kubectl gs template command has generated, it's time to create it inside the cluster and let the App Operator do its magic. Let's go for it.

$ kubectl-gs template catalog --name giantswarm --namespace default \
--description "Giant Swarm Catalog" --logo http://logo-url \
--url https://giantswarm.github.io/giantswarm-catalog | kubectl apply -f -
catalog.application.giantswarm.io/giantswarm created
$ kubectl-gs get catalogs
NAME NAMESPACE CATALOG URL AGE
chartmuseum default http://chartmuseum-chartmuseum:8080/charts/ 35m
helm-stable default https://charts.helm.sh/stable/packages/ 35m
giantswarm default https://giantswarm.github.io/giantswarm-catalog 53s
$ kubectl gs get catalog giantswarm
CATALOG APP NAME VERSION UPSTREAM VERSION AGE DESCRIPTION
... output omitted ...
giantswarm falco-app 0.3.2 0.0.1 5m26s A Helm chart for falco
... output omitted ...

Do you remember the aforementioned AppCatalogEntries that the App Operator had to create once we defined the Catalog? Here are the Falco ones.

$ kubectl get AppCatalogEntries | grep falco-app
giantswarm-falco-app-0.1.2 giantswarm falco-app 0.1.2 0.0.1 240d
giantswarm-falco-app-0.2.0 giantswarm falco-app 0.2.0 0.0.1 176d
giantswarm-falco-app-0.3.0 giantswarm falco-app 0.3.0 0.0.1 103d
giantswarm-falco-app-0.3.1 giantswarm falco-app 0.3.1 0.0.1 94d
giantswarm-falco-app-0.3.2 giantswarm falco-app 0.3.2 0.0.1 79d

Installing an App from the App Catalog

What we've done so far was deploy App Platform, which is required only once. Giant Swarm would have configured that for us already if we were using their services.

Now, it's finally time to create the CRD that will trigger the App Operator to assist in the deployment of Falco. How do we do that? kubectl gs comes to the rescue again!

$ kubectl gs template app --catalog giantswarm --name falco-app --namespace falco-ns
--version 0.3.2 --app-name my-falco --in-cluster
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
name: my-falco
labels:
app-operator.giantswarm.io/version: 0.0.0
namespace: falco-ns
spec:
name: falco-app
version: 0.3.2
namespace: falco-ns
kubeConfig:
inCluster: true
catalog: giantswarm

It is worth mentioning that we are testing on a minikube cluster, where we install applications inside the cluster itself. To achieve that, we use the --in-cluster parameter passed to the previous commands.

Otherwise, if we wanted to install or update the application in one of our managed workload clusters, we would use the --cluster parameter to indicate where the application should be deployed:

$ kubectl gs template app --catalog giantswarm --name falco-app --namespace falco-ns \
--version 0.3.2 --cluster cluster-123 --app-name my-falco
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
name: my-falco
namespace: cluster-123
spec:
name: falco-app
version: 0.3.2
namespace: falco-ns
kubeConfig:
inCluster: false
catalog: giantswarm

In the previous output, you can see how the namespace field inside the metadata section receives the name of the cluster instead of the actual namespace where the application should reside.

The reason is that, although the application will be installed on one of the workload clusters, this CRD will be created in a namespace inside the management cluster. This topic alone would be enough for a whole new post.

Here is a graphical representation of the CRDs supporting App Platform, in the management cluster:

Finally, the last step is creating the CRD for the App in the cluster. Don't forget to ensure that the namespace where the CRD will belong exists, or the kubectl apply command will fail.

$ kubectl create ns falco-ns
namespace/falco-ns created
$ kubectl gs template app --catalog giantswarm --name falco-app --namespace falco-ns \
--version 0.3.2 --in-cluster --app-name my-falco | kubectl apply -f-
app.application.giantswarm.io/my-falco created
$ kubectl gs get app -n falco-ns
NAME VERSION LAST DEPLOYED STATUS NOTES
my-falco 0.3.2 113s deployed

Here are the resulting Kubernetes resources when using regular kubectl commands.

$ kubectl get app,deployment,daemonset -n falco-ns
NAME INSTALLED VERSION CREATED AT LAST DEPLOYED STATUS
app.application.giantswarm.io/my-falco 0.3.2 4m25s 4m24s deployed
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/my-falco-falcosidekick 2/2 2 2 4m24s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/my-falco 1 1 1 1 1 <none> 4m24s
daemonset.apps/my-falco-falco-exporter 1 1 1 1 1 <none> 4m24s

The previous output might differ depending on the type of cluster you would be using, among other variables.

As you can see, once App Platform is up and running, we only need to create the namespace that should contain the Falco application (which should already exist if we are deploying from a managed workload cluster), and the CRD based on the template from the kubectl gs plugin. In a matter of seconds, Falco will be up and running, watching for threats and alerting when suspicious behaviors arise.

Managed Security

Zach Stone, Platform Engineer at Giant Swarm, walked us through some of the biggest challenges that the company's customers face and how his team is using Falco to develop thoughtful solutions.

“The biggest problem that most of our customers face isn't what happens in the cluster, it's what happens with the information once they get it out of the cluster,” asserted Stone. “People also focus too much on the capability that a tool offers and don't take a bigger look at the security processes it supports.”

“If a customer has a vulnerability management program, we can track all of the vulnerabilities in their components, but if fixing those vulnerabilities isn’t a priority, then the program doesn’t work,” remarked Stone. "The larger discussion is usually about where the alerts should go, who bears responsibility for remediation, and how to fit that work into the team's limited capacity. We spend a lot of time trying to ensure security isn't just something that sits alongside the business, but rather is a meaningful part of the daily routine."

Part of that effort is in tuning detection rules and alerting. "Any time we surface an alert, it should be actionable and have a clear owner who is invested in never seeing that alert again."

“I think Falco's superpower is in the flexibility of the policies. I'm also really excited about the changes that are slated to make it easier to update them. Most rules aren't one-size-fits-all -- for a given policy, there is usually some refinement needed to ensure the policy makes sense within our platform, and then customers modify it even further to meet their security requirements. All that customization can make it incredibly difficult to reconcile,” said Stone. “The fact that we can already do it with Falco speaks volumes about the versatility of the solution.”

Security Pack

Giant Swarm's Security Pack is a collection of open source security tools offered by Giant Swarm, which not only contains Falco but also a plethora of other open source projects, including Kyverno for policy enforcement, Trivy for image scanning, and Cosign for image signature verification.

Security does not apply to a single level and, therefore, Security Pack consists of multiple applications, each one independently installable and configurable, available via their App Platform. “Falco will be the cornerstone of our node-level security capabilities,” affirmed Stone, “the biggest opportunity for API plug-ins I see is to get feedback from the node level back into the Security Pack so that we can further contextualize events in the ecosystem.”

Conclusion

Adding simplicity to our cluster management is considered a requirement nowadays, especially in those cases where the lack of resources in an organization can keep it from achieving an acceptable level of security.

Features like Giant Swarm's App Platform and Security Pack will help organizations to finally focus on what actually matters to them: Running their business. In the future, Giant Swarm plans to launch its security pack across all its customers' clusters, enabled by default and built on Falco.

Falco – Falco

Blog: Introducing Prempti: Falco meets AI coding agents

Agents are a black box at runtime

How Prempti works

Two modes: Monitor and Guardrails

Writing rules: Familiar territory

Rule authoring with Claude Code

Let's be honest about limitations

Getting started

Explore together with us

Blog: Falco at KubeCon Europe 2026 — See You in Amsterdam! 🐦

Sneak peek

Project lightning talk

Sysdig-led workshop

Conference talk

Booth demo

Thank you!

Blog: Hey Falco Flock! 🐦 Let's Soar Into 2026

👉 Take the survey here

Blog: Introducing Falco 0.43.0

What's new? TL;DR

Latest updates

Deprecations

Legacy eBPF probe deprecation

gVisor deprecation

gRPC output and server deprecation

GPG key rotation

Container plugin improvements

Falcoctl tweaks and improvements

follow polling interval increase to 1 week

Dependency resolution improvements

Support for cosign v3

Key fixes

evt.arg.filename field reintroduction

Falcoctl signature verification fixes

Signature verification fix for full reference artifacts

Signature verification fix for authenticated registries

Breaking changes and deprecations

Bump drivers minimum required kernel version to 3.10

Deprecation warnings

Try it out

Stay connected

Blog: GPG Key Rotation for Falco Packages (2026)

The Rotation Plan

Phase 1: Soft Launch (Dec 12, 2025)

Phase 2: Hard Cut-Over (Jan 12–17, 2026)

Action Items for Users

New Users

Existing Users

Summary

Blog: Introducing Falco 0.42.0

What's new? TL;DR

Major features and improvements

Capture recording feature

Drop enter initiative

Plugin event schema versioning

Thread table auto-purging configuration

Static fields

Breaking changes and deprecations ⚠️

Event direction and evt.dir deprecation

Plugin API changes

Deprecation warnings

Try it out

Stay connected

Blog: Introducing Falco 0.41.0

What’s new? TL;DR

Major features and improvements

Reimplemented container engines support

Kubernetes operator

Security best practices improvements

Breaking changes and deprecations ⚠️

Removed command line options and equivalent configuration options

Behavior changes

Dropped features

Deprecations

Try it out

What’s next?

Stay connected

Blog: Detecting Supply Chain Attacks with Falco Actions

What is Falco?

`follow` polling interval increase to 1 week

`evt.arg.filename` field reintroduction

Bump drivers minimum required kernel version to `3.10`

Event direction and `evt.dir` deprecation