Falco – The Falco blog

Blog: Introducing Prempti: Falco meets AI coding agents

Tue, 12 May 2026 00:00:00 +0000

Today's developer workflow is increasingly reliant on AI coding agents. Tools like Claude Code sit in your terminal, read your files, run shell commands, make network requests, and write code, all on your behalf. They are fast, capable, and increasingly trusted with real tasks on real machines.

But with that trust comes a question worth taking seriously: what exactly is your coding agent doing on your machine?

Today, we're introducing an experimental project that brings Falco to this new frontier: Prempti.

Agents are a black box at runtime

When a coding agent runs a bash command, writes a file, or reads a configuration, those actions happen inside your user session, with your permissions, in your filesystem, against your credentials. Most developers using these tools have no structured visibility into that activity. You see the agent's chat output, but you don't see what's happening under the hood.

Here's a simple scenario: you ask your coding agent to refactor a module. It reads your source files. It makes edits. Then, perhaps prompted by a malicious dependency or an unexpected instruction in a file it just parsed, it attempts to read ~/.ssh/known_hosts or write a file to ~/.aws/. Should it be allowed to? Would you even know if it tried?

The demo below captures exactly this situation:

The agent tried to both read and write to sections it's not allowed to, and both were blocked. The agent itself received a structured message explaining why, and showed that to the user. This is detection and enforcement working together at the tool-call level.

How Prempti works

Prempti runs as a lightweight user-space service alongside your coding agent. It does not require root, kernel modules, or containers. When your agent makes a tool call such as a file write, a shell command, or a file read, Prempti intercepts it before it executes, evaluates it against Falco rules, and delivers a verdict:

Verdict	What Happens
Allow	The tool call proceeds normally
Deny	The tool call is blocked and the agent is told why
Ask	You are prompted to approve or reject interactively

The architecture looks like this:

Prempti's hook fires before each tool call
An interceptor sends the event to Falco via a Unix socket
Falco's rule engine evaluates the event against your policies
Matching rules produce verdicts (deny / ask / allow)
The interceptor delivers the verdict back to the agent

Prempti uses Falco's plugin system to define a new event source (coding_agent) with fields purpose-built for this context: tool.name, tool.input_command, tool.file_path, agent.cwd, and so on.

Two modes: Monitor and Guardrails

Prempti is designed to let you both observe what the agent is doing and align it with your security policy:

Monitor mode evaluates every tool call against your rules and logs the results, but does not enforce any action. This is what we recommend as a starting point: run it for a few sessions, see what your agent actually touches, and tune your rules before you enable blocking.

Guardrails mode (the default) fully enforces verdicts as explained above — deny blocks, ask prompts you, allow proceeds.

You can switch between modes at any time:

premptictl mode monitor # observe only
premptictl mode guardrails # enforce verdicts
premptictl logs # watch live events

Writing rules: Familiar territory

If you've written Falco rules before, agent security policies will feel very familiar. Here's a rule that blocks piping content directly to a shell interpreter, a classic vector for prompt injection attacks:

- rule: Deny pipe to shell
 desc: Block piping content to shell interpreters
 condition: >
 tool.name = "Bash"
 and (tool.input_command contains "| sh"
 or tool.input_command contains "| bash"
 or tool.input_command contains "| zsh")
 output: >
 Falco blocked piping to a shell interpreter (%tool.input_command)
 priority: CRITICAL
 source: coding_agent
 tags: [coding_agent_deny]

The output field is designed to be LLM-friendly, so that the agent receives it as a structured message it can surface directly to the user. Correlation IDs allow you to trace every event across your logs.

The default ruleset ships with policies covering six areas:

Working-directory boundary — monitor and ask on file access outside the session's project directory
Sensitive paths — deny reads and writes to /etc/, ~/.ssh/, ~/.aws/, cloud credentials, .env files, and similar
Sandbox disable — detect attempts to disable the agent's own sandbox configuration
Threats — credential access, destructive commands, pipe-to-shell, encoded payloads, exfiltration, IMDS access, reverse shells, and supply-chain installs from known-malicious hosts
MCP and skill content — MCP server config poisoning and slash-command file injection
Persistence vectors — hook injection, git hooks, package-registry redirects, AI API base-URL overrides, and API keys leaking into env files

You can add your own rules to ~/.prempti/rules/user/; they're preserved across upgrades.

Rule authoring with Claude Code

The project also includes a Claude Code skill for writing Falco rules for Prempti interactively. You can install it directly from the Prempti plugin marketplace:

/plugin marketplace add falcosecurity/prempti
/plugin install prempti-falco-rules@prempti-skills

Then you can ask Claude Code to create rules like:

"Block the agent from running git push"
"Deny any read outside the working directory"
"Create a rule that requires confirmation before editing Dockerfiles"

The skill guides you through writing the rule, placing it in the right directory, and validating it with Falco. It's a great example of the kind of human-AI collaboration this project is designed to enable: the agent helps you constrain itself.

Let's be honest about limitations

We want to be clear about what this project is and isn't.

Prempti intercepts tool calls as declared by the agent, not the system calls those tool calls produce. If an agent writes a malicious binary and runs it, Falco sees gcc main.c -o main and ./main, not what ./main does at the OS level. For deep syscall-level visibility on Linux, Falco's kernel instrumentation (eBPF/kmod) remains the right tool.

Prempti is also not a sandbox. It doesn't prevent a sufficiently determined agent from circumventing the hook mechanism if it can find a path the hook doesn't cover. Think of it as a policy layer at the agent level — a valuable complement to sandboxing and system hardening, not a replacement for them.

What it does provide is visibility and a programmable policy boundary that lives at the most natural enforcement point: the moment the agent decides to act.

Getting started

Download the latest release from the GitHub repository: https://github.com/falcosecurity/prempti/releases

macOS:

installer -pkg prempti-<version>-darwin-universal.pkg \
 -target CurrentUserHomeDirectory

The installer wizard handles everything. The service starts automatically on login.

Linux:

tar xzf prempti-<version>-linux-x86_64.tar.gz
cd prempti-<version>-linux-x86_64
bash install.sh

Windows:

msiexec /i prempti-<version>-windows-<arch>.msi

Verify your setup:

premptictl status
premptictl hook status

Explore together with us

Runtime security for AI coding agents is genuinely new territory. The threat models are still being defined. The right default policies are still being discovered. We believe our community of developers, security engineers, and the people running these agents day to day are the ones who will figure out what good looks like here. If you've used Prempti, we'd love to hear what you found:

What rules have you written? What did you catch?
What agents or platforms do you need support for?
What didn't work as expected?

Open an issue, start a discussion, or come chat with us in the Falco Slack. Every piece of feedback shapes what this project becomes.

Prempti is released under the Apache License 2.0. Currently supports Claude Code on Linux (x86_64, aarch64), macOS (Apple Silicon, Intel), and Windows (x86_64, ARM64). Codex integration is on the roadmap.

Blog: Introducing Falco Operator 0.2.0

Mon, 23 Mar 2026 00:00:00 +0000

Dear Falco Community, today we are excited to announce the release of Falco Operator 0.2.0, the first production-ready release of the Kubernetes operator for Falco!

Since the technical preview announced with Falco 0.41.0, we have been working hard to make the operator robust, extensible, and ready for real-world environments. This release brings a redesigned API, a new Component controller for managing the Falco ecosystem, new artifact management capabilities, enhanced observability, and a significantly improved operational model, all grounded in Kubernetes-native patterns.

We merged 58 commits since v0.1.1, delivering major new features, 10 bug fixes, and comprehensive architectural improvements. Thank you to all our contributors and the community for your feedback along the way!

Going forward, the Falco Operator is the recommended way to deploy and manage Falco on Kubernetes. While the existing Helm chart remains fully supported, we plan to transition to the operator as the standard deployment method. More details on the transition timeline will follow in a future announcement.

To learn everything about the changes, read on!

What's new? TL;DR

Key features:

Ecosystem components - deploy Falcosidekick, Falcosidekick UI, and k8s-metacollector as managed components
ConfigMap support for rules and configuration, alongside OCI artifacts and inline definitions
Structured API types for inline rules and configuration - YAML objects instead of strings
Redesigned OCI artifact API with separate image and registry configuration
Reference tracking with finalizers to prevent accidental deletion of Secrets and ConfigMaps
Enhanced observability with Kubernetes events and status conditions across all controllers
Update strategy support for DaemonSet and Deployment modes
Server-Side Apply migration for safer, conflict-free reconciliation

Key fixes:

Plugin initConfig now supports nested configuration objects
RBAC compatibility with Kubernetes 1.32+
Spurious update prevention via managed fields comparison
Correct event recording with node-level attribution

This release comes with breaking changes that require updating your existing custom resources before upgrading. Please read the migration guide carefully before proceeding.

The road to production readiness

When we introduced the Falco Operator as a technical preview in Falco 0.41.0, the vision was clear: provide a Kubernetes-native way to deploy and manage Falco that goes beyond what Helm charts and static manifests can offer. Since then, every aspect of the operator has been refined.

The reconciliation logic now uses Server-Side Apply for conflict-free updates. Status conditions follow Kubernetes conventions (Programmed, ResolvedRefs, Available, Reconciled) so that standard tooling and dashboards can monitor operator health out of the box. Finalizers protect referenced resources from accidental deletion. And the new Component controller lays the groundwork for managing the entire Falco ecosystem from a single operator.

Version 0.2.0 is the result of this effort, a release we are confident in for production environments.

Major features and improvements

Ecosystem components

The new Component custom resource (instance.falcosecurity.dev/v1alpha1) enables the operator to deploy and manage the full Falco ecosystem from a single control plane. Three component types are supported:

Type	Component	What it does
`metacollector`	k8s-metacollector	Centralized Kubernetes metadata for Falco instances
`falcosidekick`	Falcosidekick	Fan-out daemon - routes Falco events to 70+ integrations (Slack, Elasticsearch, S3, Kafka, and more)
`falcosidekick-ui`	Falcosidekick UI	Web dashboard for real-time event visualization

Deploying Falcosidekick is as simple as:

apiVersion: instance.falcosecurity.dev/v1alpha1
kind: Component
metadata:
 name: sidekick
spec:
 component:
 type: falcosidekick
 replicas: 2

The operator handles the Deployment, Service, ServiceAccount, and RBAC automatically. Each component type ships with production-ready defaults (health probes, security context, resource profiles) that can be fully customized via podTemplateSpec.

For Falcosidekick UI, note that an external Redis instance is required. If Redis is not available, the pod stays in Init:0/1 state, the built-in wait-redis init container blocks until Redis is reachable. See the component documentation for setup examples including a bundled Redis StatefulSet.

As part of this work, the internal controller structure was reorganized under controllers/instance/ with shared reconciliation logic extracted into reusable packages, improving maintainability and consistency across all instance-level controllers.

ConfigMap support for rules and configuration

Rulesfile and Config resources can now source their content from Kubernetes ConfigMaps, in addition to OCI artifacts and inline definitions. This provides a familiar, Git-friendly workflow for teams that manage configuration through standard Kubernetes tooling.

Rulesfile from a ConfigMap:

apiVersion: artifact.falcosecurity.dev/v1alpha1
kind: Rulesfile
metadata:
 name: custom-rules
spec:
 configMapRef:
 name: my-rules-configmap
 priority: 50

Config from a ConfigMap:

apiVersion: artifact.falcosecurity.dev/v1alpha1
kind: Config
metadata:
 name: custom-config
spec:
 configMapRef:
 name: my-config-configmap
 priority: 50

The referenced ConfigMap must contain the content under a key named rules.yaml (for Rulesfile) or config.yaml (for Config). All three sources (OCI, inline, and ConfigMap) can be combined in a single resource when needed.

Structured API types

The inlineRules field in the Rulesfile CRD and the config field in the Config CRD are now structured YAML objects instead of plain strings. This enables proper validation, better editor support, and eliminates the need for YAML-in-YAML escaping.

Before (v0.1.x):

spec:
 config: |-
 engine:
 kind: modern_ebpf

After (v0.2.0):

spec:
 config:
 engine:
 kind: modern_ebpf

The same applies to inlineRules: rules are now defined as structured YAML lists rather than pipe-literal strings.

Redesigned OCI artifact API

The OCI artifact reference model has been completely redesigned for clarity and extensibility. The previous flat reference and pullSecret fields are replaced with a structured image and registry model.

Before (v0.1.x):

spec:
 ociArtifact:
 reference: ghcr.io/falcosecurity/rules/falco-rules:latest
 pullSecret:
 secretName: my-secret

After (v0.2.0):

spec:
 ociArtifact:
 image:
 repository: falcosecurity/rules/falco-rules
 tag: latest
 registry:
 name: ghcr.io
 auth:
 secretRef:
 name: my-secret

This separation makes the API more explicit and enables per-registry TLS configuration, plain HTTP support, and a consistent credential model. See the migration guide for details on updating your resources.

Reference tracking with finalizers

The operator now adds finalizers to Secrets and ConfigMaps that are referenced by artifact resources. This prevents accidental deletion of credentials or configuration data that would break Falco deployments. When a referenced resource is deleted, the operator blocks the deletion until all referencing artifacts are updated or removed.

Enhanced observability

All controllers now emit Kubernetes events for significant operations: artifact creation, updates, removals, and priority changes. Events include the node name for artifact controllers, making it straightforward to trace which operations happened on which nodes.

Status conditions have been overhauled to follow Kubernetes conventions:

Artifact resources report Programmed (whether the artifact is successfully applied) and ResolvedRefs (whether referenced ConfigMaps/Secrets exist)
Falco instances report Reconciled and Available
All artifact CRDs now include printcolumns for readable kubectl get output

Update strategy support

The Falco CRD now accepts update strategy configuration for both deployment modes:

# DaemonSet mode
spec:
 type: DaemonSet
 updateStrategy:
 type: RollingUpdate
 rollingUpdate:
 maxUnavailable: 1

# Deployment mode
spec:
 type: Deployment
 strategy:
 type: RollingUpdate
 rollingUpdate:
 maxUnavailable: 1
 maxSurge: 1

Server-Side Apply

Under the hood, the operator has migrated from the dry-run/update pattern to Server-Side Apply (SSA) for all reconciliation operations. This brings:

Conflict detection: concurrent modifications to managed fields are detected and reported
Ownership tracking: the operator only manages fields it owns, leaving user-applied changes intact
Reduced spurious updates: managed fields comparison prevents unnecessary API calls

Breaking changes ⚠️

Version 0.2.0 includes several API breaking changes. All existing custom resources must be updated before upgrading. A detailed migration guide is available in the repository documentation.

Summary of breaking changes

Change	Impact	Migration
`ociArtifact.reference` replaced by `ociArtifact.image` + `ociArtifact.registry`	All Rulesfile and Plugin CRs using OCI artifacts	Split the reference into `image.repository`, `image.tag`, and `registry.name`
`ociArtifact.pullSecret` replaced by `ociArtifact.registry.auth.secretRef`	CRs with private registry credentials	Update the credential reference path
Config `spec.config` changed from string to structured YAML	All Config CRs	Remove the `\|-` pipe literal, write YAML directly
Rulesfile `spec.inlineRules` changed from string to structured YAML	Rulesfile CRs with inline rules	Remove the `\|-` pipe literal, write YAML directly
Plugin `spec.config.initConfig` changed from `map[string]string` to JSON	Plugin CRs with init config	Re-apply CRD; flat maps still validate
Falco resource `shortName` changed from `prom` to `falco`	Scripts using `kubectl get prom`	Use `kubectl get falco` instead
Condition types renamed (`ConditionReconciled` → `Reconciled`, `ConditionAvailable` → `Available`)	Monitoring tools filtering on condition types	Update condition type filters
`kubectl get` column output changed for all CRDs	Dashboard parsing or scripts	Update parsers to match new column names
RBAC permissions expanded	Security-sensitive environments	Review the updated ClusterRole

After upgrading, re-apply all CRDs and update your custom resources following the migration guide. The operator will reconcile the new format automatically.

A Helm chart is on its way

We are currently developing a Helm chart for installing the Falco Operator itself, which will simplify deployment and configuration of the operator in production environments. Stay tuned for the announcement.

Meet us at KubeCon

We will be talking about the Falco Operator during the maintainer track at the upcoming KubeCon. If you are interested in learning more, asking questions, or sharing feedback, come find us at the CNCF Falco kiosk, we would love to chat!

Try it out

Install the operator:

VERSION=latest
if [ "$VERSION" = "latest" ]; then
 kubectl apply --server-side -f https://github.com/falcosecurity/falco-operator/releases/latest/download/install.yaml
else
 kubectl apply --server-side -f https://github.com/falcosecurity/falco-operator/releases/download/${VERSION}/install.yaml
fi

Then choose how you want to get started:

Full stack quickstart

Deploy the entire Falco ecosystem in the falco namespace with one command:

VERSION=latest
if [ "$VERSION" = "latest" ]; then
 kubectl apply --server-side -f https://github.com/falcosecurity/falco-operator/releases/latest/download/quickstart.yaml
else
 kubectl apply --server-side -f https://github.com/falcosecurity/falco-operator/releases/download/${VERSION}/quickstart.yaml
fi

This deploys Falco, container and k8smeta plugins, detection rules, Falcosidekick, Falcosidekick UI with Redis, and k8s-metacollector - all pre-wired.

Verify:

kubectl get falco,plugins,rulesfiles,configs,components -n falco
kubectl get pods -n falco

Step by step

Deploy Falco:

cat <<EOF | kubectl apply -f -
apiVersion: instance.falcosecurity.dev/v1alpha1
kind: Falco
metadata:
 name: falco
spec: {}
EOF

Add the container plugin (required by the official rules for container metadata fields):

cat <<EOF | kubectl apply -f -
apiVersion: artifact.falcosecurity.dev/v1alpha1
kind: Plugin
metadata:
 name: container
spec:
 ociArtifact:
 image:
 repository: falcosecurity/plugins/plugin/container
 tag: latest
 registry:
 name: ghcr.io
EOF

And add detection rules:

cat <<EOF | kubectl apply -f -
apiVersion: artifact.falcosecurity.dev/v1alpha1
kind: Rulesfile
metadata:
 name: falco-rules
spec:
 ociArtifact:
 image:
 repository: falcosecurity/rules/falco-rules
 tag: latest
 registry:
 name: ghcr.io
 priority: 50
EOF

Optionally, add Falcosidekick to route events to your favorite integrations:

cat <<EOF | kubectl apply -f -
apiVersion: instance.falcosecurity.dev/v1alpha1
kind: Component
metadata:
 name: sidekick
spec:
 component:
 type: falcosidekick
 replicas: 2
EOF

For the complete documentation, including the CRD reference, configuration options, and architecture overview, visit the Falco Operator repository and the operator setup guide.

Stay connected

Join us on social media and in our community calls! It's always great to have new members in the community, and we're looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Falco at KubeCon Europe 2026 — See You in Amsterdam! 🐦

Wed, 18 Mar 2026 00:00:00 +0000

We're excited to share that the Falco community will be at KubeCon + CloudNativeCon Europe 2026 in Amsterdam! Whether you're a long-time contributor, a curious user, or just want to say hi, we'd love to see you there.

Falco is celebrating 10 years of development and adoption, and we are on the lookout for people who would like to say Happy Birthday to the project or share their best Falco story. Libby Schulze and I will be on the event floor with mic and camera to capture some amazing moments and memories from Falco's 10 years. So bring your best story, and we'll see you at the Falco booth!

Sneak peek

Psst... we have something really cool brewing that we will show at the Falco booth. You, our amazing reader, is the first to hear about this. It's a way to run Falco locally on your development machine, and make sure your AI coding agents are following new rules that are being defined. We'd love to get your feedback on this as we're currently building it!

Here’s where you can find us in Amsterdam and everything we have lined up:

Project lightning talk

Forensics With Falco
Speaker: Gerald Combs, Maintainer
When: Monday, March 23, 2026 — 10:27 to 10:32 CET
Where: Elicium 2

Falco has recently expanded its capabilities with capture recording, opening the door to seamless integration with forensic analysis tools like Stratoshark. In this lightning talk, Gerald will walk through how the two tools work together to provide deep visibility into container and system activity. He will demonstrate how captured event data can accelerate investigations and discuss key considerations for safely and efficiently deploying these features in production environments.

Sysdig-led workshop

Hands-On Cloud Native Security Workshop
When: Monday, March 23 — 2:00–4:00 PM CET

Run Atomic Red Team™ tests, then step into the Blue Team role to detect threats and create custom Falco™ detection rules in this hands‑on 90‑minute keyboard workshop.

Conference talk

In Falco's Nest: The Evolution of Cloud Native Runtime Security
Speakers: Iacopo Rozzo (Sysdig), Aldo Lacuku (Kong Inc.)
When: Tuesday, March 24, 2026 — 12:00 to 12:30 CET
Where: G102–103

Falco, the Cloud Native Runtime Security project, is constantly evolving to meet the demands of modern cloud environments. This maintainer track session, led by the Falco maintainers, will dive deep into the latest advancements and the strategic direction of the project. We will focus on two major areas of growth: the introduction of the new Falco Operator and the new features that enhance Falco's performance and reliability.

The new Falco Operator simplifies the deployment, configuration, and management of Falco across Kubernetes clusters, making it easier than ever for users to secure their runtime environments at scale.

Furthermore, we will explore the most significant new features integrated into Falco. This includes performance optimizations for high-throughput environments. The session will also touch upon community contributions, ecosystem integrations, and the roadmap for the upcoming release.

Booth demo

Pivoting from detection to investigation with Falco and Stratoshark
Speaker: Gerald Combs
When: Tuesday, March 24, 2026 — 15:45 CET
Where: Sysdig Booth #671

See how to move from “we detected something” to “here’s what happened” using Falco and Stratoshark. Stop by the Sysdig booth and say hello!

Thank you!

We couldn’t do this without you all in our community - the contributors, users, and everyone who shows up at events. If you’re in Amsterdam, come find us at the talks, the workshop, or the booth. We’d love to meet you and hear how you’re using Falco.

See you there! 🐦

Blog: Hey Falco Flock! 🐦 Let's Soar Into 2026

Wed, 25 Feb 2026 00:00:00 +0000

New year, new opportunities!

As we spread our wings and glide into 2026, we want to make sure this community is one you’re proud (and excited!) to be a part of. Falco has always been more than just a project: it’s a flock of builders, defenders, contributors, question-askers, doc-writers, rule-tuners, and runtime security enthusiasts.

And now we want to hear from you.

We’ve put together a quick community survey (5 minutes or less!) to better understand:

How connected you feel to the community
What you love about being a part of it
What could be better
What you’d like to see us focus on this year
What resources would make your life easier
How you’re using Falco and what tools you integrate it with

Your feedback directly shapes our focus on what we build, improve, prioritize, and invest in this year - from documentation and content to events, integrations, and contributor experience. A report detailing the responses will be shared at the same time as KubeCon Europe 2026.

Whether you’re building, using, learning, or just keeping an eye on things, your voice matters.

👉 Take the survey here

Thanks for being part of the flock. We couldn’t do this without you and we’re excited to build 2026 together!

Blog: Introducing Falco 0.43.0

Mon, 26 Jan 2026 00:00:00 +0000

Dear Falco Community, we are happy to announce the release of Falco 0.43.0 today!

This is a stabilization release that consolidates the changes introduced in 0.42.0, including the drop-enter initiative and the capture recording feature. It also introduces several deprecations to improve maintainability and fixes minor issues across falcoctl, plugins, and libs.

During this release cycle, we merged:

31 PRs on Falco, including 11 release note-worthy changes
48 PRs on Falco libs, including 17 release note-worthy changes
8 PRs on Falco drivers, including 3 release note-worthy changes

We upgraded libs to version 0.23.1 and drivers to 9.1.0+driver. Thank you to our maintainers and contributors. This would not have been possible without your support and dedication!

To learn everything about the changes, read on!

What's new? TL;DR

Key fixes:

evt.arg.filename field reintroduction
Falcoctl signature verification fixes
overflow and NULL pointer dereferences fixes for the container plugin, shipped with plugins/container/0.6.1
race condition fix for the k8smeta plugin, shipped with plugins/k8smeta/0.4.1

This release also comes with breaking changes that you should be aware of before upgrading.

Latest updates

Deprecations

In Falco 0.43.0, we are announcing the deprecation of three significant components to streamline the project, reduce maintenance burden, and focus on modern, more efficient alternatives. All these components are stable, and considering that the deprecation is first enforced in this version, they could be removed at any future version starting from 0.44.0.

Legacy eBPF probe deprecation

The "legacy" eBPF probe (configured via engine.kind=ebpf) was the original eBPF implementation in Falco. It required compiling a specific probe for each kernel version, often necessitating the dynamic usage of the falco-driver-loader or pre-built drivers. The Modern eBPF probe (engine.kind=modern_ebpf), which leverages CO-RE (Compile Once – Run Everywhere), has reached maturity and feature parity. It offers superior stability, portability (no need to compile drivers on the fly), flexibility and performance. Maintaining two eBPF drivers splits engineering effort and complicates the codebase. Users currently using the legacy eBPF probe are strongly encouraged to switch to the Modern eBPF probe by setting engine.kind=modern_ebpf in their falco.yaml, or to engine.kind=kmod if the used kernel doesn't provide support for the modern eBPF probe.

See the relevant section of the deprecation proposal for the detailed motivation behind the deprecation.

gVisor deprecation

The gVisor engine is a dedicated, internal C++ implementation designed to monitor system calls from gVisor sandboxes leveraging events coming from gVisor itself through gRPC. There is evidence that this engine is little used. Moreover, gVisor doesn't provide all information required to build all supported event types, indeed resulting in a system call source not completely equivalent to the ones provided by drivers. Finally, it requires libs being dependent on protobuf, this latter introducing a non-negligible build time overhead and maintainability burden.

See the relevant section of the deprecation proposal for the detailed motivation behind the deprecation.

gRPC output and server deprecation

The gRPC output was implemented to allow external consumers to subscribe to a stream of Falco security alerts over a gRPC connection. It was notably utilized by tools like the event-generator (in test mode) and custom integrations requiring a streaming API for alerts. The gRPC output and the gRPC server embedded in Falco add substantial complexity to the core codebase, including dependencies on specific protobuf and gRPC framework versions in Falco and libs. Over time, it has become clear that the community prefers standard, widespread integration patterns for alert consumption - primarily HTTP and the ecosystem enabled by Falcosidekick. Users consuming alerts via gRPC should migrate to the HTTP output or use Falcosidekick to forward events to their destination of choice.

See the relevant section of the deprecation proposal for the detailed motivation behind the deprecation.

GPG key rotation

In anticipation of the previous GPG key's expiration in January 2026, we have rotated the GPG key used to sign the official RPM and DEB packages. Pre-existing Falco installations (installed via apt or yum before the rotation) must manually import the new GPG key. Failure to do so may result in errors during package updates or verification failures. Please follow the "Trust the falcosecurity GPG key" step in the official documentation for your package manager:

apt (Debian/Ubuntu): Install with apt
yum/dnf (CentOS/RHEL/Fedora): Install with yum

Notice that new installations following the current documentation will automatically receive the updated key bundle and do not require additional steps.

For more details see [TRACKING] [deadline 2026-01-17] Rotate public GPG key for RPM/DEB package signing.

Container plugin improvements

The container plugin, which extracts metadata from container runtimes to enrich Falco events, includes important updates in version 0.6.1 to enhance its API capabilities and performance. This release exposes container.id, container.image, container.name, and container.type through the table API and adds comprehensive logging across all engines, while also preventing allocations by extensively using zero-allocation tools offered by the C++ (like std::string_view) and avoiding reflex matcher allocations during resolve operations.

Falcoctl tweaks and improvements

`follow` polling interval increase to 1 week

About three years ago, we started distributing Falco artifacts (rules files and plugins) via ghcr.io, and later added automatic rule updates in falcoctl with a 6h check interval. With years of data now, it’s clear we don’t need checks that frequent: our rule updates happen far less often. Moreover, due to the growth of Falco adoption, these frequent checks are now hitting ghcr.io rate limit. These two reasons drove the decision to increase the artifact follow interval from 6h to 1 week.

For more details see chore(scripts/falcoctl): increase follow interval to 1 week and Falco's Helm chart changelog.

Dependency resolution improvements

The artifact installation logic has been reworked to handle dependencies and references correctly. Previously, dependencies could be duplicated or incorrectly resolved, and signature verification was skipped for full registry references. Now dependencies are properly deduplicated, all refs are correctly resolved, and signatures are verified for all resolved dependencies, not just the top-level artifacts. This provides end-to-end verification of the entire dependency chain.

For more details see Inefficient deduplication logic and incorrect input handling for dependency resolution

Support for cosign v3

Falcoctl now supports Cosign v3 bundle format for signature verification. This is the new standard for signing OCI artifacts, replacing the legacy .sig tag format.

What this means for you:

Artifacts signed with cosign v3 are now fully supported
Backward compatibility with cosign v2 signatures is maintained

For more details see feat: Upgrade to Cosign v3 with Bundle Format

Key fixes

`evt.arg.filename` field reintroduction

As part of the recent "drop enter" optimization initiative (which removed enter events for most syscalls to improve performance), the filename argument - historically available only in the enter event for execve and execveat - was inadvertently made unavailable. This caused a regression where specific context, such as the exact path provided to the syscall (e.g., a symlink path versus the resolved binary path), was lost in the remaining exit event.

In Falco 0.43.0 (via libs 0.23.0), this has been fixed. The filename argument is now correctly populated in the exit events for these syscalls. Users can once again access this data using the evt.arg.filename field in their rules, ensuring that the critical execution context is preserved without needing the deprecated enter events.

For more details see Missing "filename" argument to execve syscall in libscap 0.22.x.

Falcoctl signature verification fixes

Signature verification fix for full reference artifacts

Fixed an issue where signature verification was skipped for artifacts specified with a full registry reference ( e.g., ghcr.io/falcosecurity/plugins/plugin/container:0.4.1). Now all artifacts are verified regardless of how they are referenced.

Signature verification fix for authenticated registries

Signature verification now works correctly on private/authenticated registries. Previously, verification would fail with authentication errors even though the artifact pull succeeded, and credentials were not being passed to the signature verification component.

Supported authentication methods:

Basic auth (Docker credentials)
OAuth2 client credentials
GCP Workload Identity (for GKE deployments)

For more details see fix(signature): pass registry credentials to cosign for signature verification

Breaking changes and deprecations

This version includes breaking changes you should be aware of before upgrading.

Bump drivers minimum required kernel version to `3.10`

Falco 0.43.0 introduces a breaking change regarding the Falco drivers. Starting with drivers version 9.1.0+driver, the minimum required Linux kernel version has been bumped to 3.10. In practice, this only affects the kmod driver and means that the kernel module will explicitly fail to compile on kernels older than 3.10. This choice is motivated by the fact that even Linux 3.10 is a 12-year-old kernel, and its support ended in 2017: maintaining support for older kernels is a maintenance burden and limits progress. This change enables the team to focus on modernizing the codebase and improving stability for current environments.

Deprecation warnings

Falco 0.43.0 introduces several deprecation warnings to help users migrate to newer components:

Legacy eBPF probe deprecation: using the legacy eBPF probe (engine.kind=ebpf) will now generate warnings
gVisor engine deprecation: using the gVisor engine (engine.kind=gvisor) will now generate warnings
gRPC deprecation: using the gRPC output or the gRPC server (grpc_output.enabled=true or grpc.enabled=true), will now generate warnings

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

Stay connected

Join us on social media and in our community calls, held every other Wednesday! It's always great to have new members in the community, and we're looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: GPG Key Rotation for Falco Packages (2026)

Fri, 12 Dec 2025 00:00:00 +0000

The GPG key used to sign official Falco packages (RPM and DEB) is set to expire on January 17, 2026. To ensure the security and continuity of our software distribution, the Falco maintainers will be rotating to a new 4096-bit RSA key.

We have designed a two-phase "Soft Launch" strategy to make this transition as smooth as possible, providing a one-month transition window before the old key is retired.

The Rotation Plan

To avoid immediate disruption, we are rolling out the new key in two distinct phases. You can follow the detailed progress in our tracking issue #3750.

Phase 1: Soft Launch (Dec 12, 2025)

What happens: The new GPG key has been published and added to our repository configuration.
Dev Builds: Will begin using the New Key immediately.
Stable Builds: No stable releases are planned for this phase. If any hotfixes are released, they will be signed with New Key as well.
Key Bundle: The official key URL has been updated to serve a bundle containing both the Old (valid) and New (valid) keys.

Phase 2: Hard Cut-Over (Jan 12–17, 2026)

What happens: This is the maintenance window where we fully switch to the new key.
Mass Resign: All existing stable packages on download.falco.org will be resigned with the New Key.
Revocation: The Old Key will be officially revoked and removed from the active bundle.
Impact: If you have not updated your keyring by this date, your package manager (apt or yum) will reject updates with a signature verification error.

Action Items for Users

We strongly recommend all users update their GPG keyring before January 12, 2026 to avoid interruption.

New Users

If you are installing Falco for the first time following our Install on a host (DEB,RPM) instructions, no action is required. The installation process will guide you to fetch the new key bundle, ensuring you are ready for both phases.

Existing Users

If you have an existing Falco installation, you must manually import the new key. We have updated the key file at our standard URL to include both the old and new keys, allowing you to transition safely.

For apt users, to update your keyring, run the following command:

# Download the updated key bundle (Old + New) and import it
curl -fsSL https://falco.org/repo/falcosecurity-packages.asc | \
 sudo gpg --dearmor -o /usr/share/keyrings/falco-archive-keyring.gpg

For yum users, to update your keyring, run the following command:

# Download the updated key bundle (Old + New) and import it
rpm --import https://falco.org/repo/falcosecurity-packages.asc

Note: These commands overwrite the existing keyring file with the new bundle. Since the bundle contains both keys, your current installation will continue to work immediately, and will remain working after the January cut-over.

For more details on apt and yum specific instructions, please refer to the Install on a host (DEB,RPM) page of our documentation.

Summary

Deadline: Update your keys before Jan 12, 2026.
Old Key (Expiring Jan 17, 2026): falcosecurity-14CB7A8D.asc (Fingerprint 2005399002D5E8FF59F28CE64021833E14CB7A8D)
New Key: falcosecurity-B35B1B1F.asc (Fingerprint 478B2FBBC75F4237B731DA4365106822B35B1B1F)
Tracking Issue: falcosecurity/falco#3750

If you encounter any issues during this transition, please reach out to us on the #falco channel on Kubernetes Slack or open an issue on GitHub.

Thank you for your attention and cooperation in keeping Falco secure!

Blog: Introducing Falco 0.42.0

Wed, 22 Oct 2025 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.42.0!

This release brings exciting new capabilities, including the capture feature, significant performance improvements, and important bug fixes that enhance Falco's capabilities. During this release cycle, we merged:

52 PRs on Falco, including 23 release note-worthy changes
110 PRs on Falco libs, including 47 release note-worthy changes
102 PRs on Falco drivers, including 29 release note-worthy changes

We upgraded libs to version 0.22.1 and drivers to v9.0.0+driver. Thank you to our maintainers and contributors. This would not have been possible without your support and dedication!

To learn everything about the changes, read on!

What's new? TL;DR

Key features:

Key fixes:

Fix thread table memory leak when parsing vfork (or equivalent clone/clone3 with CLONE_VFORK) exit from the caller process;
Enable handling of multiple actions configured with syscall_event_drops.actions;
Disable dry-run restarts when Falco runs with config-watching disabled;
Fix abseil-cpp for Alpine build;
Fix detection sandbox containers for CRI and containerd runtimes (container plugin);
Stability improvements for container plugin and static linking of libgcc/libstdc++ for legacy compatibility;

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.42.0 release contains a new capture feature and significant performance improvements. Here is a list of the key new capabilities.

Capture recording feature

Falco 0.42.0 introduces the new capture recording feature, now available at sandbox maturity. This capability allows Falco to generate .scap files whenever a detection rule is triggered automatically.

Each capture contains a detailed trace of system calls around the event, providing forensic-level visibility into what happened. The recordings can be opened directly in Stratoshark for Wireshark-style analysis of runtime behavior.

The capture system is fully configurable: you can enable global recording or tie captures to specific Falco rules for targeted runtime snapshots.

When targeting specific Falco rules (by setting mode: rules, as shown in the configuration below), users can modify individual rules to enable capture by adding capture: true and optionally capture_duration to specific rules. For example:

 - rule: Suspicious File Access
 desc: Detect suspicious file access patterns
 condition: >
 open_read and fd.name startswith "/etc/"
 output: >
 Suspicious file access (user=%user.name command=%proc.cmdline file=%fd.name)
 priority: WARNING
 capture: true
 capture_duration: 10000 # Capture for 10 seconds when this rule triggers

This configuration will capture events for 10 seconds whenever the "Suspicious File Access" rule is triggered, overriding the default duration.

Find below the configuration snippet to enable the capture feature in falco.yaml:

capture:
 # -- Set to true to enable event capturing.
 enabled: false
 # -- Prefix for capture files. Falco appends a timestamp and event number to ensure unique filenames.
 path_prefix: /tmp/falco
 # -- Capture mode. Can be "rules" or "all_rules".
 mode: rules
 # -- Default capture duration in milliseconds if not specified in the rule.
 default_duration: 5000

Learn more at KubeCon + CloudNativeCon North America 2025:

Project Lightning Talk: When Falco Spots Trouble, The Shark Swims In - Gerald Combs, Falco Core Maintainer
Beyond the Cloud(s): Falco's Ascent in Performance and Deep Visibility - Leonardo Grasso & Leonardo Di Giovanna, Sysdig

Drop enter initiative

We've just shipped a significant performance improvement: syscall enter events have been completely removed from our event pipeline.

In Falco, each system call traditionally used to generate two events: an enter event when syscall kernel processing starts (i.e., before its arguments are processed) and an exit event when the kernel processing completes. Now that we collect all relevant information on exit events, we can drop the generation and processing of enter events.

Nevertheless, for TOCTOU (Time-of-Check to Time-of-Use) mitigation, a few selected enter events are still monitored internally — their relevant data is captured and stored — but these events are no longer pushed downstream to the userspace processing pipeline.

By focusing solely on syscall exit events, we've nearly halved the number of events generated and processed by userspace, eliminating redundant data collection. This reduces the Falco instrumentation overhead, improving workloads' performance up to 20% (by reducing syscall execution latency). It also decreases Falco's CPU usage up to 30%, especially in high-syscall environments.

From a developer's perspective, this also removes ambiguity about where syscall parameters should be defined, streamlines event processing logic, and makes event handling code cleaner and easier to maintain.

Overall, you can expect better performance, leaner code, and a more predictable event model moving forward.

For more details, see:

Plugin event schema versioning

Falco 0.42.0 introduces plugin event schema validation, enabling plugins to specify their compatible event schema version.

It provides an event schema validation system for syscall events consumed by plugins that offer parsing and/or field extraction capabilities, ensuring backward compatibility and clear error reporting for plugins that depend on specific Event Schema Versions.

If the plugin does not declare a required Schema Version, it is assumed to be compatible with 3.0.0, the initial major version when the plugin event schema validation was introduced.

The plugins should implement a new function of the Plugin API to declare the required schema version. Find below the signature of the new API function:

// New plugin API functions for schema management
typedef struct {
...
// Event schema version check
//
// Return the minimum event schema version required by this plugin.
// Required: no
// Arguments:
// - s: the plugin state, returned by init(). Can be NULL.
// Return value: the event schema version string, in the following format:
// "<major>.<minor>.<patch>", e.g. "4.0.0".
// If the function is not implemented or NULL is returned, the plugin is assumed to be
// compatible with schema version 3.0.0.
//
const char* (*get_required_event_schema_version)(ss_plugin_t* s);
} plugin_api;

For more details, see:

Plugin system event schema versioning proposal

Thread table auto-purging configuration

We've added a few new falco_libs configurations for advanced users who want finer control over Falco's performance and resource usage. It introduces tunable parameters for Falco's internal thread table, which tracks active threads:

thread_table_size defines the maximum number of entries.
thread_table_auto_purging_interval_s controls how often stale threads are cleaned up.
thread_table_auto_purging_thread_timeout_s sets how long inactive threads are kept before removal.

These options let you balance memory efficiency, CPU usage, and state accuracy, with related metrics (n_drops_full_threadtable, n_store_evts_drops) available to guide tuning.

Static fields

Falco 0.42.0 adds a new static_fields configuration object allowing users to add statically defined fields to the Falco engine. The following example illustrates how to specify new static fields:

static_fields:
 foo: bar
 foo2: ${bar2}

Notice that foo2: ${bar2} leverages the Falco's behavior of expanding env variables in config YAML.

After specifying them, these fields can be used in normal rule conditions, by prepending the static. prefix (e.g.: evt.type=open and static.foo=bar). Moreover, if append_output.suggested_output is true, they'll be automatically appended to each rule output, in the form static_foo=bar.

For more details, see:

Breaking changes and deprecations ⚠️

This version comes with breaking changes that you should be aware of before upgrading.

Event direction and `evt.dir` deprecation

Following the enter events initiative, the evt.dir field, as well as the concept of "direction", have been deprecated in Falco 0.42.0 and will be removed in a future release. Until field removal and since Falco 0.42.0, specifying evt.dir='>' will match nothing, while specifying evt.dir='<' will match everything, with a warning informing the user about the deprecation. Users are encouraged to get rid of any reference to evt.dir, as its presence will result in an error at rules loading time after its removal.

Plugin API changes

Old plugins consuming syscall events not declaring the required event schema version will be incompatible with Falco 0.42.0 and later.

Deprecation warnings

Falco 0.42.0 introduces several deprecation warnings to help users migrate to newer APIs:

evt.dir field deprecation: Rules using the deprecated evt.dir field will now generate warnings;
Enter events drop stats: Prometheus metrics for enter events drop statistics have been deprecated;
Configuration warnings: Enhanced warning system for deprecated configuration options;

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

Stay connected

Join us on social media and in our community calls, held every other Wednesday! It's always great to have new members in the community, and we're looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Introducing Falco 0.41.0

Thu, 29 May 2025 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.41.0!

This version brings several new features, performance enhancements, and bug fixes that streamline Falco’s detection capabilities. During this release cycle, we merged more than 50 PRs on Falco and around 130 PRs for libs and drivers, version 0.21.0 and version 8.1.0, respectively. Thank you to our maintainers and contributors. This would not have been possible without your support and dedication!

To learn everything about the changes, read on!

What’s new? TL;DR

Key features:

Reimplemented container engines support from scratch;
A Kubernetes operator is taking shape;
Falco's config_files configuration gained support to specify the merge strategy;
Modern eBPF driver is now capable of trying to load multiple programs for each event; consequently, sendmmsg and recvmmsg will now make use of bpf_loop eBPF helper where available, boosting their performances;
New proc.aargs field available, ie: a lookup for an ancestor args field;
proc.args gained support for indexed access, to only check a certain argument;
json_include_output_fields configuration key for Falco to control whether output fields are included in the JSON message;
Ongoing work to improve libs code modularity;

Key fixes:

Avoid kmod crashing when a CPU gets enabled at runtime;
Fixed Falco Prometheus metrics with multiple event sources enabled;
Fixed RPM packages evaluation of RPM scripts;
-o options do now correctly override included config_files;

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.41.0 release contains a number of features and UX improvements. Here is a list of some of the key new capabilities.

Reimplemented container engines support

In the Falco 0.41.0 release, the Falco team has completely revised its support for container engines. Key improvements include:

Container support is now a plugin;
The plugin will attach a listener to the engine's SDKs onCreate signal; since onCreate comes way before onStart, we have plenty of time to deliver the container's metadata before the first process in the container is even started;
For now, it is bundled within Falco to avoid breaking changes, but in the future, it will need to be downloaded through falcoctl;

These changes should address all issues related to missing container metadata.

Kubernetes operator

In Falco 0.41.0, we worked hard to create a Falco k8s operator: https://github.com/falcosecurity/falco-operator/. For now, this is considered a technical preview, but we will deliver a fully functional operator very soon. Expect more news in a new blog post!

Security best practices improvements

We are grateful for the suggestions we received from security experts and adopters in our community, and so we implemented the following enhancements:

The modern eBPF probe will no longer store security sensitive settings in the .bss mmapable segment but will use dedicated maps instead. This is a security best practice because it prevents other processes running with elevated privileges from tampering with the map file descriptor, which would be harder to detect. We would like to thank Mouad Kondah for suggesting this change!

Falco will no longer consider rule files or contents of rule directories that do not have a .yml/.yaml extension. This prevents accidental processing of files that are not related to rules. We would like to thank our user Travis Smith for suggesting this change!

Breaking changes and deprecations ⚠️

This version comes with breaking changes, mostly in the configuration interface.

Removed command line options and equivalent configuration options

We removed the already deprecated options -S/--snaplen, -A, and -b, and it is now possible to achieve the same result through the Falco configuration:

for -S/--snaplen: falco_libs.snaplen config key;
for -A: base_syscalls.all config key;
for -b: buffer_format_base64 config key;

The configuration options for the container engines, added in 0.40.0, have been completely dropped in favor of the new plugin init configuration which can be found at https://github.com/falcosecurity/plugins/tree/main/plugins/container#configuration.

You can find more information on breaking changes in the tracking issue.

Behavior changes

Falco will now only consider and consequently load rules whose name ends in .yml or .yaml.

Dropped features

syslog related fields were dropped by libs, since they were unused.

Also, as a consequence of the new container plugin, some breaking changes had to take place:

the musl build is inherently not able to load plugins; that means that it loses container metadata support;
falcosecurity_scap_n_containers and falcosecurity_scap_n_missing_container_images metrics are now moved to the plugin, and their name now have the falcosecurity_plugins_ prefix;
-pc and -pk command line options are now ineffective; it is up to the container and k8smeta plugins to declare suggested fields to be used as output fields; consequently, container_image=%container.image.repository and k8s_ns=%k8s.ns.name changed their name to container_image_repository= and k8s_ns_name=;

Deprecations

In Falco 0.41.0, we have deprecated the following options:

-p cli flag; the only remaining user for it is gVisor, which will be ported to a plugin sooner or later and will then make use of the suggested output fields plugin API;

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

Falco is more mature with each release. Following its graduation, we have published the roadmap for version 1.0.0, which is guiding us in the next steps. For the next release, you can expect more stability, a refined k8s operator, improved performance, and, as always, new detections and fixes.

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Detecting Supply Chain Attacks with Falco Actions

Wed, 19 Mar 2025 00:00:00 +0000

The recently discovered CVE for the GitHub action tj-actions/changed-files brought to light a topic that is really critical for companies: supply chain attacks. With that, we want to discuss and show a bit about how Falco can help your organization detect this kind of attack and other suspect behaviors inside your CI/CD pipeline.

What is Falco?

Falco is a cloud native security tool that provides runtime security across hosts, containers, Kubernetes, and cloud environments. It leverages custom rules on Linux kernel events and other data sources through plugins, enriching event data with contextual metadata to deliver real-time alerts. Falco enables the detection of abnormal behavior, potential security threats, and compliance violations.

What is Falco Actions?

Falco Actions enable you to run Falco in GitHub Actions to detect suspicious behavior in your CI/CD workflows. If you run it in a pull request, the action will create a comment with the findings.

Thanks to ad-hoc Falco rules specific to this use case, these GitHub actions can monitor your GitHub runner and detect software supply chain attacks.

Using Falco Actions

To have Falco inside your pipeline, you need to add these two actions:

falcosecurity/falco-actions/start
falcosecurity/falco-actions/stop

Below you can see an example:

name: CI

on:
 push:
 pull_request:

jobs:
 build:
 runs-on: ubuntu-latest
 permissions:
 contents: read
 actions: read

 steps:
 - uses: actions/checkout@v4

 - name: Start Falco
 uses: falcosecurity/falco-actions/start@main
 with:
 mode: live
 falco-version: '0.40.0'
 verbose: true

 - name: My Custom Step
 run: |
 echo "This is my custom step"

 - name: Stop Falco
 uses: falcosecurity/falco-actions/start@main
 with:
 mode: live
 verbose: true

OBS: main is being used here only to simplify how it works, you should always pin your dependencies to a specific commit SHA.

After the execution, you will be able to see the results at the github action summary.

If you want a more detailed report, you can use the action falcosecurity/falco-actions/analyze; it will allow you to have a better report with information like:

Falco rules triggered during steps' execution.
Contacted IPs
Contacted DNS domains
SHA256 hash of spawned executables
Spawned container images
Written files
A summary of the report generated with OpenAI
Reputation of Contacted IPs
Reputation of SHA256 hashes

For more informations about the usage, you can check the github repository for the actions.

Default rules file

By default, Falco action will detect a variety of events, following the default CICD rules, that can be overridden if you want.

In the example from the tj-actions/changed-files exploit, one rule that would be triggered is the Process Dumping Memory of Others, which was used during the exploit to dump environment variables from the main process and print them as part of the Github runner execution.

The Falco team is always adding new rules to ensure our users get value out of the box, but you can also write your own rules according to your company policy.

Conclusions

These actions are just the beginning of having the Falco capabilities inside the CI/CD pipelines. You can customize and have your own set of rules, keeping all environments and scenarios covered and protected from supply chain attacks.

Let's meet!

As always, we meet every 2 weeks on Wednesday at 4pm UTC in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Enjoy 😎,

Igor and Edson

Blog: Detecting Threats in OVHcloud MKS Audit Logs with Falco

Thu, 13 Mar 2025 00:00:00 +0000

Detecting threats in a Kubernetes cluster can be challenging, we generally don't know where and how to start. The good news is that we have an amount of valuable logs that can help us to know what is happened in the cluster. Indeed, each action requested or done by a user or an app, in a cluster, is recorded in Audit Logs. Kubernetes events are key to understanding the behavior of a cluster.

We already provide plugins that let you parse Audit Logs and use Falco to detect threats from GKE, EKS and AKS clusters. With our latest plugin, you'll now have the same powerful threat detection capabilities for your OVHcloud MKS clusters 🎉.

What is Falco?

Falco is an Open Source cloud-native runtime security tool. It provides near real-time threat detection for cloud, container, and Kubernetes workloads by leveraging runtime insights. Falco can monitor events from various sources, including the Linux kernel, and enrich them with metadata from the Kubernetes API server, container runtime, and more.

Falco can receive Events, compare them to a set of Rules to determine the actions to perform and generate Alerts to different endpoints.

What is the OVH MKS Audit Logs plugin?

The OVH audit logs plugin (k8saudit-ovh) extends Falco's capabilities to OVHcloud Managed Kubernetes Service (MKS) clusters, providing you with the same security insights and threat detection Falco already offers for GKE, EKS and AKS environments.

With this plugin, you can seamlessly integrate MKS Audit Logs into Falco's event processing pipeline, enabling it to identify anomalies, suspicious activities, and policy violations within your MKS-based workloads.

Concretely, when a user executes some kubectl commands in an OVHcloud MKS cluster, Audit Logs will be generated. Falco is listening to them, and depending on the configured rules to follow, it will generate some alerts.

Using OVH MKS Audit Logs plugin

In order to use the OVH MKS Audit Logs plugin, you must follow several steps:

deploy an OVHcloud LDP (Logs Data Platform)
create a data stream into this LDP
connect an OVHcloud MKS cluster to the data stream (to send Audit Logs into it)

To be able to access our Kubernetes clusters' Audit Logs, you need to deploy an LDP. LDP is the managed platform for collecting, processing, analyzing, and storing your logs of the OVHcloud products. Deploy an LDP (Bare Metal Cloud universe) with whatever plan you want.

OVHcloud Kubernetes Audit Logs will be stored in a data stream. The OVHcloud Audit Logs Falco plugin receive the audit logs through Websocket so you need to enable Websocket broadcasting when you create the data stream on LDP.

Retrieve the Websocket URL, follow the guide to do so. The Websocket address have this kind of format: wss://gra.logs.ovh.com/tail/?tk=

Finally, you have to connect a MKS cluster to the LDP data stream.

Configuring Falco to use OVH Audit Logs plugin

Running locally

If you have a Falco running locally, using falcoctl, add the falcosecurity index (if it's not already the case) and install the k8saudit-ovh Falco plugin:

# Add falcosecurity index
sudo falcoctl index add falcosecurity https://falcosecurity.github.io/falcoctl/index.yaml
# Install k8saudit-ovh Falco plugin
sudo falcoctl artifact install k8saudit-ovh

Fill your falco.yaml file in order to add the plugin configuration:

plugins:
 - name: k8saudit-ovh
 library_path: /usr/share/falco/plugins/libk8saudit-ovh.so
 open_params: "<OVH LDP WEBSOCKET URL>" # gra<x>.logs.ovh.com/tail/?tk=<ID>
 - name: json
 library_path: libjson.so
 init_config: ""
load_plugins: [k8saudit-ovh, json]

stdout_output:
 enabled: true

Running in a Kubernetes cluster

If you have a Falco running in a Kubernetes cluster (on OVHcloud MKS or on another cluster), deployed with Helm, create a values.yaml file with the following content:

tty: true
kubernetes: false

# Just a Deployment with 1 replica (instead of a Daemonset) to have only one Pod that pulls the MKS Audit Logs from a OVHcloud LDP
controller:
 kind: deployment
 deployment:
 replicas: 1

falco:
 rule_matching: all
 rules_files:
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit-ovh
 library_path: libk8saudit-ovh.so
 open_params: "gra<x>.logs.ovh.com/tail/?tk=<ID>" # Replace with your LDP Websocket URL
 - name: json
 library_path: libjson.so
 init_config: ""
 # Plugins that Falco will load. Note: the same plugins are installed by the falcoctl-artifact-install init container.
 load_plugins: [k8saudit-ovh, json]

driver:
 enabled: false
collectors:
 enabled: false

# use falcoctl to install automatically the plugin and the rules
falcoctl:
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 indexes:
 - name: falcosecurity
 url: https://falcosecurity.github.io/falcoctl/index.yaml
 artifact:
 allowedTypes:
 - plugin
 - rulesfile
 install:
 resolveDeps: false
 refs: [k8saudit-rules:0, k8saudit-ovh:0.1, json:0]
 follow:
 refs: [k8saudit-rules:0]

This values.yaml file will install Falco with the k8saudit-ovh and the json plugins.

Install the latest version of Falco with helm install command:

$ helm install falco --create-namespace --namespace falco --values=values.yaml falcosecurity/falco

This command will install the latest version of Falco, with the k8saudit-ovh and json plugins, and create a new falco namespace.

Or if you already have Falco deployed in a Kubernetes cluster, you can use the helm update command instead:

$ helm update falco --create-namespace --namespace falco --values=values.yaml falcosecurity/falco

Once the Falco pod is ready, run the following command to see the logs:

kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco

You should see logs like that:

$ kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco

Mon Feb 10 09:15:35 2025: /etc/falco/k8s_audit_rules.yaml | schema validation: ok
Mon Feb 10 09:15:35 2025: Hostname value has been overridden via environment variable to: my-pool-1-node-921b61
Mon Feb 10 09:15:35 2025: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Mon Feb 10 09:15:35 2025: Starting health webserver with threadiness 2, listening on 0.0.0.0:8765
Mon Feb 10 09:15:35 2025: Loaded event sources: syscall, k8s_audit
Mon Feb 10 09:15:35 2025: Enabled event sources: k8s_audit
Mon Feb 10 09:15:35 2025: Opening 'k8s_audit' source with plugin 'k8saudit-ovh'
{"hostname":"my-pool-1-node-921b61","output":"09:15:40.698757000: Warning K8s Operation performed by user not in allowed list of users (user=csi-cinder-controller target=csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3/volumeattachments verb=patch uri=/apis/storage.k8s.io/v1/volumeattachments/csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3/status resp=200)","output_fields":{"evt.time":1739178940698757000,"ka.response.code":"200","ka.target.name":"csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3","ka.target.resource":"volumeattachments","ka.uri":"/apis/storage.k8s.io/v1/volumeattachments/csi-6afb06dce281b86b7bab718b5d966dc261b2b1554941ae449519a128cb2e3fb3/status","ka.user.name":"csi-cinder-controller","ka.verb":"patch"},"priority":"Warning","rule":"Disallowed K8s User","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:15:40.698757000Z"}
{"hostname":"my-pool-1-node-921b61","output":"09:15:57.508657000: Warning K8s Operation performed by user not in allowed list of users (user=yacht target=my-pool-1.18051c0a88716868/events verb=patch uri=/api/v1/namespaces/default/events/my-pool-1.18051c0a88716868 resp=403)","output_fields":{"evt.time":1739178957508657000,"ka.response.code":"403","ka.target.name":"my-pool-1.18051c0a88716868","ka.target.resource":"events","ka.uri":"/api/v1/namespaces/default/events/my-pool-1.18051c0a88716868","ka.user.name":"yacht","ka.verb":"patch"},"priority":"Warning","rule":"Disallowed K8s User","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:15:57.508657000Z"}
{"hostname":"my-pool-1-node-921b61","output":"09:15:57.807013000: Warning K8s Operation performed by user not in allowed list of users (user=yacht target=my-pool-1/nodepools verb=update uri=/apis/kube.cloud.ovh.com/v1alpha1/nodepools/my-pool-1/status resp=200)","output_fields":{"evt.time":1739178957807013000,"ka.response.code":"200","ka.target.name":"my-pool-1","ka.target.resource":"nodepools","ka.uri":"/apis/kube.cloud.ovh.com/v1alpha1/nodepools/my-pool-1/status","ka.user.name":"yacht","ka.verb":"update"},"priority":"Warning","rule":"Disallowed K8s User","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:15:57.807013000Z"}

Let's test it!

In order to test Falco we need to know which rules are installed by default. In our case, as we defined it in the values.yaml file, the k8saudit-ovh plugin follow the k8s_audit_rules.yaml file. You can take a look at them in order to know them.

In this blog post we will test one of the well-known default k8s audit rules:

- rule: Attach/Exec Pod
 desc: >
 Detect any attempt to attach/exec to a pod
 condition: kevt_started and pod_subresource and (kcreate or kget) and ka.target.subresource in (exec,attach) and not user_known_exec_pod_activities
 output: Attach/Exec to pod (user=%ka.user.name pod=%ka.target.name resource=%ka.target.resource ns=%ka.target.namespace action=%ka.target.subresource command=%ka.uri.param[command])
 priority: NOTICE
 source: k8s_audit
 tags: [k8s]

This rule is interesting because an event will be generated if/when an user execute commands in a pod.

Let’s test the rule!

In a tab of your terminal, watch the coming logs:

$ kubectl logs -l app.kubernetes.io/name=falco -n falco -c falco -f

In an another tab of your terminal, create a Nginx pod and execute a command into it:

$ kubectl run nginx --image=nginx

$ kubectl exec -it nginx -n hello-app -- cat /etc/shadow

Several seconds laters, in the logs you should see this you will see this Attach/Exec to pod logs:

...
{"hostname":"my-pool-1-node-921b61","output":"09:29:46.302906000: Notice Attach/Exec to pod (user=kubernetes-admin pod=nginx-676b6c5bbc-4xc6t resource=pods ns=hello-app action=exec command=cat)","output_fields":{"evt.time":1739179786302906000,"ka.target.name":"nginx-676b6c5bbc-4xc6t","ka.target.namespace":"hello-app","ka.target.resource":"pods","ka.target.subresource":"exec","ka.uri.param[command]":"cat","ka.user.name":"kubernetes-admin"},"priority":"Notice","rule":"Attach/Exec Pod","source":"k8s_audit","tags":["k8s"],"time":"2025-02-10T09:29:46.302906000Z"}
...

💪

Let's meet!

If you have planned to go to the KubeCon + CloudNative Con EU 2025 at London, don't hesitate to stop at the Falco booth in the Project Pavillon!

As always, we meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Enjoy 😎,

Aurélie

Blog: Falco plugin for collecting AKS audit logs

Sun, 09 Mar 2025 00:00:00 +0000

Troubleshooting Kubernetes events is challenging due to the multitude of data sources involved: container logs, Kubernetes events, cloud logs, and more. Among these sources, Kubernetes audit logs are especially valuable for identifying threats, as every action passing through the Kubernetes API server is recorded there.

We already provide plugins that let you parse and use Falco to detect threats in audit logs from GKE and EKS clusters. With our latest plugin, you'll now have the same powerful threat detection capabilities for your Azure AKS clusters.

What is Falco?

Falco is a Cloud Native Computing Foundation project that provides runtime threat detection. Out of the box, Falco examines syscalls to alert you to any suspicious activity. And, since containers share the same kernel as their host, Falco can monitor not only activity on the host but also activity on all of the containers running on that host. Moreover, Falco pulls data from both Kubernetes and the container runtime to add additional context to its alerts.

With Falco running on your GKE clusters you can be notified of a wide variety of events, such as:

Did someone start a container with high privileges?
Has someone shelled into a running container?
Has an executable been added to the container after it was deployed?

These are just a few examples. Falco has over 80 rules that can be used to make you aware of not only external threats but also when clusters aren't being operated in accordance with industry best practices.

What is the AKS audit logs plugin?

The AKS audit logs plugin extends Falco's capabilities to Microsoft Azure Kubernetes Service (AKS) clusters, providing you with the same security insights and threat detection Falco already offers for GKE and EKS environments. With this plugin, you can seamlessly integrate AKS audit logs into Falco's event processing pipeline, enabling it to identify anomalies, suspicious activities, and policy violations within your AKS-based workloads.

Using AKS audit logs plugin

In order to use the AKS audit log plugin, you must first configure your AKS cluster to ship the logs where we can fetch them.

The current supported output source is Event hub, so when following the guide to configure your AKS audit logs, you must have Eventhub enabled. You can also optionally send it to other sources:

Once you have the stream enabled, you must create or reuse a storage account blob container so that the plugin can track the last event that was consumed, which is done trough checkpoints.

Configuring Falco to use AKS audit logs plugin

First, using falcoctl, download the plugin:

sudo falcoctl artifact install k8saudit-aks

In your falco.yaml file, you must add the plugin configuration and later enable the plugin

config_files:
 - /etc/falco/config.d

watch_config_files: true

plugins:
# - name: k8saudit
# library_path: libk8saudit.so
# init_config: ""
# open_params: "http://:9765/k8s-audit"
# - name: json
# library_path: libjson.so
 - name: k8saudit-aks
 library_path: libk8saudit-aks.so
 init_config:
 event_hub_name: ${EVENTHUB_NAME}
 blob_storage_container_name: ${BLOB_STORAGE_CONTAINER_NAME}
 event_hub_namespace_connection_string: ${EVENTHUB_NAMESPACE_CONNECTION_STRING}
 blob_storage_connection_string: ${BLOB_STORAGE_CONNECTION_STRING}

load_plugins: [k8saudit-aks]

stdout_output:
 enabled: true

Once they are exported, run Falco and after some seconds you'll logs informing the k8saudit-aks plugin was loaded:

falco -c /etc/falco/falco.yaml -r /etc/falco/falco_rules.yaml

Tue Dec 17 18:02:07 2024: Opening 'k8s_audit' source with plugin 'k8saudit-aks'
2024/12/17 21:02:07 [k8saudit-aks] opened connection to blob storage
2024/12/17 21:02:07 [k8saudit-aks] opened blob checkpoint connection
2024/12/17 21:02:07 [k8saudit-aks] opened consumer client
2024/12/17 21:02:07 [k8saudit-aks] created eventhub processor

Testing out!

Append rule to falco_rules.yaml file:

- rule: K8s Audit Event Detected
 desc: A test rule that detects any Kubernetes audit event
 condition: ka.req exists
 output: "K8s Audit Event Detected: %ka.req"
 priority: DEBUG
 source: k8s_audit
 tags: [testing, k8s_audit]

$ falco -c /etc/falco/falco.yaml -r /etc/falco/falco_rules.yaml

Then, you should see initialization message, followed by some events from your AKS cluster. Since we have debug enabled, you should see some events from the aksService:

Thu Dec 19 11:44:55 2024: Falco version: 0.39.2 (aarch64)
Thu Dec 19 11:44:55 2024: Falco initialized with configuration files:
Thu Dec 19 11:44:55 2024: /etc/falco/config.d/engine-kind-falcoctl.yaml | schema validation: ok
Thu Dec 19 11:44:55 2024: /etc/falco/falco.yaml | schema validation: ok
Thu Dec 19 11:44:55 2024: System info: Linux version 6.8.0-51-generic (buildd@bos03-arm64-031) (aarch64-linux-gnu-gcc-13 (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0, GNU ld (GNU Binutils for Ubuntu) 2.42) #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec 5 13:32:09 UTC 2024
Thu Dec 19 11:44:55 2024: Loading plugin 'k8saudit-aks' from file /usr/share/falco/plugins/libk8saudit-aks.so
Thu Dec 19 11:44:55 2024: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Thu Dec 19 11:44:55 2024: Loading rules from:
Thu Dec 19 11:44:55 2024: /etc/falco/falco_rules.yaml | schema validation: ok
Thu Dec 19 11:44:55 2024: /etc/falco/falco_rules.local.yaml | schema validation: none
Thu Dec 19 11:44:55 2024: /etc/falco/falco_aks_audit.yaml | schema validation: ok
Thu Dec 19 11:44:55 2024: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Thu Dec 19 11:44:55 2024: Starting health webserver with threadiness 4, listening on 0.0.0.0:8765
Thu Dec 19 11:44:55 2024: Loaded event sources: syscall, k8s_audit
Thu Dec 19 11:44:55 2024: Enabled event sources: k8s_audit, syscall
Thu Dec 19 11:44:55 2024: Opening 'k8s_audit' source with plugin 'k8saudit-aks'
2024/12/19 14:44:55 [k8saudit-aks] opened connection to blob storage
2024/12/19 14:44:55 [k8saudit-aks] opened blob checkpoint connection
2024/12/19 14:44:55 [k8saudit-aks] opened consumer client
2024/12/19 14:44:55 [k8saudit-aks] created eventhub processor
Thu Dec 19 11:44:55 2024: Opening 'syscall' source with modern BPF probe.
Thu Dec 19 11:44:55 2024: One ring buffer every '2' CPUs.

10:52:03.348668000: Debug K8s Audit Event Detected: verb=create, user=aksService, groups=(system:masters,system:authenticated), target=<NA>

Let's meet!

As always, we meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Enjoy 😎,

Igor

Blog: Falco Talon v0.3.0

Tue, 11 Feb 2025 00:00:00 +0000

Today, we announce the release of Falco Talon 0.3.0 🦅!

Three updates in a row, after Falco and Falcosidekick, it's time for Falco Talon to know a new version.

What's new?

The key feature this release brings is the new actionner kubernetes:sysdig. For those who are not familiar with sysdig, it's a CLI tool that allows to capture and record the syscalls, like tcpdump does for the network packets. Old brother of Falco, they share the same libs and filters.

With this new integration, when a suspicious event occurs in a pod, Talon triggers a capture and then exports the created artifact to AWS S3 or Minio. You can configure the duration and the amount of bytes captured for each syscall. Check out the docs to discover more settings.

See this example rule:

- action: Capture the syscalls
 actionner: kubernetes:sysdig
 parameters:
 buffer_size: 2048
 duration: 20
 output:
 target: minio:s3
 parameters:
 bucket: falco-talon
 prefix: /sysdig/

After the action has been completed, you'll find the capture in Minio:

And you can run the CLI tool sysdig to explore it:

❯ sysdig -r 2025-01-23T13-26-41Z_default_cncf-597d69dbd4-h9fcb_sysdig.scap.gz evt.type=execve and evt.dir=">"

18563 14:26:38.376178286 0 bash (616444.616444) > execve filename=/usr/bin/apt
19163 14:26:38.394972623 0 apt (616445.616445) > execve filename=/usr/bin/dpkg
19599 14:26:38.399546432 0 apt (616446.616446) > execve filename=/usr/lib/apt/methods/http
20319 14:26:38.408846350 0 apt (616447.616447) > execve filename=/usr/lib/apt/methods/http
21775 14:26:38.453363037 0 apt (616448.616448) > execve filename=/usr/lib/apt/methods/gpgv
22335 14:26:38.461330752 0 apt (616449.616449) > execve filename=/usr/lib/apt/methods/gpgv
29434 14:26:38.481292691 0 gpgv (616451.616451) > execve filename=/usr/bin/apt-key
29604 14:26:38.486522901 0 apt-key (616453.616453) > execve filename=/usr/bin/apt-config
30183 14:26:38.494442117 0 apt-config (616454.616454) > execve filename=/usr/bin/dpkg
30422 14:26:38.497278722 0 apt-key (616455.616455) > execve filename=/usr/bin/apt-config
30996 14:26:38.504017535 0 apt-config (616456.616456) > execve filename=/usr/bin/dpkg

You can also explore the captures with Stratoshark, a GUI based on Wireshark.

Try it! 🏎️

In case you want to try out this Falco Talon 0.3.0, you can install the Helm chart following the instructions on the documentation

Let's meet 🤝

We meet every two weeks on Wednesday in our community calls, if you want to know the latest and the greatest, you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Enjoy 😎

Blog: Falcosidekick 2.31.0

Tue, 04 Feb 2025 00:00:00 +0000

The year 2025 is well started now. We saw a few days ago the first release of Falco for the year. It's to let fly out a new version of Falcosidekick, the 2.31.0.

New output

This release comes with a new output only, the last pillar of the observability with [OpenTelemetry].(https://opentelemetry.io/) that missing in Falcosidekick.

OTLP Metrics

You can now forward the Falco Events to the OpenTelemetery collector or any received understanding the protocol.

New features

Here's a non exhaustive list of the great features and enhancements which come with this new release:

Better logger

It was a ToDo for a while (even years), but it's now completed. The log system used by Falcosidekick has been replaced, without any breaking change for the users, but opening the door to more enhancements in the future.

More default labels for Loki

The log lines forwarded to Loki contain now by default the source namespace and pod name, if present in the alert. It will allow to filter more easily the events you want to display in your dashboards. Thanks to @afreyermuth98.

Payload format for Loki

Some users asked for the possibility to forward the Falco alerts in their JSON format to Loki. You can now use the setting loki.format for.

NATS/STAN subject

The template for the subject where to push the messages for NATS/STAN was hardcoded, it can now be overridden with nats/stan.subjecttemplate. See the example config file.

Fixes

Fix the missing templated fields as labels in Loki payload (PR#1091)
Fix the creation error of a ClusterPolicyReport (PR#1100)
Fix the missing custom headers for HTTP requests for Loki (PR#1107 thanks to @lsroe)
Fix the wrong key format of custom fields for Prometheus (PR#1110 thanks to @rubensf)

Conclusion

You can find the full changelog here.

The respective Helm charts are already updated and allow you to test by yourself all these great new features. Just issue the helm repo update; helm upgrade --reuse-values -n falco command to do so.

Once again, thanks to all the adopters and contributors who helped and contributed to this project all these years. We would never have reached this success without you.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falco Talon project docs.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Introducing Falco 0.40.0

Tue, 28 Jan 2025 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.40.0!

This version brings several new features, performance improvements, and bug fixes that streamline Falco’s detection capabilities. During this release cycle, we merged more than 52 PRs on Falco and more than 150 PRs for libs and drivers, version 0.20.0 and version 8.0.0 respectively. Thank you to our maintainers and contributors, as this would not happen without your support and dedication!

To learn all about these changes, read on!

What’s new? TL;DR

Key features:

Streamlined Falco docker images;
Falco static build has been reintroduced for x86_64 binary using musl;
New process filters allow to filter events based on process metadata;
Added support for sendmmsg and recvmmsg syscalls parameters;
Plugins suggested output fields are now available in the Falco engine;

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.40.0 release contains a number of features and UX improvements. Here is a list of some of the key new capabilities.

Streamlined Falco docker images

In the Falco 0.40.0 release, the Falco team has streamlined the Docker images to improve usability and performance. The new images are designed to be more efficient and easier to use, providing a better experience for users deploying Falco in containerized environments.

Key improvements include:

Reduced Image Size: The new images are smaller, which reduces the time required to pull and deploy them.
Optimized Layers: The layers in the Docker images have been optimized to improve build times and caching efficiency.
Enhanced Security: The images have been hardened to enhance security, reducing potential vulnerabilities.

These changes make it easier to deploy and manage Falco in various environments.

Introducing new process filters

A new set of process filters are made available in this release: proc.pgid, proc.pgid.name, proc.pgid.exe, proc.pgid.exepath, proc.is_pgid_leader. These filters enable users to filter events based on process metadata, such as the process name, executable path, and arguments. The new filters introduce the pgid field, which is directly obtained from the kernel. This ID corresponds to the host pid namespace, aiding in the creation of more reliable rules.

Plugins suggested output fields

The Falco engine now supports plugins that can suggest output fields. This feature allows plugins to provide additional context and information about an event, enhancing its visibility and understanding. The suggested output fields are displayed in the Falco output, giving users valuable insights into the event and its context. By leveraging this feature, Falco makes it easier for users to take advantage of the metadata provided by plugins and improve their security monitoring and incident response capabilities. New output fields are added only if the option is enabled and the plugin supports this new feature.

Keep an eye on the existing plugins to be updated to support the new feature.

Breaking changes and deprecations ⚠️

This version comes with breaking changes, mostly in the configuration interface.

Removed command line options and equivalent configuration options

We removed the already deprecated options --cri, --disable-cri-async, and is now possible to achieve the same result through the Falco configuration. A new configuration options has been introduced to enable and configure the supported container engines in Falco:

container_engines:
 docker:
 enabled: true
 cri:
 enabled: true
 sockets: ["/run/containerd/containerd.sock", "/run/crio/crio.sock", "/run/k3s/containerd/containerd.sock"]
 disable_async: false
 podman:
 enabled: true
 lxc:
 enabled: true
 libvirt_lxc:
 enabled: true
 bpm:
 enabled: true

Please keep in mind that the new configuration options are tagged as incubating and may change in the future.

You can also use the -o command line option:

--cri <socket_path>: use -o container_engines.cri.enabled=true, -o container_engines.cri.sockets[]=<socket_path, -o container_engines.cri.disable_async=true instead to enable the CRI runtime and configure the socket path and disable the async mode.

You can find more information on breaking changes in the tracking issue.

New docker images

With the growing prominence of the modern eBPF probe, in Falco 0.38.0 we made the strategic decision to adopt it as the default driver for Falco. This shift brings key advantages to our distribution system by removing the need to bundle the full driver-building toolchain in the standard Falco distribution. As a result, we’re transitioning the default Falco image to a no-driver/distroless configuration, simplifying deployments and reducing system complexity. For users seeking alternative setups, a different container image will still be available.

In light of this change, we’ve re-evaluated all Docker images:

Image Name	Tag (aliases)	Description
`falcosecurity/falco`	`x.y.z` (`latest`)	Distroless image without driver building toolchain support, based on the latest released tar.gz of Falco. No tools or `falcoctl` included.
`falcosecurity/falco`	`x.y.z-debian` (`latest-debian`)	Debian-based image without driver building toolchain support, based on the latest released Deb of Falco. May include some tools (e.g., `jq`, `curl`), but not `falcoctl`.
`falcosecurity/falco-driver-loader`	`x.y.z` (`latest`)	Based on `falcosecurity/falco:x.y.z-debian`, plus driver building toolchain support and the latest version of `falcoctl`. Recommended only when modern eBPF is unsupported.
`falcosecurity/falco-driver-loader`	`x.y.z-buster` (`latest`)	Similar to `falcosecurity/falco-driver-loader`, but based on a legacy Debian image (i.e., `buster`). Recommended only for old kernel versions.

The following images have been deprecated and are not anymore available in the Falco 0.40.0 release:

Image Name	Reason
`falcosecurity/falco-distroless`	Deprecated in favor of `falcosecurity/falco:x.y.z`.
`falcosecurity/falco-no-driver`	Deprecated in favor of `falcosecurity/falco:x.y.z-debian` (essentially the same image with a new name).

Deprecations

In Falco 0.40.0, we have deprecated the following options:

-S / --snaplen cli flag has been deprecated in favor of the falco_libs.snaplen configuration option;
-A cli flag has been deprecated in favor of the base_syscalls.all configuration option;
-b cli flag has been deprecated in favor of the buffer_format_base64 configuration option;

Worthy of note

Release artifacts are now built with zig, using very recent versions of clang. This change alone has resulted in up to 10% speedup in userspace benchmarks.

The first graph shows the events processed by userspace per second:

The following one shows the average of multiple runs of Google Benchmark framework embedded in libsinsp:

Additionally, artifacts now use jemalloc as the allocator library. This should help mitigate some memory fragmentation-related issues.

Furthermore, Falco debug symbol files are now attached to GitHub releases. Falco is built in RelWithDebInfo mode, enabling users to download debug symbols and attach them to their debugging sessions.

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

Falco is more mature with each release. Following its graduation we have published the roadmap for version 1.0.0 which is guiding us in the next steps. For the next release, you can expect more stability, a new container plugin, refinements to our deployment methods with a k8s operator, and as always new detections and fixes.

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Falcosidekick 2.30.0

Wed, 04 Dec 2024 00:00:00 +0000

A few days after a new release of Falco Talon, our response engine, it's time for our favorite proxy forwarder to do the same.

New outputs

A new release means new integrations. Thanks to our contributors for their helps.

Webex

Notify your team on Webex with the integration developed by @k0rventen.

OTLP Metrics

The adoption of Open Telemetry is bigger and bigger in the Cloud Native ecosystem, @ekoops introduced the OTLP Metrics in Falcosidekick.

Datalog Logs

The Falco alerts can be forwarded to Datadog as events for a while in Falcosidekick, you can now use their Logs service thanks to @yohboy.

New features

Here's a non exhaustive list of the great features and enhancements which come with this new release:

x3 throughput

@alekmaus spotted a bottleneck with the http client used to forward the events to the outputs. His fix increases up to 300% the throughput!!!

Better integration with Elasticsearch

@alekmaus worked hard to improve the integration with Elasticsearch. In addition improvments for the clients, new settings have been introduced, like the possibility to specify an ingest pipeline or an api key, to enable batching and compression. See the docs to know them all.

Better consistency for the Prometheus metrics

Falco recently integrated a direct endpoint to expose metrics in the Prometheus format. After a lot of discussions between the maintainers and the community, a convention has been chosen for the names of the metrics. This release adapts the metrics exposed by Falcosidekick to follow this convention and have a consistency accross the different components of the ecosystem.

Breaking changes: The renaming of the metrics might impact the queries for your alerts and dashboards.

Multi hosts for AlertManager

You can now specify a list of servers for the AlertManager output, which is a requirement when it's deployed in HA mode.

Fixes

The contributors fixed several bugs, here's a non exhaustive list of the more important ones:

Fix PolicyReports created in the same namespace than the previous event (PR#978)
Fix the missing customFields/extraFields in the Elasticsearch payload (PR#1033)
Fix the incorrect key name for CloudEvent spec attribute (PR#1051)

Conclusion

You can find the full changelog here.

Once again, thanks to all the adopters and contributors who helped and contributed to this project all these years. We would never have reached this success without you.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falco Talon project docs.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Falco Talon v0.2.0

Wed, 27 Nov 2024 00:00:00 +0000

Today we announce the release of Falco Talon 0.2.0 🦅!

Falco Talon 0.2.0 is a minor release that includes new actionners and outputs, add parameters to existing actionners, along one small fix on the check and print commands.

Features

Add gcp:function actionner:
- Now users can call GCP function to automate GCP tasks, with authentication and authorization out of the box.

- action: Invoke GCP function
 actionner: gcp:function
 additional_contexts:
 - aws
 parameters:
 gcp_function_name: simple-http-function
 gcp_function_location: us-central1

Add gcp:gcs output
- Now users can send output directly to GCP Google Cloud Storage, same way as s3 and minio existing outputs.
Add ignore_standalone_pods parameter for kubernetes:terminate actionner
Allow to wait until the completion of kubernetes:drain by configuring max_wait_period and wait_period_excluded_namespaces
Use smaller image for the kubernetes:tcpdump actionner

Fixes

An existing config.yaml file is not required anymore to check the syntax of your rules files.

Try it! 🏎️

In case you just want to try out the Falco Talon 0.2.0, you can install the helm chart following the instructions on the documentation

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest, you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Enjoy 😎,

Igor

Blog: Introducing Falco 0.39.2

Thu, 21 Nov 2024 00:00:00 +0000

Today we announce the release of Falco 0.39.2 🦅!

Fixes

Falco's 0.39.2 is a small patch release that includes some important bugfixes for modern eBPF driver:

check cred field is not NULL before the access; this enables Falco back with modern eBPF driver to work on GKE
address verifier issues on kernel versions >=6.11.4: there was a kernel-breaking change in the tail call ebpf API merged into the 6.11.4 to fix a CVE. Adapt our code to work again on these new versions.

Thanks to everyone in the community for helping us spot these annoying bugs and improving Falco every day 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.39.2, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest, you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Enjoy 😎,

Federico

Blog: How to Deploy Falco with k8s-metacollector + k8smeta Plugin

Mon, 14 Oct 2024 00:00:00 +0000

In today's cloud-native world, securing Kubernetes environments has become increasingly critical as containerized workloads gain complexity. Falco is designed to monitor and detect anomalous activities in Kubernetes clusters and container environments. By continuously observing system calls and enriching event data with metadata, Falco ensures that any suspicious behavior is detected in real-time, protecting against threats like privilege escalations, file tampering, and network anomalies.

In this tutorial, we will guide you through deploying Falco with two powerful components: k8s-metacollector and the k8smeta plugin. These tools significantly enhance Falco’s security event detection by adding important Kubernetes context, such as pod names, namespaces, deployment details, to the alerts. Additionally, we will explore how to leverage the new append_output feature introduced in Falco version 0.39.0. This feature allows you to append extra metadata fields to Falco’s output, without the need to modify your rules.

By the end of this guide, you will have a Falco setup capable of detecting security issues in Kubernetes with enriched metadata output, ensuring you get a complete picture of your cluster’s security posture. Whether you're an experienced Kubernetes administrator or just starting to explore container security, this guide will help you make the most of Falco's capabilities in a Kubernetes environment.

What You'll Learn:

The purpose and benefits of using the k8s-metacollector and k8smeta plugin to enrich Falco alerts with Kubernetes-specific data.
How to deploy Falco with the k8smeta plugin on a Kubernetes cluster.
How to configure and use the append_output feature to enhance Falco alerts with additional metadata fields.

Prerequisites:

A working Kubernetes cluster and some familiarity with Kubernetes concepts.
Basic knowledge of Falco and how it works.
Helm installed on your system (for easy deployment of Falco).

Let’s dive in and set up a Falco deployment that will give you deeper security insights for your Kubernetes workloads.

Step 1: Understanding k8s-metacollector and k8smeta Plugin

As Kubernetes has become the de facto platform for orchestrating containerized applications, it’s important to gain full visibility into what's happening within your cluster, especially when it comes to security monitoring. Falco can detect suspicious activities based on system calls, but to make these alerts more actionable, additional context about your Kubernetes resources (such as pod names, namespaces, and labels) is invaluable.

That’s where the k8s-metacollectorand k8smeta plugin come in.

What is the k8s-metacollector?

The k8s-metacollector is responsible for gathering Kubernetes metadata for security events and sending that information to Falco. It collects key information for different resources from your Kubernetes cluster, such as:

Pods;
Namespaces;
ReplicaSets;
Services;
Deployments;

The collected metadata provides greater clarity about where and why certain events are happening, which is crucial for pinpointing and mitigating security incidents in large-scale Kubernetes environments. Without this context, security alerts may lack the detail needed for quick and effective response.

What is the k8smeta Plugin?

The k8smeta plugin is a source plugin for Falco that works in tandem with the k8s-metacollector. While Falco generates alerts based on detected anomalies, the k8smeta plugin enriches these alerts with Kubernetes-specific metadata, which allows you to understand exactly which Kubernetes entities (pods, deployments, namespaces) are involved in the detected event. This context is vital when you're trying to correlate security incidents with the resources they affect.

Key benefits of the k8smeta plugin include:

Enriched Alerts: Falco alerts become more informative with Kubernetes-specific data like pod names, namespaces, and deployment names.
Improved Debugging: Knowing exactly which pod or namespace is involved in an alert can significantly reduce the time spent debugging and fixing security issues.
Event Correlation: The plugin makes it easier to correlate low-level system events with higher-level Kubernetes concepts, providing a clearer view of what's happening in your cluster.

By using the k8s-metacollector and k8smeta plugin together, you transform Falco’s raw system call data into rich, actionable insights that give you full visibility into your Kubernetes environment.

Step 2: Installing Falco, k8s-metacollector, and k8smeta Plugin with Helm and Configuring append_output

Deploying Falco along with the k8s-metacollector and the k8smeta plugin using Helm is a seamless process. This step will guide you through adding the Falco Security Helm chart repository, installing Falco, enabling the k8s-metacollector, and configuring the append_output feature to append Kubernetes metadata to Falco alerts.

Step 2.1: Add the Falco Helm Chart Repository

Before you install Falco, you need to add the official Falco Security Helm chart repository to your Helm setup. Run the following command:

helm repo add falcosecurity https://falcosecurity.github.io/charts

Update your local Helm repositories to ensure you’re using the latest chart version:

helm repo update

Step 2.2: Install Falco with k8s-metacollector and append_output

With the repository added, use the following command which includes the additional settings to enable the collection of Kubernetes metadata and to append this metadata to Falco alerts:

helm install falco falcosecurity/falco \
 --version 4.11.1 \
 --namespace falco \
 --create-namespace \
 --set collectors.kubernetes.enabled=true \
 --set tty=true \
 --set-json 'falco.append_output=[{"match": {"source": "syscall"},"extra_output": "pod_uid=%k8smeta.pod.uid, pod_name=%k8smeta.pod.name, namespace_name=%k8smeta.ns.name"}]'

Breaking Down the Command:

helm install falco falcosercurity/falco: Installs Falco using the latest chart from the Falco Security repository.
--version 4.11.1: Uses the 4.11.1 version of the chart. At the writing time it's the latest version.
--namespace falco: Deploys Falco into the falco namespace. This helps keep Falco’s resources organized separately from other applications.
--create-namespace: Automatically creates the falco namespace if it doesn’t already exist.
--set collectors.kubernetes.enabled=true: Enables the k8s-metacollector and configures the k8smeta plugin.
--set tty=true: Ensures that Falco logs are emitted as soon as possible.
--set-json 'falco.append_output=...': Configures the append_output feature to append specific Kubernetes metadata fields to Falco’s alerts.

Why Use the append_output Feature?

The append_output feature allows you to enrich Falco alerts with additional metadata, providing a clearer view of which Kubernetes resources are involved in each security event. This context helps security teams quickly understand the severity and scope of an incident.

For example, an alert will now include:

pod_uid: To precisely identify the pod.
pod_name: To know which pod triggered the alert.
namespace_name: Namespace where the pod is running.

Step 2.3: Verifying the Installation

Once the installation is complete, you can verify that Falco and the k8s-metacollector are working as expected by checking the status of the Falco pod in the Falco namespace:

kubectl get pods -n falco

You should see the Falco pods running successfully.

Step 3: Testing the Setup

Now that everything is in place, it's time to test the setup by deploying a simple Nginx pod and triggering Falco to generate security alerts enriched with Kubernetes metadata.

Step 3.1: Deploy an Nginx Pod

To create some activity that Falco can monitor, start by deploying an Nginx pod in the falco namespace:

kubectl run nginx --image=nginx --namespace falco

This command will launch an Nginx container in the falco namespace.

Step 3.2: Wait for the Nginx Pod to Run

Confirm that the Nginx pod is up and running by checking its status:

kubectl get pods -n falco

Once the pod is in the Running state, you can proceed to the next step.

Step 3.3: Exec Into the Nginx Pod to Trigger Alerts

Exec into the running Nginx pod to simulate an interactive terminal session, which is something Falco is configured to detect:

kubectl exec -it nginx -n falco -- /bin/bash

This command opens a shell session inside the Nginx container. Inside the container, run some basic commands like ls or echo to generate system calls that Falco can monitor.

Step 3.4: Check Falco Logs for Alerts

After executing inside the Nginx pod, check the Falco logs to see if any alerts were triggered by the kubectl exec action:

kubectl logs -n falco -l app.kubernetes.io/name=falco

In the logs, you should see alerts related to the interactive terminal session such as:

13:18:57.434030270: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=root user_uid=0 user_loginuid=-1 process=bash proc_exepath=/usr/bin/bash parent=containerd-shim command=bash terminal=34816 exe_flags=EXE_WRITABLE|EXE_LOWER_LAYER container_id=7cff9da475c6 container_image=docker.io/library/nginx container_image_tag=latest container_name=nginx k8s_ns=falco k8s_pod_name=nginx) pod_uid=2f20370c-6e0b-44b8-8ea1-2aa786d80f13, pod_name=nginx, namespace_name=falco

This confirms that Falco is properly configured to detect activity inside the pod and append useful Kubernetes metadata to the alerts.

Key Takeaways:

In this tutorial, we explored how to deploy Falco with the k8s-metacollector and k8smeta plugin to enhance security monitoring in a Kubernetes environment. By enabling Falco’s append_output feature, we were able to enrich security alerts with vital Kubernetes metadata such as pod UID, pod name, and namespace, making the alerts more actionable and informative.

Enhanced Alert Context: By appending Kubernetes metadata, you get more contextualized and meaningful alerts, enabling better incident investigation and faster resolution.
Seamless Integration: Thanks to Helm, deploying Falco alongside the k8s-metacollector and k8smeta plugin is easy and efficient, requiring just a few simple commands.
Real-Time Threat Detection: Falco continuously monitors system calls and Kubernetes events in real-time, ensuring that you’re always aware of potentially suspicious or malicious activities within your cluster.

Blog: Introducing Falco 0.39.1

Wed, 09 Oct 2024 00:00:00 +0000

Today we announce the release of Falco 0.39.1 🦅!

Fixes

Falco's 0.39.1 is a small patch release that includes some important bugfixes:

Fixed a crash when using plugin with event parsing capabilities (eg: k8smeta plugin)
Fixed a bug while parsing -o key={object} command line arguments, when the object definition contains a comma
Improved config json schema to allow null init_config for plugin info

Thanks to everyone in the community for helping us with spotting these annoying bugs and improving Falco every day 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.39.1, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Enjoy 😎,

Federico

Blog: Introducing Falco 0.39.0

Tue, 01 Oct 2024 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.39.0!

This version brings several new features, performance improvements, and bug fixes that streamline Falco’s detection capabilities. During this release cycle, we merged more than 50 PRs on Falco and more than 100 PRs for libs and drivers, version 0. 18.0 and version 7.3.0 respectively. Thank you to our maintainers and contributors, as this would not happen without your support and dedication!

To learn all about these changes, read on!

What’s new? TL;DR

Key features:

Basename operator retrieves the basename of a given path;
New fields added in proc and fd classes #1916 #1936;
Regular expression operator can be used to match values in string fields;
Append output allows to add output text or fields to a subset of loaded rules;
Schema validation for config and rules files allows Falco to warn users when unknown keys are used;
Improved engine selection in Kubernetes environments driver loader will automatically pick the most compatible driver for each node in the cluster.

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.39.0 release contains a number of feature and UX improvements, here are list of some of the key new capabilities.

New Operators

The basename() transformer operator extracts the base name, i.e. the filename without directory, of the input field. Note that the behavior ofbasename() in Falco is slightly different from the Unix basename program. For instance, basename (proc.exepath) will evaluate to "cat" for /usr/bin/cat but will evaluate to an empty string ("") for /usr/bin/. This allows, for instance, to write expressions like basename(proc.exepath) = cat to match against the original executable name even if it has been symlinked without knowing the full path, or any other file name based detection.

The regex operator checks if a string field matches a regular expression. Please note that the regex operator is considerably slower (up to an order of magnitude) than the above operators that work with strings, which are highly recommended for simple comparisons. The supported regex flavor is from the Google RE2 library. Example: fd.name regex [a-z]*/proc/[0-9]+/cmdline.

Introducing the Append Output Feature

In response to long-standing community requests, Falco has introduced a new feature in version 0.39.0 that allows users to add custom outputs and fields to events generated by Falco. This new functionality, called append_output, gives users greater control over the data produced by Falco rules.

With the append_output option, you can now easily add extra output to rules based on source, tag, or rule name—or even apply it to all rules without conditions. This option is configurable in the falco.yaml file and works by specifying a list of append entries, which are applied in the order they appear.

Here’s an example configuration:

append_output:
 - match:
 source: syscall
 extra_output: "on CPU %evt.cpu"
 extra_fields:
 - home_directory: "${HOME}"
 - evt.hostname

In this example, any rule with the syscall source will have the string on CPU %evt.cpu appended to the end of the default output line. Additionally, extra fields such as home_directory and evt.hostname will be visible in the JSON output under the output_fieldskey but won’t appear in the regular text output. Notably, environment variables are also supported.

This option is also available on the command line using the -o flag. For example:

falco ... -o 'append_output[]={"match": {"source": "syscall"}, "extra_fields": ["evt.hostname"], "extra_output": "on CPU %evt.cpu"}'

The introduction of append_output offers Falco users a flexible way to enrich event outputs, providing deeper visibility and customization tailored to their monitoring needs.

Dynamic Driver Selection in Falco with Helm: Simplifying Multi-Node Deployments

Deploying across diverse Kubernetes environments just got easier! When using the official Falco Helm chart and setting driver.kind=auto, the driver loader now intelligently handles the heavy lifting for you.

Here's how it works: the driver loader will automatically generate a new Falco configuration file and select the correct engine driver based on the specific node Falco is deployed on. This means whether you're using eBPF, kmod, or a modern eBPF driver, Falco will configure itself dynamically depending on the environment.

In many Kubernetes clusters, nodes can differ in terms of kernel versions, capabilities, and driver compatibility. With this new auto-selection feature, you can seamlessly deploy different Falco drivers across various nodes within the same cluster. Here’s a simple illustration:

+-------------------------------------------------------+
| Kubernetes Cluster |
| |
| +-------------------+ +-------------------+ |
| | Node 1 | | Node 2 | |
| | | | | |
| | Falco (eBPF probe) | | Falco (kmod) | |
| +-------------------+ +-------------------+ |
| |
| +-------------------+ |
| | Node 3 | |
| | | |
| | Falco (modern eBPF)| |
| +-------------------+ |
+-------------------------------------------------------+

In this example:

Node 1 is configured with the eBPF probe driver.
Node 2 uses the kmod (kernel module) driver.
Node 3 leverages the modern eBPF driver for cutting-edge performance.

This setup gives you flexibility and ensures that each node in your Kubernetes cluster is running Falco in the most optimized way possible, without manual configuration. Simply set driver.kind=auto in the Helm chart and let Falco do the rest.

Breaking changes and deprecations ⚠️

This version comes with breaking changes, mostly in the configuration interface

Removed command line options and equivalent configuration options

We removed the already deprecated options -D, -t, -T and is now possible to achieve the same result through the Falco configuration You con still use the -o command line option:

-T : use -o rules[].disable.tag=<tag> instead. Turn off any rules with a tag=<tag>. This option can be passed multiple times. This option can not be mixed with -t;
-t : use -o rules[].disable.rule=* -o rules[].enable.tag=<tag> instead. Only enable those rules with a tag=<tag>. This option can be passed multiple times;
D : use -o rules[].disable.rule=<wildcard-pattern> instead. Turn off any rules with names having the substring . This option can be passed multiple times.

You can find more information on breaking changes in the tracking issue.

Notable Bug Fixes

Prometheus Compliant metrics: some metrics have been reworked to follow the prometheus best practices #3319;
Fixed ebpf drivers to use the correct memory barrier primitive for ARM64, preventing to read incomplete data from the ring buffers #2067;
Fixed an issue where stats messages were written to stdout and could mix with regular Falco event output #3338;
fs.path fields now account for dirfd, fixing discrepancies with fd.name #1993;

Deprecations

In Falco 0.39.0, the --cri and --disable-cri-async options were deprecated, and they will be completely removed in Falco 0.40.0. Moving forward, configuring container runtimes should be done through the falco.yaml file. Below is an example of the new configuration format:

container_engines:
 docker:
 enabled: true
 cri:
 enabled: true
 sockets: ["/run/containerd/containerd.sock", "/run/crio/crio.sock", "/run/k3s/containerd/containerd.sock"]
 disable_async: false
 podman:
 enabled: true
 lxc:
 enabled: true
 libvirt_lxc:
 enabled: true
 bpm:
 enabled: true

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

Falco is more mature with each release. Following its graduation we have published the roadmap for version 1.0.0 which is guiding us in the next steps. For the next release, you can expect more stability, streamlined container images, refinements to our rule syntax, new detections and plugins.

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Introducing Falco Talon v0.1.0

Mon, 09 Sep 2024 00:00:00 +0000

More than 7 years ago, frustrated by the lack of integrations between Falco and third parties, I created Falcosidekick. The tool evolved much more than expected, with the help of dozens of contributors, individuals or for companies, to have now almost 70 different integrations, and more are coming. Its baby brother came few years later, Falcosidekick UI, helping people to visualize in real time the alerts leveraged by Falco and fine tuning their rules.

A frustation remained after all. With Falco, we have an amazing tool to detect suspicious events in our Linux hosts, VM and Kubernetes clusters, with Falcosidekick, we can easily notify our Dev/Secops, index the alerts in some SIEM, etc. But a last piece was missing: how to react to these events?

With the integrations of well known FaaS in Falcosidekick, we started a series of blog posts to show how to create from scratch what we call a "response engine". All these systems are modular, flexible, robust, but they all require a considerable amount of work from the user, to deal with the Falco payload format, the errors, the retries, the authentication to the API (AWS, Kubernetes Control Plane), the logs, the metrics, etc. Not all users and companies have the skills and/or the budget to maintain such an architecture.

To answer these needs, we designed and created Falco Talon. The the first version is officially out!.

What is Falco Talon?

Falco Talon is a Response Engine for managing threats in Kubernetes clusters. It enhances the solutions proposed by the Falco community with a no-code tailor-made solution. With easy rules, you can react to events from Falco in milliseconds.

Why did we created Falco Talon?

Over the years, the Falco community proposed different methods to react to the Falco Events, what we call a response engine.

All these methods rely on a 3rd party FaaS (Function as a Service) and come with drawbacks, all actions must be developped by the users to manage:

The errors
The Falco event format
The authentication
The K8s SDK complexity
The security
The upgrades of the dependencies
Latency
Complexity to manage sequential actions
Intrication between the function and the configuration

This is why we started to develop a custom solution specifically built for Falco: Falco Talon:

Tailor made for the Falco events
Easy to define rules
No-code implementation for end-users
UX close to Falco with the rules (yaml files with append, override mechanisms)
Allow to set up sequential actions to run
Structured logs (with a trace id)
OTEL/Prometheus Metrics and OTEL Traces

What is it good for?

React in real-time to the Falco Events
Allow fine granularity to match the events to react to
Responding to default rules with specific overrides

What is it not (yet?) good for?

Complex reaction worflows with conditions between the steps
Run actions at the host/node level through SSH (like Ansible does)

Docs

A dedicated website has been created to host the documentation: https://docs.falco-talon.org.

How Falco Talon works

As the same manner Falcosidekick works, Falco Talon receives the events from Falco by http. All you have to do to connect Falco and Falco Talon is to set in your falco.yaml:

jsonOutput: true
jsonIncludeOutputProperty: true
httpOutput:
 enabled: true
 url: "http://<falco-talon>:2803/"

If you already use Falcosidekick to forward your Falco events to the world, an integration is available since Falcosidekick 2.29.0:

talon:
 address: "http://<falco-talon>:2803/"
 checkcert: false

When the events are received by Falco Talon, an internal queue system based on NATS Jetstream is in charge to deduplicate them, to avoid to trigger the same action for the same cause for nothing.

Falco Talon will then compare the event with the rules created by the user, if an event matches with a rule, a series of actions are sequentially performed. At the end of each step, a notification with the status is sent, and a log is emmited.

Rules

The rules are the "core" of Falco Talon as they describe which actions to trigger for which Falco event.

All rules are written as yaml file, evaluated in the order they are given to Falco Talon (as arguments or in the config file), with rules specified later in the file overriding the previous ones.

The rules are composed of 2 blocks:

the action block defines which actionner to use with its parameters, this block can be imported by multiple rules (like the macros can be used in the Falco rules)
the rule block defines the criterias to match to trigger the actions

The criterias to match the event with the actions can use all elements that compose a Falco event JSON payload:

the Falco rule name
the priority
the tags
the output fields

Examples

When Falco Talon receives an event triggered by the Falco rule named Terminal shell in container, and this event doesn't concern the kubernetes namespaces kube-system and falco, then the related pod is labeled suspicious: true

- action: Label Pod as Suspicious
 description: "Add the label suspicious=true"
 actionner: "kubernetes:label"
 parameters:
 labels:
 suspicious: "true"

- rule: Terminal shell in container
 description: >
 Label the pod outside kube-system and falco namespaces if a shell is started inside
 match:
 rules:
 - Terminal shell in container
 output_fields:
 - k8s.ns.name!=kube-system, k8s.ns.name!=falco
 actions:
 - action: Label Pod as Suspicious

The action block are useful but not mandatory, the same result can be done by specifying the action in the rule block directly:

- rule: Terminal shell in container
 match:
 rules:
 - Terminal shell in container
 output_fields:
 - k8s.ns.name!=kube-system, k8s.ns.name!=falco
 actions:
 - action: Label Pod as Suspicious
 actionner: "kubernetes:label"
 parameters:
 labels:
 suspicious: "true"

Actionners

The actionners are on-catalog actions you can use. You just have to specify which one you want use to use, its parameters, and Falco Talon will manage for you all the complexity. This is how we created a no code response engine.

Within this first version, we tried to integrate as much useful actionners as possible, which allow you to manage a large variety of situations and reactions in Kubernetes.

The available actionners are:

kubernetes:terminate
kubernetes:label
kubernetes:networkpolicy
kubernetes:exec
kubernetes:script
kubernetes:log
kubernetes:delete
kubernetes:drain
kubernetes:download
kubernetes:tcpdump
aws:lambda
calico:networkpolicy
cilium:networkpolicy

To know more about what the actionners do, what parameters they require, you can read on docs/actionners.

You can notice all actionners names are composed of 2 elements x:y, the first element is the category of the actionner. All actionners in the same category share the same client, it avoid to have multi inits and instances.

Outputs

Some actionners require an output, an output is a target for the artifact created by the actionner, for example for the file retrieved by kubernetes:download or the .pcap created by kubernetes:tcpdump.

3 outputs are available today:

local:file (only useful for local tests)
aws:s3
minio:s3

The list of the available outputs can be found on docs/outputs.

Example

- rule: Redirect STDOUT/STDIN to Network Connection in Container
 match:
 rules:
 - Redirect STDOUT/STDIN to Network Connection in Container
 actions:
 - action: Run tcpdump for 5s
 actionner: "kubernetes:tcpdump"
 parameters:
 snaplen: 512
 duration: 5
 output:
 target: "aws:s3"
 parameters:
 bucket: <my-bucket>
 prefix: /tcpdump/
 region: us-east-1

Notifiers

Even we're talking about a "response engine", a framework to automatically react to some events, we still want (we humans), to be noticed of what's happening or keep traces of the performed actions.

Apart from logs output to stdout, some notifiers can be used to forward action results:

elasticsearch
k8sevents
loki
slack
smtp
webhook

The list of the available notifiers can be found on docs/notifiers.

Examples

k8sevents:

action: kubernetes:tcpdumpthought,
apiVersion: v1
eventTime: "2024-09-05T12:52:10.819462Z"
firstTimestamp: null
involvedObject:
 kind: Pod
 namespace: default
kind: Event
lastTimestamp: null
message: |
 Status: success
 Message: action
 Rule: Redirect STDOUT/STDIN to Network Connection in Container
 Action: Run tcpdump for 5s
 Actionner: kubernetes:tcpdump
 Event: Redirect STDOUT/STDIN to Network Connection in Container
 Namespace: default
 Pod: cncf-55696bc998-5xjcb
 Output: a tcpdump "tcpdump.pcap" has been created
 TraceID: c954bd8b3391a08f23079552fdd639f3
metadata:
 creationTimestamp: "2024-09-05T12:52:10Z"
 generateName: falco-talon-
 name: falco-talon-zgxfm
 namespace: default
 resourceVersion: "115862544"
 uid: 3b4bd17f-ed1a-4693-bfd7-d10f674a8008
reason: falco-talon:action:kubernetes:tcpdump:success
reportingComponent: falcosecurity.org/falco-talon
reportingInstance: falco-talon
source:
 component: falco-talon
type: Normal

slack:

A tool designed for the production

I spent 10 years of my career as a DevOps/SRE, managing traditional and cloud infrastructures, I know how painful it is to manage systems not well designed for the runtime. This is why we tried from the beginning to create a tool easy to rule all along it lifecycle.

A CLI to validate the rules

As it is for the Falco rules, the best way to manage the lifecycle of the rules for Falco Talon is to follow the GitOps principles. This requires to set up a validation of their syntax as step in the CI.

The Falco Talon binary can also be used as a CLI, allowing to perfom tasks on the rules, like checking their validity or printing their results after the merges/overrides of several files:

$ falco-talon rules check --help

Check Falco Talon Rules file

Usage:
 falco-talon rules check [flags]

Flags:
 -h, --help help for check

Global Flags:
 -c, --config string Falco Talon Config File (default "/etc/falco-talon/config.yaml")
 -r, --rules stringArray Falco Talon Rules File (default [/etc/falco-talon/rules.yaml])

Examples

With a valid rules file:

$ falco-talon rules check -c ./config.yaml -r ./rules.yaml

2024-09-05T16:42:28+02:00 INF rules result="rules file valid"

With an invalid rules file:

$ falco-talon rules check -c ./config.yaml -r ./rules.yaml

2024-09-05T16:44:01+02:00 ERR rules error="unknown actionner" action="Label Pod as Suspicious" actionner=foor:bar rule="Terminal shell in container"
2024-09-05T16:44:01+02:00 FTL rules error="invalid rules"
exit status 1

Structured Logs

The logs, whatever the component emitting them, keep always the same structure and contain a trace_id field, allowing to follow the workflow performed by Falco Talon.

The value of trace_id is also used to create the TraceId the OTEL Traces, by using a log backend like Loki, it becomes easy to correlate the traces with the logs in the same UI, like Grafana.

The CLI contains more features, take a look at them on docs /installation_usage/usage.

Example

Each step is clearly identified by the tag after the log level:

2024-09-05T14:52:03+02:00 INF event event="Redirect STDOUT/STDIN to Network Connection in Container" output=<truncated> priority=Critical source=syscall trace_id=c954bd8b3391a08f23079552fdd639f3

2024-09-05T14:52:03+02:00 INF match event="Redirect STDOUT/STDIN to Network Connection in Container" output=<truncated> priority=Critical rule="Reverse shell detected" source=syscall trace_id=c954bd8b3391a08f23079552fdd639f3

2024-09-05T14:52:10+02:00 INF action action="Run tcpdump for 5s" actionner=kubernetes:tcpdump event=test namespace=default output="a tcpdump 'tcpdump.pcap' has been created" pod=cncf-55696bc998-5xjcb rule="Reverse shell detected" status=success trace_id=c954bd8b3391a08f23079552fdd639f3

2024-09-05T14:52:10+02:00 INF notification action="Run tcpdump for 5s" actionner=kubernetes:tcpdump notifier=k8sevents rule="Reverse shell detected" stage=action status=success trace_id=c954bd8b3391a08f23079552fdd639f3

2024-09-05T14:52:11+02:00 INF output action="Run tcpdump for 5s" bucket=xxxxx category=aws file=tcpdump.pcap key=2024-09-05T14-52-10Z_default_cncf-55696bc998-5xjcb_tcpdump.pcap output="the file 'tcpdump.pcap' has been uploaded as the key 'tcpdump/2024-09-05T14-52-10Z_default_cncf-55696bc998-5xjcb_tcpdump.pcap' to the bucket 'xxxxx'" output_target=aws:s3 prefix=tcpdump/ region=us-east-1 status=success trace_id=c954bd8b3391a08f23079552fdd639f3

2024-09-05T14:52:11+02:00 INF notification action="Run tcpdump for 5s" actionner=kubernetes:tcpdump notifier=k8sevents output_target=aws:s3 rule="Reverse shell detected" stage=output status=success trace_id=c954bd8b3391a08f23079552fdd639f3

Metrics

Falco Talon exposes the traditional /metrics endpoint with metrics in the Prometheus format.

To keep a consistency, all metrics related to Falco Talon itself are prefixed with falcosecurity_falco_talon_, it follows the same convention used by Falco for its metrics.

For people interested by the metrics in the OTEL format, it's also available, see docs installation_usage/metrics

Example

# HELP action_total number of actions
# TYPE action_total counter
falcosecurity_falco_talon_action_total{action="Disable outbound connections",actionner="kubernetes:networkpolicy",event="Test logs",namespace="falco",otel_scope_name="github.com/Falco-Talon/falco-talon",otel_scope_version="devel",pod="falco-5b7kc",rule="Suspicious outbound connection",status="failure"} 6
falcosecurity_falco_talon_action_total{action="Terminate Pod",actionner="kubernetes:terminate",event="Test logs",namespace="falco",otel_scope_name="github.com/Falco-Talon/falco-talon",otel_scope_version="devel",pod="falco-5b7kc",rule="Suspicious outbound connection",status="failure"} 6
# HELP event_total number of received events
# TYPE event_total counter
falcosecurity_falco_talon_event_total{event="Unexpected outbound connection destination",otel_scope_name="github.com/Falco-Talon/falco-talon",otel_scope_version="devel",priority="Critical",source="syscalls"} 2
# HELP match_total number of matched events
# TYPE match_total counter
falcosecurity_falco_talon_match_total{event="Unexpected outbound connection destination",otel_scope_name="github.com/Falco-Talon/falco-talon",otel_scope_version="devel",priority="Critical",rule="Suspicious outbound connection",source="syscalls"} 2

OTEL Traces

We know following logs can be not really convenient, and they may lack of useful informations. You can therefore enable the emits of Traces in the OTEL format. All backends accepting this format can be used to store and visualize them.

To know how to set up the traces, see docs installation_usage/traces.

Examples

In Grafana with Tempo:

In Jaeger:

Installation

The easiest way, for now, to deploy Falco Talon is to use the Helm chart included in the repo.

with Helm

Since version 0.2.0, chart has been moved under the official falcosecurity/charts repository

The procedure to install the v0.1.0 of Falco Talon is:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update falcosecurity
helm upgrade --install falco-talon falcosecurity/falco-talon

Shoutout

I would like to shoutout some persons without the project would have been possible:

Dan Papandrea who thought about the first specs of the project with me and found the name Falco Talon
Igor Eulalio who develops Falco Talon with me, introduced amazing features like the traces, and injected so much energy in the project
Rachid Zarouali, the tester #1, a lot of features came from his ideas and feedbacks, he's also always a pleasure to present a talk about Falco Talon with him
Nigel Douglas who tests and promotes Falco Talon with talks and blog posts since the alpha stages
Carlos Tadeu Panato Júnior the magician of the CI, who still continue to manage the upgrade of the dependencies

What's next?

This first release, the v0.1.0, is just GA and it's the beginning of the journey. All your feebacks and ideas are welcome, this project has for DNA to improve the security of the Kubernetes clusters by answering real needs and usages.

The next big step to achieve is to join officially the falcosecurity organization. An issue has been created in the evolution repo to do so. Don't hesitate to vote for 🙏!

Thomas

To go further:

GitHub repo of the Falco Talon project: https://github.com/falco-talon/falco-talon
Official docs of Falco Talon: https://docs.falco-talon.org/
A record of a talk (by Rachid and Thomas) in French to introduce Falco Talon: https://www.youtube.com/watch?v=Mx28fhyKX7Q

Blog: Introducing Falco 0.38.2

Mon, 19 Aug 2024 00:00:00 +0000

Today we announce the release of Falco 0.38.2 🦅!

Fixes

Falco's 0.38.2 is a patch release that includes the most important bugfixes addressed this summer ☀️:

Fixed a crash when using transformer operators (e.g. tolower()) with a parameter that evaluates to an empty string
Fixed a bug and a regression that could result in incorrect comparison between ipv4 addresses and ipv6 subnets and vice versa
Fixed an issue that could result in missing exe_upper_layer flag
Fixed kernel module build for Linux 6.10
Fixed a bug that may result in kernel module crashes on recent versions of RHEL 9
Added additional logging to better troubleshoot hard to reproduce issues like "could not parse param ... for event ... of type ...: expected length X, found Y"

This patch also introduces a small change with the format of the new experimental Prometheus metrics to make them easier to use. Metrics are now distinguished by the file_name or rule_name labels, in line with Prometheus best practices and supporting groupBy queries.

Thanks to everyone in the community for helping us with spotting these annoying bugs and improving Falco every day 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.38.2, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Enjoy 😎,

Luca

Blog: Halfway Through GSoC 2024: My Progress and Plans with Falco

Wed, 24 Jul 2024 00:00:00 +0000

Hello Falco community, I'm Kiriti, a current GSoC mentee under Falco Security. I have been working diligently to improve the testing and benchmarking capabilities of Falco’s event-generator project. Now that we've reached the midterm of GSoC, I'm eager to share the journey so far. In this blog, I'll delve into the details of my contributions, particularly focusing on two key PRs that have been merged, and outline my plans for the remainder of the program.

My Project: Enhancing Falco's Event-Generator

The event-generator is a vital utility within the Falco ecosystem, designed to test Falco's detection capabilities. My Google Summer of Code project focuses on upgrading the event-generator to enhance its testing and benchmarking capabilities, reliability, and consistency. Additionally, I am developing new Continuous Integration (CI) pipelines based on the upgraded event-generator. The ultimate goal is to evolve the event-generator into the standard tool for systematically assessing the correctness and performance of Falco’s threat detection capabilities during every release and development cycle.

My Journey So Far:

Before being selected for GSoC, I contributed to the event-generator repository. I am grateful to Leonardo Grasso and Federico Di Pierro, who played a vital role in getting my PRs merged during the pre-GSoC contribution phase. These contributions helped me understand the event-generator codebase. I am also thankful to my mentors, Jason Dellaluce and Aldo Lacuku, for selecting me as a GSoC mentee. I will share my complete story of getting selected to GSoC in future.

After my selection, Jason, Aldo, and I collectively designed a plan to enhance the event-generator. The community bonding period was crucial in designing and understanding the implementation plan. You can view our idea here, which we will implement during this GSoC period.

Once the coding period began, we managed to merge two key PRs before the midterm. These PRs partially added support for testing Falco rules using declarative YAML files in the event-generator. We also added support for a container runner, which spawns a new container and runs the specified steps inside it. This is particularly useful for testing Falco rules that trigger when certain events are executed inside a container.

Detailed Look at the Merged PRs:

PR1: Add support for declarative yaml file testing
- what's new added in this PR?:
  - Added a new sub command for run command called declarative:
```
event-generator run declarative [yaml-file-path]
```
  - Implemented a helper function that parses the YAML file and returns the content in a specified format. The function signature is as follows:
```
 func parseYamlFile(filepath string) (declarative.Tests, error)
```
    Each yaml file structure should be in the following format
```
 type SyscallStep struct {
 Syscall string `yaml:"syscall"`
 Args map[string]string `yaml:"args"`
 }

 type Test struct {
 Rule string `yaml:"rule"`
 Runner string `yaml:"runner"`
 Before string `yaml:"before"`
 Steps []SyscallStep `yaml:"steps"`
 After string `yaml:"after"`
 }

 type Tests struct {
 Tests []Test `yaml:"tests"`
 }
```
  - Implemented a host runner A host runner is represented with the following interface
```
 type Runner interface {
 Setup(beforeScript string) error
 ExecuteStep(step SyscallStep) error
 Cleanup(afterScript string) error
 }
```
    The Setup method runs a shell script (beforeScript) before executing the specified steps using the ExecuteStep method. The Cleanup method runs a shell script (afterScript) after executing the steps.
    
    The ExecuteStep method makes some syscalls specified in the YAML file using helper functions. For example, when a write syscall is used in the YAML file steps, it runs the following write syscall helper function:
  - Added helper for making a write syscall: The function signature is as follows
```
 func WriteSyscall(filePath string, content string) error
```
PR2: Add container runner support
- To implement a container runner, we needed the ability to spawn a container and execute the events inside it. We achieved this using the Docker GO SDK.
- The container runner interface is similar to the host runner, with two new parameters: ContainerId and Image.
- The Setup method spawns a container with the given image name, saves the ContainerId, and also executes the beforeScript. The Cleanup method removes the container after executing the steps.
```
 type Runner interface {
 ContainerId string
 Image string
 Setup(beforeScript string) error
 ExecuteStep(step SyscallStep) error
 Cleanup(afterScript string) error
 }
```

Future Work:

The upcoming tasks we are going to handle are:

Implement ExecuteStep method in container runner
Add support/ helper functions to make various syscalls
Improve benchmarking capabilities of the event-generator
Integrate the event-generator in falco ci pipeline

Conclusion:

Participating in GSoC with Falco Security has been an incredible journey so far. Enhancing the event-generator has provided me with invaluable insights into cloud-native runtime security and the complexities of Falco’s detection capabilities. The support and guidance from my mentors, Jason and Aldo, through our weekly 1:1 calls, have been crucial in overcoming challenges and driving the project forward.

As I look ahead, I am excited about the upcoming tasks and the potential impact our improvements will have on the Falco ecosystem. I eagerly anticipate continuing this journey and sharing more updates on our progress. Thank you for following along!

Blog: Deploy Falco on a Talos cluster

Mon, 22 Jul 2024 00:00:00 +0000

Talos Linux is an OS designed for Kubernetes, with in mind to be secure, immutable and minimal. It offers a solution for having secure nodes for your Kubernetes cluster. Running Falco on them requires some configurations we'll see in this blog post. The good news is everything is available to collect the syscalls with eBPF and also the audit logs from the Kubernetes control plane.

In this tutorial we'll use a local Talos cluster created with Docker containers for convenience, adapt the configurations to your own context.

Requirements

For this tutorial, you'll need several tools installed:

Set up the Talos cluster

We'll start with a 2 workers cluster:

talosctl cluster create --workers 2 --wait-timeout 5m

After a few minutes, your containers and so your cluster should be up and running. You can check the status with:

talosctl cluster show --nodes 10.5.0.2,10.5.0.3,10.5.0.4

Output:

PROVISIONER docker
NAME talos-default
NETWORK NAME talos-default
NETWORK CIDR 10.5.0.0/24
NETWORK GATEWAY
NETWORK MTU 1500

NODES:

NAME TYPE IP CPU RAM DISK
talos-default-controlplane-1 controlplane 10.5.0.2 - - -
talos-default-worker-1 worker 10.5.0.3 - - -
talos-default-worker-2 worker 10.5.0.4 - - -

Get the kubeconfig

The talosctl CLI allows to easily set up your kubeconfig file for managing the apps in your fresh new cluster:

talosctl kubeconfig -n 10.5.0.2 -f

Check you have access to the cluster:

kubectl cluster-info

Output:

Kubernetes control plane is running at https://10.5.0.2:6443
CoreDNS is running at https://10.5.0.2:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Patch the cluster

When you deploy Falco with Helm in a Kubernetes cluster, an initContainer is bootstrapped to inject the eBPF probe into the kernel of each node. This behavior requires some privileges but Talos, designed to be secured, doesn't allow that by default. It's possible anyway by patching the nodes.

Create this patch.yaml file:

cluster:
 apiServer:
 admissionControl:
 - name: PodSecurity
 configuration:
 exemptions:
 namespaces:
 - falco

As you can see, we allow the pods in the namespace falco to use PodSecurity settings.

And now patch the cluster:

talosctl patch machineconfig --patch @patch.yaml --nodes 10.5.0.2

Output:

patched MachineConfigs.config.talos.dev/v1alpha1 at the node 10.5.0.2
Applied configuration without a reboot

Install Falco

We'll use Helm to deploy Falco.

Install the Helm registry for the Falco chart:

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

Create the values.yaml file:

cat << EOF > values.yaml
driver:
 kind: modern_ebpf
tty: true

falcosidekick:
 enabled: true
 replicaCount: 1
 webui:
 enabled: true
 replicaCount: 1
 redis:
 storageEnabled: false
 service:
 type: NodePort
 port: 2802
 targetPort: 2802
 nodePort: 30128

falcoctl:
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 artifact:
 install:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]
 follow:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]

services:
 - name: k8saudit-webhook
 type: ClusterIP
 ports:
 - port: 9765
 targetPort: 9765
 protocol: TCP
 name: http

falco:
 rules_files:
 - /etc/falco/falco_rules.yaml
 - /etc/falco/falco-incubating_rules.yaml
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit
 library_path: libk8saudit.so
 init_config: ""
 open_params: "http://:9765/k8s-audit"
 - name: json
 library_path: libjson.so
 init_config: ""
 load_plugins: [k8saudit, json]
EOF

Deploy Falco:

helm upgrade -i falco falcosecurity/falco -n falco --create-namespace -f values.yaml

Follow the deployment:

kubectl get pods -w -n falco

Before moving on, let's take time to explain why some of these values.

 driver:
 kind: modern_ebpf
 tty: true

We use the modern_epbf probe to collec the syscall events.
tty: true allows to get the alerts in the stdout immediatly, without any buffering.

 falcosidekick:
 enabled: true
 replicaCount: 1
 webui:
 enabled: true
 replicaCount: 1
 redis:
 storageEnabled: false
 service:
 type: NodePort
 port: 2802
 targetPort: 2802
 nodePort: 30128

We install Falcosidekick and its UI. All settings for the forwarding of the events between Falco and Falcosidekick are managed by the Helm chart.
As it's local cluster, we set the replicaCounts to 1, it loses the HA but save resources.
The UI will be exposed directly by the nodes on the port 30128, very convenient for a local cluster, prefer an ingress or just a port-forward for production.

 falcoctl:
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 artifact:
 install:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]
 follow:
 refs: [falco-rules:latest, falco-incubating-rules:latest, k8saudit-rules:latest]

falcoctl controls which plugins and rules to install and follow.
We install the stable and incubating rules for Falco
We install and follow the rules for the Kubernetes audit logs, the relevant plugins k8saudit and json will be automatically installed by falcoctl.

 services:
 - name: k8saudit-webhook
 type: ClusterIP
 ports:
 - port: 9765
 targetPort: 9765
 protocol: TCP
 name: http

The k8saudit plugin requires to create a Service listen the incoming events from the control plane.

 falco:
 rules_files:
 - /etc/falco/falco_rules.yaml
 - /etc/falco/falco-incubating_rules.yaml
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit
 library_path: libk8saudit.so
 init_config: ""
 open_params: "http://:9765/k8s-audit"
 - name: json
 library_path: libjson.so
 init_config: ""
 load_plugins: [k8saudit, json]

We load the rules for the syscalls and for the audit logs.
We load the plugins and their config. The k8saudit plugin will listen on the same port than configured in the services section.

Forward the audit logs to Falco

In a classic context, the control plane is configured to send its audit logs to an endpoint, like the k8saudit plugin. With Talos, it's not yet possible to configure this endpoint, but fortunately for us, these audit logs are written as files in the /var/log/audit/kube/ folder of the master nodes.

We'll use Fluent Bit to parse these files and forward them to the k8saudit plugin.

Install the Helm registry for the Fluent Bit chart:

helm repo add fluent https://fluent.github.io/helm-charts
helm repo update

Create the values.yaml file:

cat << EOF > values.yaml
podAnnotations:
 fluentbit.io/exclude: 'true'

daemonSetVolumes:
 - name: varlog
 hostPath:
 path: /var/log

daemonSetVolumeMounts:
 - name: varlog
 mountPath: /var/log

tolerations:
 - operator: Exists
 effect: NoSchedule

nodeSelector:
 node-role.kubernetes.io/control-plane: ""

config:
 service: |
 [SERVICE]
 Flush 5
 Daemon Off
 Log_Level warn
 HTTP_Server On
 HTTP_Listen 0.0.0.0
 HTTP_Port 2020
 Health_Check On
 Parsers_File /fluent-bit/etc/parsers.conf
 Parsers_File /fluent-bit/etc/conf/custom_parsers.conf

 inputs: |
 [INPUT]
 Name tail
 Alias audit
 Path /var/log/audit/kube/*.log
 Parser audit
 Tag audit.*
 Ignore_older true

 customParsers: |
 [PARSER]
 Name audit
 Format json
 Time_Key requestReceivedTimestamp
 Time_Format %Y-%m-%dT%H:%M:%S.%L%z

 outputs: |
 [OUTPUT]
 Name http
 Alias http
 Match *
 Host falco-k8saudit-webhook.falco.svc.cluster.local
 Port 9765
 URI /k8s-audit
 Format json
EOF

Deploy Fluent Bit:

helm upgrade -i fluent-bit fluent/fluent-bit -n kube-system -f values.yaml

To be allowed to mount the folder with the logs, we install Fluent Bit in the namespace kube-system.

Follow the deployment:

kubectl get pods -n kube-system -w -l app.kubernetes.io/name=fluent-bit

Some explanations of the values.yaml.

daemonSetVolumes:
 - name: varlog
 hostPath:
 path: /var/log

daemonSetVolumeMounts:
 - name: varlog
 mountPath: /var/log

The host folder with the logs is mounted inside the Fluent Bit pod.

tolerations:
 - operator: Exists
 effect: NoSchedule

nodeSelector:
 node-role.kubernetes.io/control-plane: ""

These settings are there to deploy Fluent Bit on the master nodes only.

config:
 inputs: |
 [INPUT]
 Name tail
 Alias audit
 Path /var/log/audit/kube/*.log
 Parser audit
 Tag audit.*
 Ignore_older true

Fluent Bit will parse the files *.logs from the folder /var/log/audit/kube/.

config:
 outputs: |
 [OUTPUT]
 Name http
 Alias http
 Match *
 Host falco-k8saudit-webhook.falco.svc.cluster.local
 Port 9765
 URI /k8s-audit
 Format json

The logs are forwarded to the endpoint falco-k8saudit-webhook.falco.svc.cluster.local:9765/k8s-audit, which is listened by the k8saudit plugin.

Visalize the alerts

Everything should be set up and running from now. You can access to the Falcosidekick-UI by the URL http://10.5.0.2:30128.

The default credentials are admin/admin.

Conclusion

Talos Linux is a more and more famous solution for creating resilient and secure Kubernetes clusters, but the trust doesn't exclude control. Mixing Talos and Falco makes you gain a step upper in term of security for your applications. Thanks to our modern eBPF probe and our k8saudit plugin, you can see how easy and quick it is to install Falco in Talos and start to observe what's happening.

Thanks to Quentin Joly for his blog post about Talos which helped me a lot to write this one.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falcosidekick UI project on GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Falcosidekick 2.29.0

Tue, 02 Jul 2024 00:00:00 +0000

Almost 1 year without a release of Falcosidekick, but version 2.29.0 is finally here. Thanks to all contributors for their patience, you made amazing contributions and we're happy to finally have them available for all users.

Like for every releases, a small recap about its adoption. Falcosidekick continues to be adopted, even if the rate is not as high as before, but we're sure it will explode once again with this new fresh version.

Once more, Falcosidekick expands Falco's integrability with a lot of new outputs. That and the introduction of many new features has been possible thanks to the hard work of the community. You can find a comprehensive list of these in the changelog.

New outputs

More and more systems are integrated as outputs in Falcosidekick, more and more often directly by the companies themselves and not their end users. It shows Falco and Falcosidekick are seen as major components in the security fields, and trusted as de facto standards.

Dynatrace

Mario Kahlhofer, aka @blu3r4y, from Dynatrace, integrated the well known observability and security platform he works for. You can even read his blog post about, to discover how to correlate the Falco events with their APM agent events.

Sumologic

Carlo Mencarelli, aka @mencarellic, did the exporter of the Falco events to Sumologic, the SaaS platform for your logs.

OTLP Traces

It started as an internal hackaton at Grafana Labs and became a real integration thanks to JuanJo Ciarlante (@jjo). You can now export the Falco event as traces, to have an automatic correlation between the detected events.

[!WARNING] It works only for the syscall related events.

Quickwit

After a demo of Falco at a CNCF Meetup, the Quickwit team wanted to add their product as a new output for Falcosidekick, and they did it. You can now easily index your Falco events in their search engine thanks to the work of Idriss Neumann (@idrissneumann).

Falco Talon

New born in the Falco ecosystem, trying to complete the last missing piece: the reaction. You can now forward the Falco events to Falco Talon, a tailor made no-code response engine for Falco. The project is still in alpha stage, but moves quickly. Stay tuned.

New features

Aside from new outputs, we introduced very important and useful new features. Let's do a recap of them.

Revamp of the Policy Report output

The Policy Report feature in Kubernetes evolved since its integration in Falcosidekick, it was the time to do some clean up. The report now contains more information, and their displays in the Policy Reporter UI is better.

New outputFieldFormat setting

Some systems perform deduplication of the events, for example the on-call platforms. They use the content of the output to do so, but the current format starting with a timestamp prevents the process to run as expected. A new setting outputFieldFormat is now available allows to "format" the output field of the Falco payload before forwarding it to the outputs.

The default format received from Falco is : <timestamp>: <priority> <output> which corresponds to this:

14:37:27.505989596: Warning Detected ptrace PTRACE_ATTACH attempt (proc_pcmdline=%proc.pcmdline evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty exe_flags=%evt.arg.flags %container.info)

By removing the <timestamp> and <priority>, you get:

Detected ptrace PTRACE_ATTACH attempt (proc_pcmdline=%proc.pcmdline evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty exe_flags=%evt.arg.flags %container.info)

If you use the settings customFields and templatedFields of Falcosidekick to inject new elements in the output_fields, it's also possible to have them in the output with the tokens <custom_fields> and <templated_fields>.

Alternative endpoints for AWS S3

Some projects like Minio are S3-compliant, you can now use them as target for the AWS S3 output by changing the endpoint to use. Thanks to @gysel for this feature.

Split of the docs

The main README of the project became really huge over the years, with all those available outputs. We did a big refactor and you can now find one file per output, with more details about the configuration, the default values and some tips. The docs are here, and any help is welcome to make them even better.

Fixes

The contributors fixed several bugs, here's a non exhaustive list of the more important ones:

Fix missing root CA for the Kafka output (thanks to @claviola)
Fix bug with the extension source in the CloudEvent output
Fix panics in the Prometheus output when hostname field is missing
Fix locks in the Loki output (thanks to @bsod90)
Fix mTLS client verification failures due to missing ClientCAs (thanks to @jgmartinez)
Fix wrong env vars for pagerduty output
Remove hard settings for usernames in Mattermost and Rocketchat
Fix multi lines json in the error lines (thanks to @idrissneumann)
Fix duplicated custom headers in clients
Fix the labels for the AlertManager output (thanks to @Umaaz)

Conclusion

You can find the full changelog here.

Once again, thanks to all the adopters and contributors who helped and contributed to this project all these years. We would never have reached this success without you.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falco Talon project docs.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Introducing Falco 0.38.1

Wed, 19 Jun 2024 00:00:00 +0000

Today we announce the release of Falco 0.38.1 🦅!

Fixes

Falco's 0.38.1 is a patch release aimed at addressing a few important bugs. It includes the following fixes:

A Falco crash while running with plugins and metrics enabled has been solved (https://github.com/falcosecurity/falco/issues/3229)
Falco -p output format option can now be passed to plugin events while -pc and -pk can only be used for syscall sources. Fixes an issue that could result in Falco exiting with LOAD_ERR_COMPILE_OUTPUT on startup with k8s clusters that had -pk and audit enabled (https://github.com/falcosecurity/falco/pull/3239)
Fixed an issue that could prevent the integer compare operators <, <=, >, >= in rules from working properly (https://github.com/falcosecurity/falco/issues/3245)
Ignore NSS user and group entries while loading users and groups (https://github.com/falcosecurity/libs/pull/1909)
Issues related to the new metric-related plugins API (https://github.com/falcosecurity/libs/pull/1885). Plugin API was also bumped to 3.6.0.
Plugin metrics are now enabled in Falco (https://github.com/falcosecurity/falco/pull/3228). Note that plugin must make use of the new metrics-related API to expose metrics.
Libs were updated to 0.17.2

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.38.1, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Introducing Falco 0.38.0

Thu, 30 May 2024 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.38.0! This is the first Falco release since its graduation within the CNCF, and, as usual, brings many improvements and features alongside some pretty big changes in its configuration mechanism.

This release brings an easier to use mechanism to install and configure your drivers, new rule language features, better support for Falco metrics and many more improvements.

During this release cycle, we merged more than 100 PRs on Falco and more than 180 PRs for libs and drivers, version 0.17.0 and version 7.2.0 respectively. Thank you to our maintainers and contributors, as this would not happen without your support and dedication!

To learn all about these changes, read on!

What’s new? TL;DR

Key features:

New capabilities in falcoctl to automatically select the best driver for your system and make it easier to install
The Falco configuration file can now be split into multiple files to make it more manageable
Rule selection from configuration file or command line
Field transformers and value comparison
Prometheus metrics support
Plugin API improvements

This release also comes with breaking changes that you should be aware of before upgrading.

Major features and improvements

The 0.38.0 release contains a number of feature and UX improvements, here are list of some of the key new capabilities.

Driver loader magic ✨

If we could pick the most common issue that we've heard from adopters and we experienced first hand is the fact that sometimes we all struggle with installing and upgrading Falco drivers. The Falco team has been tirelessly working for years to improve the installation experience and Linux kernel compatibility with massive changes such as the introduction of the new CO-RE eBPF probe and most recently the complete rewrite of our driver loading component, integrated in falcoctl. With this new version of falcoctl, integrated in Falco 0.38.0, our loading tool will automatically detect your system and pick the most compatible driver without any intervention; on recent kernel versions this is likely the modern eBPF probe. As you probably know, the modern probe does not require any extra driver download or compilation, making it load almost instantly. Of course, the tool also allows to select the preferred driver if the automatic choice is not optimal for your use case. On top of that, our driver loader tool can now automatically download kernel headers for many distributions supported by driverkit so in many cases you will be able to install even the kernel module without having to install kernel headers first. Read more about how to configure this functionality in the installation documentation page.

Organize your Falco configuration files 🗃️

Our falco.yaml configuration file gains more options, fine tuning configuration flags and feature selection for every release; in fact, they are so many that some people would like to better organize them in separate configuration files which can also be kept across Falco upgrades. Starting from this release you can add list of files or directory to the config_files configuration entry, which comes populated with the /etc/falco/config.d/ directory by default. Any additional file is read in order and can override settings in falco.yaml. Read more in the configuration options section of the documentation.

Choose which rules to load at runtime 📝

We distribute several files that contain community contributed rules and you can always write your own. But how do you select which rules Falco will load at runtime? There are several ways, including using overrides or specifying command line options such as -D, -t and -T. However, those do not allow you to express something as simple as "I would like to exclude all rules except for this one" or "I would like to include a specific tag and disable some of its rules". Furthermore, you couldn't specify this configuration in your falco.yaml file. To make this possible, we introduced a new configuration option, rules, that can be specified both in the configuration file or the command line. For instance, you can now write:

rules:
 - disable:
 rule: "*"
 - enable:
 rule: Netcat Remote Code Execution in Container
 - enable:
 rule: Delete or rename shell history

To finely control your rule loading without modifying the rule files themselves. Read more in controlling rules.

Field transformers and value comparison in conditions

Up until now we couldn't write a condition that catches operations like "a process deleting its own executable" because you couldn't use a field value on the right hand side of the condition. Since this version we have added a syntax to do just that with the val() operator:

evt.type = unlink and proc.exepath = val(fs.path.name)

will trigger only if the process exepath is the same as the unlink argument target, meaning that the process is trying to delete its own executable!

In addition you can also apply simple transform operators to both sides of the comparison: toupper() and tolower() will convert casing as you'd expect and b64() can decode base64. Stay tuned for additional transformers to cover more use cases! Read more on transform operators in the documentation.

Prometheus Metrics support 🔥

If you have been following Falco development, you probably know we are constantly improving support for metrics that tell you how the Falco engine is doing. We now have introduced Prometheus support so you can better integrate Falco with your existing performance monitoring infrastructure, and paves the way for the community to create an official Grafana dashboard that can be integrated in our charts.

Plugin API improvements ⚙️

Plugins are getting more powerful at each version. We now have a set of experimental APIs to expose metrics and read more into the Falco internal state that our expert plugin authors have been asking about. Stay tuned for more in-depth documentation on those!

Breaking changes and deprecations ⚠️

This version comes with breaking changes, mostly in the configuration interface

Changed configuration options

The syscall_buf_size_preset Falco configuration option has been replaced by engine.<driver>.buf_size_preset (e.g. engine.kmod.buf_size_preset)
The syscall_drop_failed_exit Falco configuration option has been replaced by engine.<driver>.drop_failed_exit (e.g. engine.kmod.drop_failed_exit)
The modern_bpf.cpus_for_each_syscall_buffer Falco configuration option has been replaced by engine.modern_ebpf.cpus_for_each_buffer
The syscall_event_drops Falco configuration option has been replaced by the metrics config plus some automatic notification on drops.

Removed command line options and equivalent configuration options

The --modern_ebpf command line option has been replaced by engine.kind: modern_ebpf in falco.yaml (or, on the command line -o engine.kind=modern_ebpf). Likewise, --nodriver is now engine.kind: nodriver.

The environment variable FALCO_BPF_PROBE is replaced by engine.ebpf.probe configuration option. Example:

engine:
 kind: ebpf
 ebpf:
 # path to the elf file to load.
 probe: ${HOME}/.falco/falco-bpf.o

The -e option to load capture files is no longer available. In order to read a capture file use the configuration option engine.replay.capture_file. Since options can be specified on both the command line and the configuration file, an equivalent command line as falco -e <file.scap> is falco -o engine.kind=replay -o engine.replay.capture_file=<file.scap>

The gVisor command line options have been replaced by equivalent configuration options. -g/--gvisor-config is now engine.gvisor.config while --gvisor-root is now engine.gvisor.root. Example falco.yaml configuration file:

engine:
 kind: gvisor
 gvisor:
 # A Falco-compatible configuration file can be generated with
 # '--gvisor-generate-config' and utilized for both runsc and Falco.
 config: "/etc/docker/runsc_falco_config.json"
 # Set gVisor root directory for storage of container state when used
 # in conjunction with 'gvisor.config'. The 'gvisor.root' to be passed
 # is the one usually passed to 'runsc --root' flag.
 root: "/var/run/docker/runtime-runc/moby"

Or, equivalent writing on the command line:

falco -o engine.kind=gvisor -o engine.gvisor.config=/etc/docker/runsc_falco_config.json -o engine.gvisor.root=/var/run/docker/runtime-runc/moby

You can find more information on breaking changes in the tracking issue.

Deprecations

In Falco 0.39.0 we will remove the -D, -t, -T options, continuing our tradition of removing single-character options that nobody remembers what they do.

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

Falco is more mature with each release. Following its graduation we have published the roadmap for version 1.0.0 which is guiding us in the next steps. As you can see, this version is addressing some of the roadmap points with our changes to configuration and CLI options and adding rule constructs and drivers. For the next release, you can expect more stability, streamlined container images, refinements to our rule syntax, new detections and more.

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Track the Bitcoin transactions with Falco

Wed, 13 Mar 2024 00:00:00 +0000

The number of plugins available for Falco continues to grow thanks to our wonderful community. Thank you all for your help!

You can find the list of available plugins here.

The vast majority of plugins developed allow Falco to ingest logs from different sources and raise alerts when suspicious elements are identified by its rules. In order to show that any event stream can be a source if you have the right plugin, and to have something fun to show users during my talks, I developed a Falco plugin to track Bitcoin transactions.

How does it work?

I discovered the site https://www.blockchain.com/ exposes a public flux, accessible via a websocket, by subscribing to it you can retrieve transactions carried out on the blockchain in real time. This is perfect for a Falco plugin as it allows you to test the ingestion of events via a websocket, and serve as a basis for other plugins.

I am not going to describe the internal workings of the plugin here, nor how it was developed. If you are interested, you can look at the code here.

Alternatively, read our documentation explaining how to create a plugin from A to Z: https://falco.org/docs/concepts/plugins/developers-guide/how-to-develop/.

Default rules

The plugin comes with its default set of rules, we will use them as a working example. You are free to play with it for your own needs, such as monitoring suspicious movements of your wallet.

You can find the Falco rules file provided here.

Installation of the plugin

We will see the 3 classic ways to install the plugin:

via sources
with falcoctl
in kubernetes via Helm

Via sources

The prerequisites are:

Golang >= 1.19
make
Falco >= 0.36
Git

We will start by installing download the sources, build and install the plugin:

git clone https://github.com/Issif/bitcoin-plugin.git
cd bitcoin-plugin
sudo make install

We will create a falco.yaml file containing:

plugins:
 - name: bitcoin
 library_path: /usr/share/falco/plugins/libbitcoin.so
 init_config: ''
 open_params: ''

load_plugins: [bitcoin]

stdout_output:
 enabled: true

The plugin comes with a default set of rules which will be sufficient for testing. All that remains is to start Falco with this command:

sudo falco -c falco.yaml -r rules/bitcoin_rules.yaml

14:44:21.721357000: Notice The wallet bc1qfpeps3wcmzk422hvm5jeq5lelnqlzznjwyfy69 sent 96.78318104 BTC to (bc1q4hwcl377ereljtyn2t7ljdrh9umyxz5uuyl3qn,bc1qfpeps3wcmzk422hvm5jeq5lelnqlzznjwyfy69) in the transaction aab62fd0b529cd9da163508ba879d488ff64cce4c130caf6c8bd21ab1701ed46
14:44:27.020379000: Notice The wallet bc1qwk9hqnckv0ryhsnsdefcsmlpn3zx7uq3agdsw9 sent 68.68462728 BTC to (bc1qg0nkd5nckxvwlslf6lznukgat2vukrnrrcwjcv) in the transaction 734526413f6e3eefdf4adc4258e01375ccc145b9d02b7e0ab45517be0f57e7d9
14:44:29.393013000: Notice The wallet bc1qm34lsc65zpw79lxes69zkqmk6ee3ewf0j77s3h sent 14.94446421 BTC to (3F9e4JvPryCxC5A6TS4VHeT2EJSK2ivjBV,bc1qm34lsc65zpw79lxes69zkqmk6ee3ewf0j77s3h,bc1qaeq3z2edsuspt82qw7uflg0k860clxs7qjhrh0,bc1qnvhjvpa6gaglrg9lxg7w68ye8jdjcj2nk08y20,bc1q22hp7n28whk5h94z93vm05hfx2zxs8ca9gglk7,bc1qe9yxu2myqvt3kegknzj45u704dhtapwy7lxhnv,bc1qye5rp8pcqt4ej3nsz70c3lngacmew2fc4tfljd,bc1qcfqke8as8y08mkclcun9r3hlq4xl5za2vz3n2p,bc1qtqkjq4wq234netyucg247sm6nge9qu7m2fd28g,12Q4AHgzFmKWmY1Z2LEohMoxLVhvCAKsNV,3EyJiePQX4BUt8XXaAG3JmfhwB7cQ8ggp6,bc1q36ary7yaf2eeg6006h4m33drsgw4xa3pu6yvnn,13ybpB8kTgk8bCsnRrpyemNZdE2PJSHMEs,135dx8ncZzWSjhre8ecGG1yenmLwvNZPz4,3Nzr6LAJXstT8ET2CAGMH6h5vfgrh7Q94g,bc1qjhrhwpyc0z8zh6v22vhf5arzf6vcr47tgtkj5a,bc1qpxlsyrcmwuf2rk52emvfe0dvugphzzkxlyzvxv,bc1qf23j9ls2axtl6shpry40l4qat5c695x40vpfm8,bc1qsxsdunam68jkeuu7c3mplza4h74nrjhhu9w7dl,32e54ctKqWXfzKpdNKcCBBdsRoFHKoLijH,bc1qw2gafqcg2267xm2t0r4gfzu7ff392e2vl6s3zc,bc1qc0dwh27y56yajhz0k039j5p7xkwfjprhz7rfkq,3D493LGN6PchbRPtnJQo6dSUTLB8u5vN3i,3DhzjabzhAXTBU9vksNdBZFhZzMYzK7vix,18ex2LKyiLpjaSQStY1CLNbLbSToRkJAy6,bc1q3jvuvkvpukp0mnksfmpvnqq in the transaction 40c33db54610869c75b101431690e73b584b8cc77802eea76fa2d41bbb615852
14:44:29.395043000: Notice The wallet 3Hi5VHVgmYZYfAPc9aNvQoNXyEv5rYvJQN sent 50.00000000 BTC to (bc1q582qfqtlvfv038jf6k6s6xvd30we7x66katshx,bc1q6j3gxn68m5pkzhtytn3h464kgjnvce79x8nmwq,bc1qmxaaz6g07re55ekmtlmtrc5kj0kpj3lngy5y60,1CPjdsfkqiW6LB2ZNTDYczjKCzPpiJZ4Ci,1JtUKazSgYN6hCM7HPkvzL7JLVXwkL4stN,3GzfFtGVte95ZMFfQsrz3FFgFDHU8Zw6gS,bc1qcyl4sxkczex6gxldrfmfdctr2qsun4cgpufz8j,bc1q0realpv9h4zp3yhdwjeg78njqg97f9sm6ex3xrw8mkrz8g6qamsqua6tcw) in the transaction 3025c4566dc6cd6452c0c9ae6dc8cff9583df4530326b29e38e0a5e763a6c1c9
14:44:32.577196000: Notice The wallet bc1qfpeps3wcmzk422hvm5jeq5lelnqlzznjwyfy69 sent 96.43310490 BTC to (bc1qzrzhnlaru0pqmcxwm80vvvsqpdll9g6t39y686,bc1qfpeps3wcmzk422hvm5jeq5lelnqlzznjwyfy69) in the transaction 1083e02c554454db4dcff02f7418198aae5b563c4ec286b4c3ae4d30e649e8d5
14:44:32.577917000: Notice The wallet bc1qvruk6nhq5rz7whvx9cz6peqrp3nrutae59d63q sent 13.48137244 BTC to (1EtV3erwXxeKLhCvXq1BwKit7pMcB5BDvV,bc1qxgepulgdkjju7s8el6932m57svej5uzfvx7207) in the transaction 3e000a5745d7d5b6d2791bff75b9045696c2bea497363e845593ac249cc194b5

We can clearly see transactions (sending and receiving) for amounts exceeding 1 BTC appearing in real time.

With falcoctl

The prerequisites are:

Falco >= 0.36
Falcoctl >= 0.6
Git

Falcoctl is the CLI tool that we developed to facilitate the installation of artifacts around Falco, such as rules and plugins. To find out more, here is a blog article about it.

sudo falcoctl index add bitcoin https://raw.githubusercontent.com/Issif/bitcoin-plugin/main/index.yaml
sudo falcoctl artifact install bitcoin-rules:latest

Both the plugin and the rules will be downloaded thanks to the dependency:

 INFO Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO Resolving dependencies ...
 INFO Installing the following artifacts: [ghcr.io/issif/bitcoin-plugin/ruleset/bitcoin:latest bitcoin:0.2.0]
 INFO Preparing to pull "ghcr.io/issif/bitcoin-plugin/ruleset/bitcoin:latest"
 INFO Pulling 8758e31efdff: ############################################# 100%
 INFO Pulling 326b3ec82baf: ############################################# 100%
 INFO Pulling 8aec149e9934: ############################################# 100%
 INFO Artifact successfully installed in "/etc/falco"
 INFO Preparing to pull "ghcr.io/issif/bitcoin-plugin/plugin/bitcoin:0.2.0"
 INFO Pulling e7f990e1e4e6: ############################################# 100%
 INFO Pulling 0dfca1bb2434: ############################################# 100%
 INFO Pulling f269eb62cbf6: ############################################# 100%
 INFO Artifact successfully installed in "/usr/share/falco/plugins"

As with the installation via sources, the falco.org file should look like:

plugins:
 - name: bitcoin
 library_path: /usr/share/falco/plugins/libbitcoin.so
 init_config: ''
 open_params: ''

load_plugins: [bitcoin]

stdout_output:
 enabled: true

And Falco will be started by the command:

sudo falco -c falco.yaml -r /etc/falco/bitcoin_rules.yaml

In Kubernetes via Helm

The prerequisites are:

Helm

The installation will consist of just adapting the values in the values.yaml file. Everything will be automatically managed by the templates:

tty: true
kubernetes: false

falco:
 rules_files:
 - /etc/falco/bitcoin_rules.yaml
 plugins:
 - name: bitcoin
 library_path: libbitcoin.so
 load_plugins: [bitcoin]

falcosidekick:
 enabled: true
 webui:
 enabled: true

driver:
 enabled: false
collectors:
 enabled: false

controller:
 kind: deployment
 deployment:
 replicas: 1

falcoctl:
 config:
 indexes:
 - name: bitcoin
 url: https://raw.githubusercontent.com/Issif/bitcoin-plugin/main/index.yaml
 artifact:
 install:
 refs: ["bitcoin:0"]
 follow:
 refs: ["bitcoin-rules:0"]

And the classic Helm command for installation:

helm install falco-bitcoin -n falco falcosecurity/falco -f values.yaml --create-namespace

After a few seconds, you should have the pod running:

❯ kubectl get pods -n falco -l app.kubernetes.io/instance=falco-bitcoin
NAME READY STATUS RESTARTS AGE
falco-bitcoin-7474fbfcb5-srgsg 2/2 Running 110 (17m ago) 10d

And new events in falcosidekick-ui:

Conclusion

This plugin has no great purpose other than to dismantle the almost infinite possibilities that open up to Falco thanks to its plugin system. If you wish to be alerted on Telegram of a strange outgoing movement from your wallet, it is now possible with Falco!

Falco is no longer limited to securing Cloud environments. SaaS or others can also be used in a unified way. The Falco rules syntax has proven to benefit security practitioners in an ecosystem rich with numerous potential integration points.

Blog: Preventing attacker persistence with Falco on AWS

Mon, 11 Mar 2024 00:00:00 +0000

I recently read an interesting blog on how hackers could use a Lambda function alongside Lex to establish persistence in an AWS account. For those unfamiliar with the term, persistence is when attackers leverage some technique to retain access to systems without being detected. A recent news article cited a study that reported that some Chinese hackers have lurked in systems for up to FIVE YEARS! Luckily for all of us, Falco can be used to detect the exact scenario detailed in the blog and immediately raise an alert.

The blog detailed how an attacker who has gained access to an AWS account could modify an existing Lex-based Lambda function to provide a set of AWS credentials. In short, the attacker modified the function of the Lex-based chatbot to respond whenever a secret phrase was entered with the Lambda’s AWS key ID and secret key.

The author notes there are several ways to establish persistence on AWS, and maybe this wasn’t the most practical, but I still found it a fun exercise. It got me thinking: How could Falco help here? My immediate thought was to use Falco’s AWS Cloudtrail plugin.

The plugin, as the name implies, ingests Cloudtrail events. The events can be evaluated against a set of rules to alert engineers of any suspicious activity. There are currently just over 20 different rules that can be assessed. They include scenarios like creating new users, having someone log into the root account without MFA, changing permissions on an S3 bucket, and, most relevant to our discussion here, modifying a Lambda function.

- rule: Update Lambda Function Code
 desc: Detect updates to a Lambda function code.
 condition:
 ct.name="UpdateFunctionCode20150331v2" and not ct.error exists
 output:
 The code of a Lambda function has been updated.
 (requesting user=%ct.user,
 requesting IP=%ct.srcip,
 AWS region=%ct.region,
 lambda function=%ct.request.functionname)
 priority: WARNING
 tags:
 - cloud
 - MITRE_TA0003_persistence
 - aws_lambda
 source: aws_cloudtrail

Alternatively, you could create custom rules based on a whole list of AWS CloudTrail events. In the below Falco rule, we referenced the Cloudtrail event name UpdateFunctionCode20150331v2. Over 3000 event names can be used with Falco for deep incident response and forensics in the cloud.

So, in the scenario above, whenever the attacker modifies the function, an entry is written to the Cloudtrail logs (note that Cloudtrail is enabled by default, so no extra work is needed to get it running). Those logs would be immediately forwarded to Falco via the Cloudtrail plugin. Falco would evaluate the event against the ruleset and fire off an alert that the rule had been violated.

 The code of a Lambda function has been updated.
(requesting user=mikegcoleman, requesting IP=10.0.01, AWS region=us-west-1,lambda function=AirlinesBusinessLogic)

Upon receiving the alert, an AWS engineer could review whether or not the change had been authorized and act accordingly.

Now, it’s important to note that there are other ways that Lambda functions can be stored. For instance, the function's code can be zipped up and then stored in an S3 bucket. The rules we used above wouldn't cover this scenario. You could very likely craft a rule to ensure that a bucket’s contents have not been modified. We will save that for another blog.

The Falco plugins repository includes the AWS Cloudtrail plugin and plugins for Okta, GitHub, Google Cloud, and more. In fact, I recently published a walkthrough on how to use the Google Cloud plugin. Be sure to check it if you’d like to learn more about how plugins are installed and configured.

Blog: Falco Weekly 10 - 2024

Fri, 08 Mar 2024 00:00:00 +0000

What happened in Falco this week?

First of all, you probably already heard it, Falco is now graduated!
If you missed this important news, go ahead and give our graduation blog post a read!

Let's go through the major changes that happened in various repositories under the falcosecurity organization during the last week.

Libs

We are approaching the 0.15.0 tag, therefore mostly bugfixes were merged, plus a great new feature and some refactors:

The so-called "kmod configure system" was finally merged: https://github.com/falcosecurity/libs/pull/1452. This helps us to ensure that our kernel module builds even when some features get backported from more recent kernels (ie: when checking for kernel release version in the code is not enough). Kudos to Angelo Puglisi for shipping such a feature! Also, keep an eye for the very same thing for bpf too: https://github.com/falcosecurity/libs/pull/1729! Thanks to the kmod configure system, our kernel-testing matrix is now fully green for kmod!
A big CRI API refactor finally landed: https://github.com/falcosecurity/libs/pull/1600
Proceeding with the journey around compiler sanitizers, we now have proper cmake options to enable ASAN and UBSAN: https://github.com/falcosecurity/libs/pull/1721
Fixed a crash when reading proclist from scap: https://github.com/falcosecurity/libs/pull/1726
Fixed some socketpair fds problems: https://github.com/falcosecurity/libs/pull/1733
Fixed and added some more tests: https://github.com/falcosecurity/libs/pull/1736, https://github.com/falcosecurity/libs/pull/1727

Falco

Some small changes happened too, in Falco main repository:

The proposal about features adoption and deprecation was merged: https://github.com/falcosecurity/falco/pull/2986!
Added a new configuration key falco_libs.thread_table_size to customize max thread table size in libsinsp: https://github.com/falcosecurity/falco/pull/3071
Throw an error when an invalid macro/list name is used: https://github.com/falcosecurity/falco/pull/3116
Fixed up directory iteration options while iterating over rule folder: https://github.com/falcosecurity/falco/pull/3127

Join relevant discussions

Our new discussion section: https://github.com/falcosecurity/falco/discussions
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840
Breaking changes in Falco 0.39.0: https://github.com/falcosecurity/falco/issues/3045

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list
Open a discussion in our discussion section

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Falco Graduates within the CNCF!

Thu, 29 Feb 2024 00:00:00 +0000

Today, the Falco project hit a big milestone: becoming a CNCF Graduated Project! Falco's graduation indicates the project's maturity and dependability, but most importantly, it is the culmination of a fantastic amount of work.

The journey for Falco started in 2016 when the first commit was made. Today, Falco has become synonymous with "runtime security" due to its comprehensive approach to securing the highly complex and dynamic environments of the modern cloud era.

“Falco approaches the security challenges associated with managing cloud native environments holistically,” says Loris Degioanni, the project’s founder. ”Runtime security is more than eBPF-based data collection, it requires enrichment, orchestrator integration, correlation of different data sources, and a rich, well-maintained policy library. All of the things that Falco provides to its users.”

This holistic, runtime-centric approach to security is what makes Falco unique. It enables any organization to secure their infrastructure — from scrappy startups to Fortune 500s. Since joining the CNCF, some of the largest enterprises in the world, including Amazon, Apple, IBM, and Red Hat, have contributed to Falco. The project also has a strong, rapidly growing community of adopters and has been downloaded over 100 million times!

To the thousands of people who have helped Falco fly over the past years, thank you — sincerely. Thank you to the Falco maintainers. Thank you to our CNCF Technical Oversight Committee sponsors, Emily Fox and Justin Cormack. Thank you to anyone who has ever raised an issue, submitted a pull request on GitHub, or just took part in our community.

Thank you, also, to the thousands of organizations who have entrusted Falco with the security of your runtime environments.

For us, Falco’s Graduation represents a calling to continue to improve the project in a way that serves its users. We believe that runtime protection is vital to security, and that Falco is well positioned to power that security as we move deeper into the cloud era.

In the future, Falco will have even stronger detections, richer signals, lower noise, and better performance. Its breadth of coverage will increase with more data sources, including cloud logs and key developer touchpoints like GitHub. Future versions of Falco will be even easier to deploy and manage in production.

Our mission is to make Falco a powerful companion that brings you peace of mind, knowing your cloud native apps are well looked after. We hope you will continue to be part of this journey with us.

Words from our community

Graduation means a lot to the members of our community. We asked them to share their thoughts and feelings with us on this occasion. Here is what they had to say:

"Since joining the Falco project in 2020, I've been inspired by our community's growth and commitment to open source values. Falco has unequivocally established itself as the quintessential tool for cloud native runtime security, leveraging key technologies like eBPF – notably, becoming one of the largest open source eBPF codebases. As we reach the CNCF graduation milestone, I'm immensely proud of our collective achievements and want to thank every contributor who has played a role in this journey." - Leonardo Grasso, Falco core maintainer

"Linux kernel security monitoring is undeniably mission-critical, in great demand, yet daunting to master. Working within the kernel can be intimidating due to its potential impact on application performance and the sheer volume of events on modern servers. Since joining in 2022, The Falco Project has adapted to meet new demands while staying true to its mission, and this journey continues to accelerate. Observing Falco's effectiveness and value in real-world production settings is truly beautiful." - Melissa Kilby, Falco core maintainer

"Back when I joined the Falco community, the project was a teenager that needed some love; here we are, it took a little while but it is now an adult! And I loved every little bit of its growth! What do I love more? The fact that we still have a lot of space for improvements, everywhere. This is good for users, the wider community and for us, developers and maintainers of the projects, to keep the fun with it." - Federico Di Pierro, Falco core maintainer

"It has been a long journey since my first try of Falco. It was its very first release, Kubernetes was a small thing and the containers started to become the game changer we know now. I'm very proud having been a modest piece of this achievement, developing tools for Falco made me a better DevOps, a better Go developer, an international speaker and it made me meet amazing users and contributors. It's a good thing to see it’s rising as a standard for the runtime security in the industry. I hope it will help even more SREs to peacefully sleep at night." - Thomas Labarussias, Falco core maintainer

"The project's rapid growth is evident with each new milestone, and I take great pride in being a member of this team, of this family. There's boundless potential for the project's expansion, and I see this milestone as the first step toward an even brighter future." - Andrea Terzolo, Falco core maintainer

"I have been following Falco for many years now and I am impressed by how far the project has come. I am personally proud and happy that I could be a part of the stellar team that drives Falco; thanks to maintainer's and contributor's efforts we were able to achieve incredible goals and it is great to see the project being recognized alongside the most successful in the CNCF. Thanks to everyone who has been with us in this journey and everyone who will join us in the future." - Luca Guerra, Falco core maintainer

"It's hard to believe that more than two years have already passed since I joined the amazing open source community of Falco. I can't express my deep gratitude for all our supporters and for the Falco family. This project gave me the privilege of connecting with incredibly skilled humans, and of witnessing the growth of a beautiful piece of technology that's now a fundamental security asset for countless organizations in the industry. Looking back, I feel immensely proud of all the collective efforts that led the project to this huge milestone, and I can't wait to see what the future holds from this point forward." - Jason Dellaluce, Falco core maintainer

"Today, I am immensely proud as Falco graduates within the CNCF. This achievement is a testament to the dedication of our maintainers and the broader community of adopters and contributors. Together, we’ve propelled Falco to become the industry’s de-facto standard for runtime threat detection in the cloud. I am deeply grateful for the collaborative efforts that have brought us to this moment. The tireless work, expertise, and passion shared by our team and community have elevated Falco’s capabilities, ensuring its effectiveness in safeguarding organizations against evolving security threats. As we enter this new chapter within the CNCF, I am confident in our collective ability to continue innovating and strengthening Falco’s position as a vital tool in the fight for cloud security." - Michele Zuccala, Falco core maintainer

"Since I joined, contributing to the ecosystem with passion and excitement to know more about and to contribute to this project, I've found a welcoming, vibrant, and healthy community as well as a strong maintainership. With time, the community grew and grew, and independence and diversity increased during these years. Nowadays, runtime security in the cloud native world is becoming more and more fundamental in our architectures, and Falco has become one of the de-facto standards for increasing observability in our Linux kernel-based cloud native systems. The project has evolved a lot during the last couple of years. I remember the design proposal for plugins back then! Thanks to the incredible work of the maintainers and the contributors, and now I can't explain how much this big step and acknowledgment from the CNCF matter! Congratulations to the Falco family!" - Massimiliano Giovagnoli, Falco core maintainer

Blog: Falco Weekly 7 - 2024

Fri, 16 Feb 2024 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

Multiple fixes and some cleanups happened in the libs repo:

newfstatat syscall is now configured with UF_ALWAYS_DROP: https://github.com/falcosecurity/libs/pull/1683
Fixed null destination address in sendto and sendmsg in modern bpf: https://github.com/falcosecurity/libs/pull/1687
Added a CT_UNKNOWN container type zero value and properly initialize uninitialized value: https://github.com/falcosecurity/libs/pull/1688
Fix in chisels: don't fail if a chisel directory does not exist: https://github.com/falcosecurity/libs/pull/1689
Cleaned up more memory reads/writes in filterchecks to avoid UBs: https://github.com/falcosecurity/libs/pull/1690
Properly initialize m_exe... fields in threadinfo: https://github.com/falcosecurity/libs/pull/1691
Fixed a small source of memleak in scap platform: https://github.com/falcosecurity/libs/pull/1692
Properly enforce the static CRT on Windows by default: https://github.com/falcosecurity/libs/pull/1695

Falco

Falco has seens quite a bit of C++ improvements, thanks to Samuel Gaist! Keep up the great job!

C++ cleanups: https://github.com/falcosecurity/falco/pull/3069, https://github.com/falcosecurity/falco/pull/3074, https://github.com/falcosecurity/falco/pull/3083, https://github.com/falcosecurity/falco/pull/3085
Consolidated Faclo engine and rule loader tests: https://github.com/falcosecurity/falco/pull/3066
Added http-headers option to Falco driver-loader images: https://github.com/falcosecurity/falco/pull/3075
Cleaned up an unused builder Dockerfile: https://github.com/falcosecurity/falco/pull/3088
Fixed some compiler warnings: https://github.com/falcosecurity/falco/pull/3089
Cleaned up falco_engine deps and include paths: https://github.com/falcosecurity/falco/pull/3090

Falcoctl

Falcoctl has seen a small yet important fix:

Correctly report artifact type: https://github.com/falcosecurity/falcoctl/pull/442

Kernel-testing

Even if the effort was part of last week, and since we skipped last "Weekly Recap", it is important to mention that the kernel-testing framework recently got a big update:

All images build is now tested in PR CI when they are modified
Images are now build and published on ghcr.io/falcosecurity/kernel-testing repo
They are published under main tag and under latest|$tag for releases
The image name is built as: $distro-{kernel,image}:$kernelrelease-$arch-$imagetag, eg: amazonlinux2-kernel:5.10-x86_64-v0.3.2
Ubuntu-6.3 images were bumped to 6.5 kernel
A new arch-6.7 image was added to the test matrix
A composite action was added and is now used by libs CI

As always, you can find detailed kernel-testing outputs against our drivers under https://falcosecurity.github.io/libs/matrix/.

Charts

Thanks to Aldo's continuous effort, we now have much better documentation all around the repo:

Updated docs for Falco exporter: https://github.com/falcosecurity/charts/pull/623,
Process all charts for changes in values.yaml: https://github.com/falcosecurity/charts/pull/624
Updated contributing section: https://github.com/falcosecurity/charts/pull/625
Fixed typos, formatting and dead links in Falco chart docs: https://github.com/falcosecurity/charts/pull/627
Fixed dead links for Falco exporter: https://github.com/falcosecurity/charts/pull/628
Fixed link tags in readme: https://github.com/falcosecurity/charts/pull/629

Join relevant discussions

Our new discussion section: https://github.com/falcosecurity/falco/discussions
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840
Breaking changes in Falco 0.39.0: https://github.com/falcosecurity/falco/issues/3045

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list
Open a discussion in our discussion section

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Introducing Falco 0.37.1

Tue, 13 Feb 2024 00:00:00 +0000

Today we announce the release of Falco 0.37.1 🦅!

Fixes

Falco's 0.37.1 release is a small patch aimed at addressing a few minor bugs. It includes the following:

Added --http-insecure flag to driver loader images
Added new env variable FALCOCTL_DRIVER_HTTP_HEADERS understood by driver loader images to pass a comma separated list of http headers for driver download, eg: FALCOCTL_DRIVER_HTTP_HEADERS='x-emc-namespace: default,Proxy-Authenticate: Basic'
Falcoctl was bumped to v0.7.2, fixing an issue building Flatcar drivers and a bug withing the kernel release fixup method to build drivers download URLs
Fixed a nasty bug that caused Falco to crash when a priority higher than debug was set in the config: https://github.com/falcosecurity/falco/pull/3060
Libs were updated to 0.14.3

Last, but not least, as recommended by the CNCF, we now link libelf dynamically instead of statically, so that the library remains separable from Falco at runtime.
This has multiple outcomes:

Falco static (musl) build is disabled for now; we are experimenting with some solutions and we will hopefully be able to bring it back up soon
Users of docker images won't notice anything since they already shipped libelf library
Users of deb and rpm packages won't notice anything since libelf was already a nested dependency
Users of the tar.gz package will need to manually install libelf where not present

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.37.1, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Automate Kubernetes Network Security with Falco Talon

Fri, 09 Feb 2024 00:00:00 +0000

Falco Talon Repository: https://github.com/Falco-Talon/falco-talon
Falco Talon Documentation: https://docs.falco-talon.org/

Falco Talon is currently under active development and remains in the alpha stage; therefore, breaking changes may occur at any time, and the documentation may not always be up to date.

Setting up robust network security in Kubernetes is a challenge that demands both precision and adaptability. NetworkPolicy offers the potential for highly specific network configurations, enabling or blocking traffic based on a comprehensive set of criteria. However, the dynamic nature of network topologies and the complexities of managing policy implementations present ongoing challenges. The need for constant policy updates, especially in response to changing threat landscapes, introduces risks such as the potential for misconfiguration and the unintended dropping of packets.

The Challenge of IP-Based Network Policies

Building network policies around IP addresses is notoriously challenging. For instance, threat feeds, which list known malicious IP addresses, are constantly changing. An IP address associated with a malicious entity one week might be reassigned and deemed safe the next. This fluidity necessitates an agile approach to network policy management, integrating solutions like NetworkSets to dynamically update policies based on the latest intelligence. However, the sheer volume of threat intelligence feeds – from Tor IP lists to cryptomining blocklists – complicates this integration, making it a daunting task to maintain accurate network controls.

Here, Falco Talon emerges as a transformative solution. By leveraging Falco's detection capabilities, such as identifying Outbound Connections to C2 Servers, Falco Talon can instantly update Kubernetes network policies to block all egress traffic except allowed CIDR ranges. This is facilitated through the kubernetes:networkpolicy Talon action, demonstrating a seamless integration of dynamic threat detection with network policy enforcement.

- action: Disable outbound connections
actionner: kubernetes:networkpolicy
parameters:
allow:
- "192.168.1.0/24"
- "172.17.0.0/16"
- "10.0.0.0/32"
- rule: Suspicious outbound connection
match:
rules:
- Outbound Connections to C2 Servers
actions:
- action: Disable outbound connections

While this would certainly block the egress network connections, it’s not really convenient for some organizations as it’s a scorched-earth approach to blocking traffic but still relies on an IP-based allowlist on each Talon rule - rather than blocking the actual suspicious IP address specified in the Falco rule. Instead, we will propose the use of labels as a response action in Talon to better isolate network traffic at runtime.

Shifting from IPs to Labels for Network Security

While IP-based blocking is effective in response to specific threats detected by Falco, it's not the most scalable solution for ongoing network policy management in production environments. An alternative approach focuses on using labels to create quarantine-style network policies. This method involves configuring a network policy that applies a default-deny stance on all ingress and egress traffic for pods matching certain labels, effectively isolating potentially compromised workloads. This can easily be achieved with the below one-liner:

kubectl label pod <pod-name> -n <namespace-name> quarantine=true

This is certainly a cleaner approach than the previous enforced network policy implementation, but the challenge here is the manual process of labeling those suspected workloads, which can be cumbersome and slow in response to emerging threats. How many minutes will it take our security team to enforce this label action, and what happens if this happens over the weekend?

Falco Talon addresses this gap with its kubernetes:labelize response action. Upon detecting a threat, such as the Detect outbound connections to common miner pool ports, Talon can automatically apply a quarantine:true label to the affected pod, triggering the enforcement of the quarantine network policy in real-time. This capability not only enhances the speed and efficiency of response actions but also underscores the power of integrating dynamic threat detection with network policy enforcement.

- action: Quarantine Pod in Network Policy
actionner: kubernetes:labelize
parameters:
labels:
quarantine: true
- rule: Suspicious outbound connection
match:
rules:
- Detect outbound connections to common miner pool ports
actions:
- action: Quarantine Pod in Network Policy

While Talon can apply the label, you still need a component that will enforce the quarantine. Most CNI's can do this, but for the purpose of this blog I'll add an example with Calico:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: quarantine
spec:
selector: quarantine == "true"
ingress:
- action: Deny
source: {}
destination: {}
egress:
- action: Deny
source: {}
destination: {}
types:
- Ingress
- Egress

Conclusion

The integration of Falco Talon into Kubernetes network security strategies represents a significant advancement in the field. By automating the enforcement of network policies based on real-time threat detection, Falco Talon simplifies the complexities associated with managing network security in a constantly evolving landscape. Whether responding to immediate threats through IP-based policies or proactively isolating workloads with label-based quarantine policies, Falco Talon provides a flexible, powerful tool for enhancing the security and resilience of Kubernetes environments. As organizations navigate the challenges of cloud-native security, solutions like Falco Talon offer a beacon of adaptability and effectiveness, ensuring that network security keeps pace with the dynamic nature of containerized deployments.

Blog: Introducing Falco 0.37.0

Tue, 30 Jan 2024 00:00:00 +0000

Dear Falco Community, today we are happy to announce the release of Falco 0.37.0!

This release brings an improved installation experience, a new way to modify Falco rules, and some great UX improvements. There are, as to be expected, a handful of breaking changes. But, rest assured, we've done all we can to help you with any changes you might need to make.

During this release cycle, we merged more than 100 PRs on Falco and more than 160 PRs for libs and drivers, version 0.14.2 and version 7.0.0 respectively. Thank you to our maintainers and contributors, as this would not happen without your support and dedication!

To learn all about these changes, read on!

What’s new? TL;DR

Key features:

A new way to enrich syscalls with Kubernetes metadata, replacing the old Kubernetes collector.
New capabilities in falcoctl to download and build our kernel drivers, replacing the old falco-driver-loader script.
Support for 32-bit syscall emulation on x86_64 in all kernel drivers (modern_ebpf, ebpf, kernel module).
A new override key to easily modify rules, lists, and macros.

Key UX improvements:

Introduction of a new engine key in falco.yaml to replace all other methods for opening engines such as FALCO_BPF_PROBE, --modern-bpf, -g, and -e.
Expansion of environment variables in falco.yaml even when they are part of a string.

This release also comes with breaking changes, we'd suggest to read them before upgrading. If you use helm, make sure to read the Helm chart breaking changes page as well.

Major features and improvements

The 0.37.0 release contains a number of feature and UX improvements, here are list of some of the key new capabilities.

A new way to enrich syscalls with K8s metadata

Falco 0.37.0 introduces a new method to enrich syscalls with Kubernetes metadata to help address scalability and other issues with the old collector. Falco always had Kubernetes support, but sometimes we need new approaches to keep up with the bigger and bigger scale that we see in production clusters today. You can find more technical details here.

While the collector was previously integrated into Falco, this feature uses a new architecture which leverages a plugin (k8smeta) and a remote collector (k8s-metacollector).

The plugin gathers details about Kubernetes resources from the remote collector. It stores this information and provides access to Falco upon request. The plugin specifically acquires data for the node where the associated Falco instance is deployed, resulting in node-level granularity. In contrast, the collector runs at the cluster level.

Within a given cluster there may be multiple k8smeta plugins (one per node), but only one collector exists per cluster.

More technical details about the architecture and design choices are here.

It’s important to note that both new components are considered experimental, which means although they are functional and tested, they are currently in active development. They may undergo changes in behavior as necessary without prioritizing backward compatibility.

Fields supported by the new `k8smeta` plugin

This section provides details on:

Kubernetes fields that are supported out-of-the-box by Falco through container runtime enrichment.
Fields the new k8smeta plugin supports
Fields have been deprecated.

The following fields are automatically populated with data from the container runtime, making them compatible with Falco without needing the old k8s collector or the new k8smeta plugin. These fields will continue to function as before, and no changes have been made:

k8s.pod.name
k8s.pod.id/k8s.pod.uid
k8s.pod.sandbox_id
k8s.pod.full_sandbox_id
k8s.pod.label
k8s.pod.labels
k8s.pod.ip
k8s.pod.cni.json
k8s.ns.name

All other fields with the k8s.* prefix previously supported by the old collector (e.g., k8s.deployment.name) are now deprecated and will return <NA> if used in rules.

These fields are now provided by the new plugin under the k8smeta.* prefix. A complete list of these fields can be found here

The new fields introduced by the k8smeta plugin are additive. They do not replace the fields provided by the container runtime. This means you can use both k8s.pod.name and k8smeta.pod.name simultaneously. While they may return the same value, the data is collected from different sources (container runtime for k8s fields, the Kubernetes API server for k8smeta). As a result, their availability and reliability may differ during the lifecycle of an application. While it may seem redundant, this approach should offer flexibility to users.

To wrap up:

If k8s.pod.* and k8s.ns.name fields meet your needs, you can use Falco without plugins. The default container runtime information in Falco should be enough.
If k8s.pod.* and k8s.ns.name fields are insufficient, you should evaluate the new k8smeta plugin.
The old k8s.* fields (excluding k8s.pod.* and k8s.ns.name) are now deprecated, and if used in Falco rules, they return <NA>

If you’d like to read more about this new feature check out the documentation for the k8smeta pluginand the k8s-metacollector, while if you want to deploy this solution with our helm chart check out the dedicated section.

New Falcoctl capabilities

Since falcoctl 0.7.0, users have been able to quickly download and compile Falco drivers using the falcoctl driver command. Starting with Falco 0.37.0 the falcoctl driver command will be used by the Falco installation process in place of the old falco-driver-loader script.

For example, to install the kernel module:.

Specify which driver we want to use

falcoctl driver config --type kmod

Install the driver

falcoctl driver install

By default, the falcoctl driver install command tries to download a prebuilt driver from the official Falco download s3 bucket. If a driver is found, then it is inserted into ${HOME}/.falco/. Otherwise, the script tries to compile the driver locally.

You can find more details on installing each driver type in our docs.

Finally, while the falcoctl driver command replaces the old falco-driver-loader script it’s important to note that, even though there is no change in terms of usage, the Docker images falco-driver-loader and falco-driver-loader-legacy no longer utilize the old falco-driver-loader script; instead, they now use falcoctl.

32-bit syscall emulation

The support for 32-bit syscalls has consistently been a highly requested feature for a long time. Until now, this support was only available in the kernel module, but starting from Falco 0.37.0, we have finally extended this support to the ebpf and modern_ebpf drivers. This feature is crucial as it addresses a security gap that has existed for some time.

It’s important to note that this feature is specifically for 32-bits syscalls emulated on the x86_64 architecture. Falco does not support pure 32-bit architectures.

Follow these steps to try out this new feature:

Create a C program ia32.c

#include <stdio.h>
#include <unistd.h>
#include <sys/syscall.h>
#include <sys/types.h>

int main() {
 syscall(__NR_close, -1);
 return 0;
}

Compile it gcc ia32.c -o ia32 -m32
Start Falco with the following rule evt.type = close and proc.name contains ia32
Execute the binary

./ia32

You should see the rule triggered

New override key

Falco 0.37.0 replaces the append: true key-value pair with a new override section. The override section allows you to either replace or append keys to a rule, macro, or list value . It’s important to note that you cannot append and replace the same key; you must choose one or the other. Choosing both will result in an error.

The keys that can be modified vary according to the rules component being overridden. See the override documentation for the full list of keys that can be modified.

The override section can either be in a custom rules file or can be in the same file as the component being overridden. In either case, the override section needs to be specified after the rule that is being modified. When the override is in the same file, the override section needs to be below the original rule, list, or macro definition. If the override is in another file, that file needs to be loaded after the original rules file.

A quick example from the documentation illustrates how this new feature works.

In this example, the original rule is in falco_rules.yaml and the override is specified in falco_rules.local.yaml.

/etc/falco/falco_rules.yaml

- rule: program_accesses_file
 desc: track whenever a set of programs opens a file
 condition: proc.name in (cat, ls) and evt.type=open
 output: a tracked program opened a file (user=%user.name command=%proc.cmdline file=%fd.name)
 priority: INFO

/etc/falco/falco_rules.local.yaml

- rule: program_accesses_file
 condition: and not user.name=root
 output: A file (user=%user.name command=%proc.cmdline file=%fd.name) was opened by a monitored program
 override:
 condition: append
 output: replace

The modified program_accesses_filerule would trigger when ls or cat use open on a file, unless they were run by root.

The new output message would be A file (user=%user.name command=%proc.cmdline file=%fd.name) was opened by a monitored program

A final note, the old append: true method of modifying values has been deprecated and will be removed in Falco 1.0.0.

Additional UX improvements

Introduce a new unique engine key in falco.yaml to replace all the other methods of opening engines (FALCO_BPF_PROBE, --modern-bpf, -g, -e). See the deprecated features section for more info.
Falco now expands environment variables in falco.yaml even when they are part of a string. It is now possible to use syntax similar to this:

ebpf:
 probe: ${HOME}/.falco/falco-bpf.o

Our gVisor integration has also been improved by adding support for more events, including write, socketpair, timerfd_create and an updated configuration generator. In addition, we added support for any gVisor container ID format, making Falco more robust and compatible with gVisor sandboxed containers beyond Docker and Kubernetes.

Breaking changes

This is a list of breaking changes introduced in Falco 0.37.0

The Rate-limiter mechanism was removed as it is no longer used.
--userspace CLI option was removed as it’s no longer used.
The falco-driver-loader script is removed and embedded into falcoctl
The Helm chart 4.0.0 contains several modifications to work with the new k8s metadata collector. Please read its breaking change file for more information.
The new falcoctl driver implementation will drop:
- --source-only
- BPF_USE_LOCAL_KERNEL_SOURCES
- DRIVER_CURL_OPTIONS
- The FALCO_BPF_PROBE environment variable won't be used by the new falcoctl driver loader as it is already deprecated and scheduled to be removed in the next major version.
Various environment variables have been replaced as part of the new falcoctl driver feature:
- DRIVERS_REPO has been replaced by FALCOCTL_DRIVER_NAME or the --name command line argument.
- DRIVERS_NAME has been replaced by FALCOCTL_DRIVER_REPOS or the --repo command line argument.
- DRIVER_KERNEL_RELEASE has been replaced by --kernelrelease command line argument.
- DRIVER_KERNEL_VERSION has been replaced by --kernelversion command line argument.
- DRIVER_INSECURE_DOWNLOAD has been replaced by --http-insecure command line argument.
Remove -K/-k options from Falco in favor of the new k8s plugin
Dropped plugins shipped with Falco since plugins will now be managed by falcoctl. If you want to use a plugin like k8saudit be sure to install it at init time with falcoctl.
A new feature in Falco 0.37.0 allows environment variables to be expanded even if they are part of a string. This new functionality introduces a minor breaking change.

Previously, environment variables used in YAML that were empty or defined as “” would be expanded to the default value. This was inconsistent with how YAML was handled in other cases, where we only returned the default values if the node was not defined.

With Falco 0.37.0 we will return the default value for nodes that cannot be parsed to the chosen type. The program_output command will be environment-expanded at init time instead of letting popen; thus, the shell expands it.

This is technically a breaking change, even if no behavioral change is expected.

Note that you can avoid environment var expansion by using ${{FOO}} instead of ${FOO}. It will resolve to ${FOO} and won't be resolved to the environment var value.

You can find more information on breaking changes in the tracking issue.

Deprecated features

This is a list of features that will be removed in Falco 0.38.0

Modern probe Docker builder is no longer used.
syscall_buf_size_preset Falco config in favor of engine.kmod/ebpf/modern_ebpf.buf_size_preset.
syscall_drop_failed_exit Falco config in favor of engine.kmod/ebpf/modern_ebpf.drop_failed_exit.
modern_bpf.cpus_for_each_syscall_bufferFalco config in favor of engine.modern_ebpf.cpus_for_each_buffer.
FALCO_BPF_PROBE environment variable in favor of engine.ebpf.probe.
-e command line flag in favor of engine.replay.capture_file.
g,gvisor-config command line flag in favor of engine.gvisor.config.
gvisor-root command line flag in favor of engine.gvisor.root.
modern-bpf command line flag in favor of engine.kind=modern_ebpf.
nodriver command line flag in favor of engine.kind=nodriver.
syscall_event_drops falco config will be replaced by the metrics config plus some automatic notification on drops.

Be sure to check the tracker issue for more information.

Try it out

Interested in trying out the new features? Use the resources below to get started.

Container Images
- falco (DockerHub, AWS ECR Gallery)
- falco-no-driver (DockerHub, AWS ECR Gallery)
- falco-driver-loader (DockerHub, AWS ECR Gallery)
CentOS/Amazon Linux
Debian/Ubuntu
openSUSE
Linux binary package

What’s next?

The community is active on many fronts, and we plan on delivering more great features and stability fixes during the next release cycle!

Some of the things we are currently working on include:

Implement further improvements to our rule framework and rule syntax.
Add new features and enhancements to falcoctl to make it even more powerful.
Enhance the quantity, quality, and presentation of metrics in Falco.

And much much more

Stay connected

Join us on social media and in our weekly community calls! It’s always great to have new members in the community, and we’re looking forward to hearing your feedback and ideas.

You can find all the most up-to-date information at https://falco.org/community/.

Blog: Falco Weekly 4 - 2024

Fri, 26 Jan 2024 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

Libs will need a 0.14.2 tag for the Falco 0.37.0 release, with the revert of https://github.com/falcosecurity/libs/pull/1533 PR.
During our release process, we found out that the new std::filesystem based implementaton was up to 8x time slower than the old ones; that's because it supports much more cases and does many more checks.
Therefore, in https://github.com/falcosecurity/libs/pull/1645, we revert to the old sorcery implementation, plus some minor improvements and added tests.

Moreover, many more changes landed in libs, that won't be part of the upcoming Falco 0.37.0 release:

Modernized C++ struct/enum/union declarations: https://github.com/falcosecurity/libs/pull/1588
Added support for newfstatat syscall: https://github.com/falcosecurity/libs/pull/1628
Fixed a potential deadlock for kmod: https://github.com/falcosecurity/libs/pull/1629
Big effort by our hero, Jason, to cleanup some stale macros: https://github.com/falcosecurity/libs/pull/1633,https://github.com/falcosecurity/libs/pull/1634,https://github.com/falcosecurity/libs/pull/1635,https://github.com/falcosecurity/libs/pull/1637,https://github.com/falcosecurity/libs/pull/1638
A small fix for old ebpf driver to support some GKE envs: https://github.com/falcosecurity/libs/pull/1642
Solved a data race and segfault in logger: https://github.com/falcosecurity/libs/pull/1643
Allow to selectively disable bpf and kmod engines from cmake: https://github.com/falcosecurity/libs/pull/1644

Falco

Falco tag 0.37.0-rc2 is out! Try it!

Moreover:

syscall_event_drops was soft-deprecated to get ready for Falco 0.38.0 upcoming cleanups: https://github.com/falcosecurity/falco/pull/3015
Avoid storing escaped strings in engine: https://github.com/falcosecurity/falco/pull/3028
Bumped falcoctl to v0.7.1 and rules to 3.0.0: https://github.com/falcosecurity/falco/pull/3030,https://github.com/falcosecurity/falco/pull/3034
Fixed nlohmann_json library include paths when using system one: https://github.com/falcosecurity/falco/pull/3032
Fixes to new libsinsp state metrics handling: https://github.com/falcosecurity/falco/pull/3033

We are in the testing phase so any feedback would be appreciated! Moreover, we crafted a dedicated helm chart to test the new k8smeta plugin and the k8s-metacollector, you can read more about it here. Please note these 2 new components will be officially released with Falco 0.37.0 as EXPERIMENTAL features.

As a final reminder, please take a look at our polls if you have some spare seconds.

Falcoctl

Falcoctl 0.7.1 is out! Try it! and contains a small fix for the driver-loader on COS.

Moreover, we added dependabot configs, that then bumped lots of deps to their latest compatible versions: https://github.com/falcosecurity/falcoctl/pull/385.

Join relevant discussions

Our new discussion section: https://github.com/falcosecurity/falco/discussions
Breaking changes in Falco 0.37.0: https://github.com/falcosecurity/falco/issues/2763
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list
Open a discussion in our discussion section

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Andrea, Federico

Blog: Falco Weekly 3 - 2024

Fri, 19 Jan 2024 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

Libs tag 0.14.1 is out! Try it!

It fixes the following things:

fix(gvisor): gVisor engine crashes with non-hex container IDs: https://github.com/falcosecurity/libs/issues/1602
fix(gvisor): handle arbitrary sandbox IDs: https://github.com/falcosecurity/libs/pull/1612
fix(libsinsp): modify switch case: https://github.com/falcosecurity/libs/pull/1620
fix(libsinsp): Add new cgroup layout for podman: https://github.com/falcosecurity/libs/pull/1613
fix(libsinsp): consistent thread info filtering while dumping: https://github.com/falcosecurity/libs/pull/1606
fix(libsinsp): do not suppress zero ptids: https://github.com/falcosecurity/libs/pull/1598
fix(libsinsp): fix resolved PT_FSPATH and PT_FSRELPATH evt params: https://github.com/falcosecurity/libs/pull/1597

You can find a detailed summary on the release page.

Falco

Falco tag 0.37.0-rc1 is out! Try it!

Some final cleanup before the final tag:

cleanup(falco.yaml): rename none in nodriver: https://github.com/falcosecurity/falco/pull/3012
update(config): graduate outputs_queue to stable: https://github.com/falcosecurity/falco/pull/3016

As a final reminder, please take a look at our polls if you have some spare seconds.

Falcoctl

Falcoctl 0.7.0 is out! Try it!

These are some of the most relevant changes:

update(output): complete rework of the output system: https://github.com/falcosecurity/falcoctl/pull/335
update(cmd): remove redundant configuration for error handling: https://github.com/falcosecurity/falcoctl/pull/337
new(cmd): add artifact config command: https://github.com/falcosecurity/falcoctl/pull/340
feat(artifact/config): fetch config layer for a specific platform: https://github.com/falcosecurity/falcoctl/pull/349
new(artifact/manifest): add manifest command: https://github.com/falcosecurity/falcoctl/pull/351
new: driver command: https://github.com/falcosecurity/falcoctl/pull/343
new(pkg/driver): fixed some kernel version related issues: https://github.com/falcosecurity/falcoctl/pull/364
cleanup(cmd,internal,pkg): move driver config options to be common to all driver commands: https://github.com/falcosecurity/falcoctl/pull/365
fix(pkg/driver): do not call FixupKernel when building drivers: https://github.com/falcosecurity/falcoctl/pull/373
new: introduce asset artifact type: https://github.com/falcosecurity/falcoctl/pull/309

You can find a detailed summary on the release page.

Join relevant discussions

Our new discussion section: https://github.com/falcosecurity/falco/discussions
Breaking changes in Falco 0.37.0: https://github.com/falcosecurity/falco/issues/2763
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list
Open a discussion in our discussion section

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Andrea, Federico

Blog: Falco Weekly 50 - 2023

Fri, 15 Dec 2023 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

The anticipated 0.14.0 libs tag (and its driver counterpart) are going to be tagged soon, by the end of next week.
A xmas present for you all! :christmas_tree:

Mostly fixes were merged during this week:

Populate labels field for pod sandbox containers: https://github.com/falcosecurity/libs/pull/1564
Improved libscap modern bpf tests and CI checks: https://github.com/falcosecurity/libs/pull/1568
Avoid a double free when an exception is thrown during sinsp initialization: https://github.com/falcosecurity/libs/pull/1569
Made our pkg-config files paths-relative: https://github.com/falcosecurity/libs/pull/1570
Fixed some paths handling in fs.path: https://github.com/falcosecurity/libs/pull/1571
Do not include NULL terminator in enter event strings: https://github.com/falcosecurity/libs/pull/1574
Started a dedicated container engines test suite: https://github.com/falcosecurity/libs/pull/1544
Rewritten scary concatenate_paths function leveraging modern c++17 std::filesystem: https://github.com/falcosecurity/libs/pull/1533
Use a smart pointer for m_resolver in sinsp_dns_manager to avoid leaks: https://github.com/falcosecurity/libs/pull/1558

Also, thanks to actuated.dev for offering us arm64 github action runners, CI has been fully ported to github actions, except for a single CircleCI job! https://github.com/falcosecurity/libs/pull/1555

Rumors have it coming next:

Drivers build fix against linux 6.7-rc4+: https://github.com/falcosecurity/libs/pull/1566
Add k8s.pod.uid, k8s.pod.sandbox_id and mark k8s.pod.id as legacy: https://github.com/falcosecurity/libs/pull/1575

Falco

Falco has seen some big new features this week!

Env variables expansion was extended to all scalar values in Falco configuration file! https://github.com/falcosecurity/falco/pull/2918, https://github.com/falcosecurity/falco/pull/2972
Leveraging the above, engine.ebpf.probe path now defaults to ${HOME}/.falco/falco-bpf.o: https://github.com/falcosecurity/falco/pull/2971
CI has been ported to use actuated.dev github action arm64 runners! https://github.com/falcosecurity/falco/pull/2945, https://github.com/falcosecurity/falco/pull/2967
Monitor more types of events for Falco hot reload feature: https://github.com/falcosecurity/falco/pull/2965
libs and driver were bumped to latest master: https://github.com/falcosecurity/falco/pull/2970

Finally, the new falcoctl based driver-loader was finally merged in Falco: https://github.com/falcosecurity/falco/pull/2905.
If you can, please make sure to give it a spin and let us know any feedback, it is very valuable for us!
To try it out:

docker pull falcosecurity/falco-driver-loader:master
docker run --rm -i -t \
 --privileged \
 -v /root/.falco:/root/.falco \
 -v /proc:/host/proc:ro \
 -v /boot:/host/boot:ro \
 -v /lib/modules:/host/lib/modules \
 -v /usr:/host/usr:ro \
 -v /etc:/host/etc:ro \
 falcosecurity/falco-driver-loader:master

Falcoctl

Some fixes on top of the new driver-loader happened:

Cleanup eBPF probe symlink in Cleanup method: https://github.com/falcosecurity/falcoctl/pull/371
Do not call FixupKernel when building drivers: https://github.com/falcosecurity/falcoctl/pull/373

Moreover, we finally merged the new asset artifact type PR! https://github.com/falcosecurity/falcoctl/pull/309

Falcoctl is quite ready for v0.7.0 release; we only need more driver-loader testing!

Driverkit

Driverkit has seen a small bug fix release this week: https://github.com/falcosecurity/driverkit/releases/tag/v0.16.2.
It contains a fix to docker go package multiplexed output support: https://github.com/falcosecurity/driverkit/pull/310.

Moreover, we merged a PR that opens up the possibility for Driverkit to directly use cmake to configure and then build our drivers: https://github.com/falcosecurity/driverkit/pull/309.

What's next?
The cmake PR is opened and works super good; build times are as good as before, so no penalty! https://github.com/falcosecurity/driverkit/pull/302.
Moreover, we are going to make use of actuated.dev arm64 runners in driverkit too, porting its CI to github actions: https://github.com/falcosecurity/driverkit/pull/311.

Join relevant discussions!

Breaking changes in Falco 0.37.0: https://github.com/falcosecurity/falco/issues/2763
Breaking changes in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Andrea, Federico

Blog: Install and Test Atomic Red in Kubernetes

Tue, 12 Dec 2023 00:00:00 +0000

In cloud-native environments, where applications scale up and down much faster than traditional monolithic application architectures, the ability to proactively identify and respond to threats in real time is paramount. As more organizations embrace cloud-native architectures for application delivery, more robust security measures need to be introduced. In this blog post, we delve into the dynamic realm of Kubernetes threat detection by exploring how open source Falco can seamlessly detect Atomic Red Team tests in real time within Kubernetes environments.

Atomic Red Team is a powerful framework designed to simulate real-world attacks, providing organizations with a controlled environment to validate the effectiveness of their security measures. We take this a step further by deploying Atomic Red to Kubernetes with a single command, creating a realistic testing ground for evaluating the responsiveness of Falco.

Our journey begins with the effortless deployment of Atomic Red to Kubernetes, showcasing the simplicity and efficiency of orchestrating security testing within containerized environments. Once deployed, we invoke specific Atomic Red Team tests, simulating a range of threat scenarios. The true test lies in Falco's ability to detect these threats in line with the MITRE ATT&CK framework, a globally-recognized matrix mapping adversary techniques to defensive tactics.

This exploration is not just about identifying threats; it's a collaborative effort to enhance Falco's coverage. Should we identify any gaps in detection, we dive deeper, revising the executed techniques and crafting custom rules. This iterative process aims to extend our MITRE ATT&CK coverage, aligning Falco with the industry's best practices for threat detection and mitigation.

Deploying Atomic Red Team

To avoid any potential service disruption in production environments, we recommend installing Atomic Red in a test lab environment, or at least a staging environment of Kubernetes. We have a step-by-step video for installing Atomic Red on Youtube

Before we start the deployment, remember to create the atomic-red network namespace.

kubectl create ns atomic-red

A single pod will be deployed with privileged set to true. Atomic Red requires admin-level securityContext to perform certain actions that require elevated permissions.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
 name: atomicred
 namespace: atomic-red
 labels:
 app: atomicred
spec:
 replicas: 1
 selector:
 matchLabels:
 app: atomicred
 template:
 metadata:
 labels:
 app: atomicred
 spec:
 containers:
 - name: atomicred
 image: issif/atomic-red:latest
 imagePullPolicy: "IfNotPresent"
 command: ["sleep", "3560d"]
 securityContext:
 privileged: true
 nodeSelector:
 kubernetes.io/os: linux
EOF

Note: This creates a pod called 'atomicred' in the 'atomic-red' network namespace. You can check on the state of the installation with the below command:

kubectl get pods -n atomic-red

If you ever want to remove the Atomic Red project from your Kubernetes cluster, simply run:

kubectl delete deployment atomicred -n atomic-red

Familiarize Yourself with Atomic Red Tests

Once deployed, you will need to shell into the Atomic Red pod to perform the following test scenarios. This might seem a little confusing, but Atomic Red was developed with PowerShell in mind, so the below instructions ask the user to shell into a container, and once they are in the running pod, they must run Powershell to import and invoke the various Atomic Test Scenarios.

Once you are familiar with this logic, you’ll find Atomic Red is a truly simple security simulation tool.

kubectl exec -it -n atomic-red deploy/atomicred -- bash

As mentioned, you need to run Powershell once you are in the Atomic Red pod:

pwsh

Now, you can finally load the Atomic Red Team module:

Import-Module "~/AtomicRedTeam/invoke-atomicredteam/Invoke-AtomicRedTeam.psd1" -Force

Check the details of the TTPs:

Invoke-AtomicTest T1070.004 -ShowDetails

Check the prerequisites to ensure the test conditions are right:

Invoke-AtomicTest T1070.004 -GetPreReqs

You can see in the below screenshot, there are no prerequisites required to perform these tests. As a result, we can invoke the bulk file deletion test scenario.

Remove the feature flags to execute the test simulation.

Invoke-AtomicTest T1070.004

This test will attempt to delete individual files or individual directories. When we have Falco installed, this Atomic test should trigger the 'Warning bulk data removed from disk' rule by default. Next, we discuss Falco’s installation.

Congrats! Now that we know how Atomic Red works, let’s install Falco and run it side-by-side against Atomic Red to prove it detects these tests in real time. We will need to open two terminal windows to see the real-time response detections.

Installing & Testing Falco

For this lab guide, we can install Falco via Helm on a fixed version prior to the segregation of rules into different rules feeds, such as 'incubating', 'sandbox' and 'stable'. The reason we are doing this is to ensure all Falco rules are accessible in our lab scenario. To use the latest version of Falco, simply remove the '--version' feature flag from the Helm install script.

helm install falco falcosecurity/falco --namespace falco \
--create-namespace \
--set tty=true \
--version 3.3.0

Just like the Atomic Red deployment, we need to monitor the progress of the Falco installation. The pods will change state a few times during the installation, but should eventually all be in a 'RUNNING' status after about a minute or so.

Please use the below command to check on the status change of Falco pods:

kubectl get pods -n falco -w

Once Falco is installed, we can track the events as they are generated using the following command in the second terminal window.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco

Jump back into the first terminal window and re-run the bulk file deletion Atomic Test - 'T1070.004':

Invoke-AtomicTest T1070.004

You’re going to identify certain noise in the detection rules. For example, all Atomic Tests are run under the 'Root' user, therefore, we will always get a detection for scripts running under root. To ignore this noise, let’s instead just check for the specific Falco rule we are looking to detect:

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Bulk data has been removed from disk'

Hurrah! We see the exact detection matching the context of the Atomic Red test scenario. Let’s move on to the next Atomic Test to invoke. There are a bunch of test scenarios for Linux that you can test out today.

Check out the list on the official Atomic Red Team Github project.

T1556.003 Modify Authentication Process

In this scenario, Atomic Red generates three Pluggable Authentication Modules (PAM): two malicious PAM rules for Linux and FreeBSD, as well as a malicious PAM module for Linux. These programs can be used to open and read sensitive file content, and we can agree that they are non-trusted programs. Again, we have an out-of-the-box rule for these activities:

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Sensitive file opened for reading by non-trusted program'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1556.003

T1036.005 Masquerading: Match Legitimate Name or Location

This test scenario executes a process from a directory masquerading as the current parent directory.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Executing binary not part of base'

Now, it's time to simulate our threat.

Invoke-AtomicTest T1036.005

You can see that in the left terminal window, there is an echo message in the terminal saying '”Hello from the Atomic Red Team.”' Any string output in the command line can also be detected in Falco’s outputs.

T1070.002 Indicator Removal on Host

Adversaries may clear system logs to hide evidence of an intrusion. macOS and Linux both keep track of system or user-initiated actions via system logs. The majority of native system logging is stored under the '/var/log/' directory.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Log files were tampered'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1070.002

T1070.003 Clear Command History

In addition to clearing system logs, an adversary may clear the command history of a compromised account to conceal the actions undertaken during an intrusion. Various command interpreters keep track of the commands users type in their terminals so that users can retrace what they've done.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Shell history had been deleted or renamed'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1070.003

You can see from the below screenshot that four different operations were performed. Therefore, four unique Falco detections were triggered on those individual attempts to clear the command line history.

T1014 Loadable Kernel Module Based Rootkit

Adversaries may use Rootkits to hide the presence of programs, files, network connections, services, drivers, and other system components. Rootkits are programs that hide the existence of malware by intercepting/hooking and modifying operating system API calls that supply system information.

Rootkits may reside at the user or kernel level in the operating system or lower, to include a hypervisor, Master Boot Record, or System Firmware. As such, it’s critical that Falco detects Rootkits in real time.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Linux Kernel Module injection from container detected'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1014

Falco is detecting the Linux kernel module injection attempt, whether it was a successful execution or not.

T1037.004 [CUSTOM RULE] Boot Initialization - RC Scripts

Adversaries may establish persistence by modifying RC scripts that are executed during a Unix-like system’s startup. These files allow system administrators to map and start custom services at startup for different run levels. RC scripts require root privileges to modify.

Command to simulate the 'T1037.004' test:

Invoke-AtomicTest T1037.004

You’ll notice that this is the first test where we don’t get a useful Falco detection related to the threat. As a result, we need to create a 'custom-rules.yaml' file with the custom Falco rule for detecting boot initialization scripts.

cat > custom-rules.yaml <<EOF
customRules:
custom_rules.yaml: |-
- rule: Base64-encoded Python Script Execution
desc: >
This rule detects base64-encoded Python scripts on command line arguments.
condition: >
spawned_process and (
((proc.cmdline contains "python -c" or proc.cmdline contains "python3 -c" or proc.cmdline contains "python2 -c") and (proc.cmdline contains "echo" or proc.cmdline icontains "base64")) or ((proc.cmdline contains "import" and proc.cmdline contains "base64" and proc.cmdline contains "decode")))
output: >
Potentially malicious Python script encoded on command line
(proc.cmdline=%proc.cmdline user.name=%user.name proc.name=%proc.name
proc.pname=%proc.pname evt.type=%evt.type gparent=%proc.aname[2]
ggparent=%proc.aname[3] gggparent=%proc.aname[4] evt.res=%evt.res
container.id=%container.id container.name=%container.name file=%fd.name)
priority: warning
tags:
- T1037.004
- MITRE_Defense_Evasion
source: syscall
EOF

Adversaries can establish persistence by adding a malicious binary path or shell commands to 'rc.local', 'rc.common', and other RC scripts specific to the Unix-like distribution. Upon reboot, the system executes the script's contents as root, resulting in persistence.

Let’s try upgrading Falco to reflect the 'custom-rules.yaml' file:

helm upgrade falco falcosecurity/falco \
-n falco \
--set tty=true \
--version 3.3.0 \
--reuse-values \
-f custom-rules.yaml

Granted, there’s no obvious formatting issues when creating the 'custom-rules.yaml' manifest, you should be able to successfully upgrade Falco and the pods should be in a 'RUNNING' state. If there was an issue with the custom rules file, the Falco pod state will likely change to 'CrashLoopBackOff'.

Let’s see if we detect the Atomic test after upgrading Falco with the newly-created custom rule. Remember to have Falco running in a second terminal window with the following command:

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Potentially malicious Python script'

We can simulate the technique ID 'T1037.004' one last time:

Invoke-AtomicTest T1037.004

Hurrah! We detected the boot initialization scripts with the above command. To recap, adversaries looking to abuse those RC scripts are especially effective for lightweight Unix-like distributions using the root user as default, such as IoT or embedded systems. If you are wondering why the Falco rule was specifically looking at Base64-encoded Python scripts, well, we need to look back at the details associated with the Atomic test simulation.

Invoke-AtomicTest T1037.004 -ShowDetails

We can see from the command that it is using the 'python3' command to run Python scripts. However, the script itself is executed as a base64-encoded string to evade some traditional detection tools.

Conclusion

In conclusion, this article serves as a comprehensive guide aimed at enhancing knowledge in cloud-native threat simulation and detection. Its primary audience includes security practitioners, DevOps teams, and anyone involved in securing Kubernetes environments. The focal point is the utilization of Atomic Red for deploying test simulations aligned with the MITRE ATT&CK framework and leveraging Falco to detect these threats in real time.

The article not only illustrates how Atomic Red can be instrumental in improving the rules shipped with Falco, but also emphasizes its role in identifying potential gaps in coverage. By doing so, it offers a valuable resource for customers seeking alternative methods to validate the effectiveness of their Falco rules and, consequently, fortify the security posture of their Kubernetes deployments.

The notion of creating a realistic testing ground for evaluating the responsiveness of Falco is particularly beneficial for DevOps teams. This facet addresses the need for hands-on experience in threat simulation, adversarial approaches, and understanding the intricacies of the MITRE ATT&CK framework. In providing this practical testing environment, the article empowers DevOps teams to proactively enhance their security measures, ensuring robust coverage and responsiveness in the dynamic landscape of Kubernetes threat detection.

Blog: Falco Weekly 48 - 2023

Fri, 01 Dec 2023 00:00:00 +0000

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

The anticipated 0.14.0 libs tag (and its driver counterpart) are still a bit late, unfortunately.

Anyway, spring cleaning went on once again this week!

cleaned up dup3 flags param: https://github.com/falcosecurity/libs/pull/1469
cleaned up other params inconsistencies in the drivers: https://github.com/falcosecurity/libs/pull/1512
dropped b64 dep: https://github.com/falcosecurity/libs/pull/1518
dropped tinydir dep: https://github.com/falcosecurity/libs/pull/1516
removed some warning suppressions: https://github.com/falcosecurity/libs/pull/1519
cleaned up big unused function sinsp_evt::get_param_as_json: https://github.com/falcosecurity/libs/pull/1523

The big safer parameter handling PR was merged, making libs much more robust! Moreover, ppc64le support was extended to kmod and legacy ebpf probe, and we added CI jobs to test the build of drivers on it! Thanks to Afsan Hossain for his big contribution!

Finally, some more fixes:

build on s390x was fixed: https://github.com/falcosecurity/libs/pull/1522
some recently introduced regressions were fixed: https://github.com/falcosecurity/libs/pull/1524
fixed a memleak in sinsp_dns_manager: https://github.com/falcosecurity/libs/pull/1526

Rumors have it coming next week:

More fixes: https://github.com/falcosecurity/libs/pull/1530, https://github.com/falcosecurity/libs/pull/1528

Falco

We bumped libs and driver to latest master: https://github.com/falcosecurity/falco/pull/2929.
Moreover, Falco will now print system info during startup: https://github.com/falcosecurity/falco/pull/2927.
Falco does now expose a new config option to enable libsinsp state metrics: https://github.com/falcosecurity/falco/pull/2883 Finally, the new driver selection mechanism PR was merged!

Falcoctl

Some fixes on top of the new driver-loader happened:

fixed up naming for the new Falco driver selection in config: https://github.com/falcosecurity/falcoctl/pull/357
small fix for host-root driver-loader configuration: https://github.com/falcosecurity/falcoctl/pull/358
do not fail when /sys/kernel/debug fails to be mounted: https://github.com/falcosecurity/falcoctl/pull/361

Join relevant discussions!

Breaking change in Falco 0.37.0: https://github.com/falcosecurity/falco/issues/2763
Breaking change in Falco 0.38.0: https://github.com/falcosecurity/falco/issues/2840
Falco metrics exposed to final users: https://github.com/falcosecurity/falco/issues/2928
Create a more coherent stats model for libs: https://github.com/falcosecurity/libs/issues/1463
Allow loading tracepoints other than the ones needed by Falco: https://github.com/falcosecurity/libs/issues/1376

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Federico

Blog: Using Falco to Create Custom Identity Detections

Tue, 28 Nov 2023 00:00:00 +0000

Identity Threat Detection & Response (ITDR) in the cloud is of paramount importance to limit access to sensitive data and maintain the integrity of cloud infrastructure. Leading cloud providers like AWS, Microsoft Azure, and Google Cloud have implemented robust Identity and Access Management (IAM) controls, as well as Multi-Factor Authentication (MFA) options, to ensure that users have the standardized access control limitations.

However, as the saying goes, "Trust, but verify." Even with these layers of security, there's a growing concern about what happens when a rogue employee or an external adversary manages to compromise an identity provider. Recent months have witnessed a surge in attacks targeting popular identity providers like Okta, underscoring the critical need for timely and effective detection capabilities. In fact, (Crowdstrike’s 2023 Threat Hunting) report had classified 62% of all interactive cyber intrusions as having involved some form of compromised identities.

Without proper detection, incidents such as the attacks on organizations like Caesars and MGM might go unnoticed until it's too late. Fortunately, open source Falco offers a Dedicated plugin for the Okta identity platform, empowering security teams to respond swiftly and with the context required to take real action against potential threats.

In this blog post, we will delve into how Falco fulfills the requirements for ITDR capabilities. We'll illustrate the significance of Falco's adaptable rule logic and provide readers with a real-world example of crafting custom rules derived directly from Okta audit logs.

Understanding the rule logic

The Falco Okta plugin comes with a set of valuable default rules for Okta logs, which are designed to assist you in enhancing the security of your Okta platform.

A typical illustration of the importance of these rules lies in the process of initiating a password reset within the Okta platform. In practice, an insider threat might reset a password, opt not to inform the end-user about the reset, and potentially carry out an account takeover in this context. Whenever a specific action is executed through the Okta user interface, there is a straightforward method to access the associated activity logs on the web user interface

Nonetheless, this default perspective is not particularly suitable for our Falco rule since it's focused on a specific user making changes to another specific user account. The system log focus is on algorithmically-generated actor IDs linked to both the user initiating the password reset and the user whose account password has been reset.

While this event information may serve a purpose if you only wish to trigger detection events between these specific accounts, in practical scenarios, a more comprehensive view encompassing all users is typically required.

For a more efficient approach, it's advisable to select the User updated password in Okta event behavior shown at the bottom of the screenshot above. This action will automatically narrow down the search view to display event information exclusively related to password updates in Okta. With this method, we can effectively set up alerts for all password update activities.

Utilizing Falco, we can extract the precise event type value from the system log and incorporate it into a Falco rule.

Fortunately, this task has already been handled for us in the Okta plugin ruleset. It's worth noting that the value found in the system logs aligns with our specified Falco conditions, ensuring that we attain equivalent visibility within Falco as we do in the Okta UI. However, it's important to be aware that in extensive production environments, this rule can generate a significant amount of noise. If needed, you have the option to filter this activity specifically for password resets targeting the 'Admin' privileged group, as they are frequently the primary targets of cyberattacks.

- rule: User password reset by OKTA admin
desc: Detect a password reset on a user done by OKTA Admin Account
condition: okta.evt.type = "user.account.reset_password"
output: "A user password has been reset by an OKTA Admin account (user=%okta.actor.name, ip=%okta.client.ip, target user=%okta.target.user.name)"
priority: NOTICE
source: okta
tags: [okta]

Here is the output displayed in our Falcosidekick UI, presenting the context of the actor's name and the corresponding IP address responsible for the change. It could be advantageous to explore the possibility of incorporating additional Output fields to enhance incident response capabilities when addressing identity threats.

Building a custom Falco rule

The next example extends beyond the scope of the default Falco rules. For instance, if an identity integration or application were to be disassociated from the user Nigel Douglas, it might be an attempt to compromise security measures within established workflows or systems - a good example of Impair Defenses technique in the MITRE ATT&CK matrix.

Consequently, we will replicate this specific action and create a custom Falco rule to identify such behavior. As shown in the screenshot below, the admin user is seen removing the application assignment from the user Nigel.

After the application has been unassigned, we receive the updated event type data in Okta, much like our previous workflow. The Okta query that provides the results of the application removal is as follows:

eventType eq "application.user_membership.remove"

With this understanding, we can navigate to the falco_rules.local.yaml file in our Falco installation, which is often used to define a custom ruleset. Based on the construction of the previous Falco rule and the string identified in the screenshot above application.user_membership.remove, we will update the condition to check for this type. In your case, the Falco rule should be structured as follows:

- rule: Remove app membership
desc: Detect membership removal in OKTA
condition: okta.evt.type = "application.user_membership.remove"
output: "A user has removed the following app in OKTA (user=%okta.actor.name, ip=%okta.client.ip, target user=%okta.target.user.name)"
priority: CRITICAL
source: okta
tags: [custom_rule, mitre_defense_evasion, T1562, impair_defenses]

To ensure the custom rule is applied, you might need to restart Falco (depending on the version). If you have deployed Falco and Falcosidekick via a Docker compose file, this can be achieved by simply executing a stop command to halt the containers, followed by the up -d command, which restarts the Docker containers with the same configurations specified in the docker-compose.yaml file.

While Docker is not the only deployment option for Falco, it is undeniably a very convenient option for these types of test environments. The source of the docker-compose.yaml can be accessed here .

docker compose -f docker-compose.yaml stop
docker compose -f docker-compose.yaml up -d

In order to assist with incident response initiatives, you will now notice the custom tags linked to MITRE ATT&CK tactics and techniques in your alert output. This enhanced alert output facilitates incident responders in recognizing the specific issues related to this behavior, enabling them to potentially detect insider threats at the earliest stage. These custom tags are then integrated into the Falco rule for further context.

Why Falco instead of a traditional logging solution?

While it's possible to forward all your Okta logs to a centralized Security Incident & Event Management (SIEM) system, certain limitations become apparent. One prominent concern pertains to storage, as a substantial number of events must be retained in a centralized backend database, requiring aggregation and indexing to produce security alerts. This can impose a significant operational burden on organizations since they are effectively storing a multitude of events, a significant portion of which may be extraneous, potentially resulting in substantial costs associated with ingestion charges.

Similarly, once the events are in the system, it becomes crucial to have a solid grasp of crafting effective detection rules. Instead of managing intricate scripts and queries to minimize false positives, Falco streamlines the process by offering a unified rules language applicable across host endpoints, cloud services, CI/CD services, and Okta logs. This approach enables swift rule development and immediate testing within your environment without incurring ingestion charges. Furthermore, Falco addresses the issue of centralized storage bloat through its intelligent streaming engine, which processes event context and makes decisions on whether to trigger alerts based on specific event metadata, rather than indiscriminately ingesting all associated events.

Finally, the entire process of manually executing Okta search queries in the web UI, or managing intricate detection scripts, can be time-consuming and often results in coverage gaps. Falco offers a solution by delivering a nearly real-time detection engine that enables the use of macros and lists for complex querying. For instance, consider the task of verifying whether the user Nigel Douglas is logging in from their usual IP address. Instead of navigating through complex Okta queries, you can simply use the actor ID, cross-referencing it with the typical IP they use for sign-ins, and also taking into account the context of their access, such as interactions with the Admin Dashboard or other elements within the Okta user interface.

This is how Okta queries can be structured:

eventType eq "user.session.access_admin_app" and client.ip_address eq "78.xx.xxx.249" and target.id eq "00u9xcz5aphuQ8ZQq5d7" and outcome.result eq "SUCCESS"

Yet, if our aim is to identify successful login attempts from questionable geographic regions or IP addresses, we must integrate lists and diligently keep them up-to-date. This is where the following Falco rule proves its worth:

- rule: Suspicious Login for Nigel Douglas
desc: Detect suspicious login attempts from known suspicious IPs
condition: okta.evt.type = "user.session.access_admin_app" and okta.client.ip in (suspicious_ips) and okta.target.user.id= "00u9xcz5aphuQ8ZQq5d7"
output: Suspicious IP Inbound Request
(okta.actor.name=%okta.actor.name, okta.client.ip=%okta.client.ip, okta.target.user.id=%okta.target.user.id, okta.target.user.name=%okta.target.user.name, okta.app=%okta.app, okta.evt.type=%okta.evt.type)
priority: CRITICAL
tags: [custom_rule, mitre_initial_access, T1078, valid_accounts]
source: okta

Next, we create a list that compiles all potentially identified malicious actors who may attempt an account takeover on a legitimate account, which should typically be accessed from a consistent IP address. This list logic can be applied to geolocations, such as countries, instead of specific IPs. In both scenarios, the list is named suspicious_ips and is referred to in the Falco rule conditions as follows:

- list: suspicious_ips
items: ["103.236.201.88", "104.244.74.23", "107.189.13.251", "118.163.74.160", "125.212.241.131", "176.58.100.98", "176.58.121.177", "179.43.128.16", "179.48.251.188", "180.150.226.99", "181.214.39.73", "185.10.16.41", "185.100.85.132", "185.100.85.22", "185.100.85.23", "185.100.85.25", "185.191.204.254", "185.195.71.10", "185.195.71.12", "185.195.71.4", "185.195.71.5", "185.195.71.6", "185.195.71.7", "185.195.71.8", "185.220.101.3", "185.82.219.109", "195.80.151.30", "198.58.107.53", "198.98.60.90", "199.249.230.100", "199.249.230.107", "199.249.230.109", "199.249.230.113", "199.249.230.117", "199.249.230.119", "199.249.230.121", "199.249.230.140", "199.249.230.157", "199.249.230.165", "199.249.230.180", "199.249.230.70", "199.249.230.71", "199.249.230.78", "199.249.230.85", "199.249.230.88", "199.249.230.89", "200.122.181.2", "204.194.29.4", "205.185.119.35"]

Finally, we get the detection that there was a login from a suspicious IP address. We changed the Priority to CRITICAL to reflect the severity of a suspicious login from a malicious IP.

This is just one instance demonstrating the capabilities of Falco. I encourage you, as the reader, to explore various innovative approaches for crafting customized detection rules that align with your unique zero-trust architecture strategy. Should you have recommendations on enhancing default detection rules in Falco for Okta identity logs, please don't hesitate to reach out to us directly. We are always open to discussions: https://falco.org/community/ .

Strengthening Identity Security with Falco: Next Steps

In a landscape where identity threats are on the rise, extending to identity providers themselves, as exemplified by the recent Okta security breach, organizations are compelled to enhance their identity management and cybersecurity preparedness. After reading this article, you should hopefully have a deeper appreciation for Falco and its user identity security approach.

Evaluating your existing runtime security can be a valuable starting point, particularly if you identify gaps in Okta log coverage, making Falco a worthwhile consideration. And this very plugin logic can be extended to AWS and Google Cloud Platform via their own response logging services - AWS Cloudtrail and GCP Audit Logs.

For those interested in delving deeper into recent identity-related attacks, you can explore the article on DarkReading, where we delve into how ITDR solutions can be employed to detect Okta Cross-Tenant Impersonation Attacks .

If you're interested in installing Falco on a test machine and integrating the Okta plugin, you can find helpful deployment script at Luca Guerra’s Github repo: https://github.com/LucaGuerra/falcosidekick-ui-compose/blob/main/falco.yaml .

To configure the Okta plugin, you can easily uncomment the section below and input your Okta details as needed. If you're uncertain about how to obtain your Okta API token, you can refer to this resource for guidance: https://developer.okta.com/docs/guides/create-an-api-token/main

 - name: okta
library_path: libokta.so
init_config:
organization: xxxxx # as in https://xxxxx.okta.com
api_token: yyyyy # your Okta API token
open_params: ''

Blog: Falco Weekly 47 - 2023

Fri, 24 Nov 2023 00:00:00 +0000

Another week, another load of improvements everywhere in the falcosecurity!

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

The anticipated 0.14.0 libs tag (and its driver counterpart) are a bit late, unfortunately.

Anyway, spring cleaning went on this week!

removed stopwatch implementation, now unused: https://github.com/falcosecurity/libs/pull/1493
removed unused sinsp_test.cpp file: https://github.com/falcosecurity/libs/pull/1499
removed jq dep: https://github.com/falcosecurity/libs/pull/1500

Moreover, some fixes on the recently introduced async event queue class happened: https://github.com/falcosecurity/libs/pull/1490, https://github.com/falcosecurity/libs/pull/1504. Finally, some fixes around the stats code: https://github.com/falcosecurity/libs/pull/1505, https://github.com/falcosecurity/libs/pull/1506.

Rumors have it coming next week:

New big cleanup: deprecation of tracers: https://github.com/falcosecurity/libs/pull/1503
ppc64le support for bpf and kmod + CI build jobs: https://github.com/falcosecurity/libs/pull/1497
remove old metaevents implementation: https://github.com/falcosecurity/libs/pull/1495
Small fix on top of ia32 work: https://github.com/falcosecurity/libs/pull/1501

Second part of an effort by Luca Guerra to clean up libsinsp from potential undefined behavior: https://github.com/falcosecurity/libs/pull/1502.
This is so important that deserved to be left alone :)

Falco

We have a new official adopter! Welcome to Thought Machine: https://github.com/falcosecurity/falco/pull/2919 Small cleanup to avoid Falco configuratiom to be inited twice: https://github.com/falcosecurity/falco/pull/2917

Falcoctl

The new driver command was merged! https://github.com/falcosecurity/falcoctl/pull/343 We are now in the process of adding tests and eventually fixing spotted bugs :) Also, the new asset artifact type PR is being reviewed: https://github.com/falcosecurity/falcoctl/pull/309.

Others

Driverkit v0.16.0 was just released, and contains some fixes, a new local build processor and preliminary SLES support.

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Adding runtime threat detection to Google Kubernetes Engine with Falco

Mon, 20 Nov 2023 00:00:00 +0000

One of the big advantages of running your workloads on a managed Kubernetes service like Google Kubernetes Engine (GKE) is that Google ensures your clusters are being deployed and managed following industry best practices.

While GKE clusters are incredibly secure and reliable, there is always room for improvement.

In this blog, we’re going to describe how you can enhance GKE’s already great security by adding runtime threat detection with Falco.

What is Falco?

With Falco running on your GKE clusters you can be notified of a wide variety of events, such as:

Did someone start a container with high privileges?
Has someone shelled into a running container?
Has an executable been added to the container after it was deployed?

These are just a few examples. Falco has over 80 rules that can be used to make you aware of not only external threats but also when clusters aren’t being operated in accordance with industry best practices.

GKE Installation considerations

There are two different ways to install Falco on GKE. The first is using the prepackaged click-to-run offering in the Google Cloud Marketplace. The second is using Falco’s helm chart. The click-to-run offering is probably the simplest way to get up and running with Falco on GKE, but the drawback is that the version offered often lags behind the latest release.

It’s also important to note that as of this writing, you cannot run Falco on GKE clusters running in Autopilot mode. This is primarily because Falco uses an init container running with privileged access to install its driver, and Autopilot does not allow the execution of privileged containers.

Something else to be aware of is that Falco on GKE needs to use one of Falco’s eBPF drivers. Falco uses a driver to capture syscall events, and this driver is offered as a loadable kernel module or as an eBPF probe. There are actually two eBPF probes with Falco. One is called ‘eBPF’ (or classic eBPF) and the other is referred to as ‘modern eBPF’ - you can learn more about them in the Falco docs.

On the Google Cloud side, GKE uses Container-Optimized OS (COS) as the default operating system for its worker node pools. COS is a security-enhanced operating system that limits access to certain parts of the underlying OS. Because of this security constraint, Falco cannot insert its kernel module to process system calls. However, COS does support eBPF, so that’s the option we’ll use (more specifically we’ll use the classic eBPF probe)

Installing Falco via the Google Cloud Marketplace

Note: If you’d like to follow along, you’ll need to ensure your Google Cloud account has the appropriate permissions.

Installing Falco via the Google Cloud Marketplace is a pretty straightforward process.

Log into your Google Cloud account, and ensure you have the required permissions to deploy a new GKE cluster or operate an existing one.
Navigate to the Falco offering in the Google Cloud Marketplace.
Click the configure button.
From the next dialog you can choose the zone where your GKE cluster will run as well as the network and subnet on which it will run. For this walkthrough, the default values are fine.
You then choose whether or not you’d like to deploy Falco onto a new GKE cluster or use an existing one. Be aware that if you click Create New Cluster, Google Cloud will immediately start deploying a new cluster. Also, any Autopilot clusters that you have in your project will be grayed out and cannot be selected.
You can then choose which namespace Falco will run in. To keep things consistent with the rest of this blog, change it from default to falco.
Again, to keep things consistent with the rest of the blog, change the app instance name to falco.
Falco rules have different priority levels, you can choose the minimum priority level you’d like to run. The priority levels are ordered by severity, and typically the higher you make the minimum level, the fewer alerts you will receive (which helps to cut down on noise). For this example just leave it as debug.
Stackdriver is the old name for Google Cloud’s logging and monitoring suite. If you’d like to examine Falco’s metrics (not the actual alerts, but metrics on how Falco is performing) you can select that option. We won’t be covering that in this blog, so go ahead and leave it unchecked.
Click DEPLOY to deploy Falco onto the target cluster. (If you choose to deploy a new cluster, you will need to wait until that finishes to click the DEPLOY button.)

With that, Falco should be running on your GKE cluster. You can skip the next section, and continue with “Testing Falco”.

Installing Falco with Helm

Helm is the defacto way to install Falco on Kubernetes. Falco maintains an official Helm chart, and that chart is maintained as part of the overall Falco project.

If you’d like to follow along, you will need the following:

A Google Cloud account with appropriate permissions
A GKE cluster that you can operate
Helm and kubectl installed on your local computer or, alternatively, you can use Google Cloud Shell which has both Helm and kubectl already installed.

NOTE: Ensure that your kubectl context is set to the cluster on which you wish to install Falco.

With the pre-requisites out of the way, let's get started with the actual install.

Add the Falco chart to the Helm repository.

helm repo add falcosecurity \
https://falcosecurity.github.io/charts && \
helm repo update

Create the falco namespace for Falco to run in.
```
kubectl create namespace falco
```
Use Helm to deploy Falco. Notice that we use the driver.kind parameter to set the kernel driver to the eBPF probe.
```
helm install falco \
-n falco \
--set tty=true \
--set driver.kind=ebpf \
falcosecurity/falco
```
Wait for the Falco pods to come online.
```
kubectl get pods -n falco -w
```
Eventually you should see something similar to this:
```
falco-wfglg 2/2 Running 0 76s
falco-mdrlb 2/2 Running 0 91s
falco-7vxz6 2/2 Running 0 91s
```
Note: You will see one Falco entry for each of the nodes in your cluster. In this case, Falco is running on a 3-node cluster, so there are 3 entries.

Verify Falco is running correctly by examining the logs.

kubectl logs -n falco -c falco -l app.kubernetes.io/name=falco

You should see entries similar to this:

Fri Nov 3 15:48:07 2023: Falco version: 0.36.2 (x86_64)
Fri Nov 3 15:48:07 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Fri Nov 3 15:48:07 2023: Loading rules from file /etc/falco/falco_rules.yaml
Fri Nov 3 15:48:07 2023: The chosen syscall buffer dimension is: 8388608 bytes (8 MBs)
Fri Nov 3 15:48:07 2023: Starting health webserver with threadiness 2, listening on port 8765
Fri Nov 3 15:48:07 2023: Loaded event sources: syscall
Fri Nov 3 15:48:07 2023: Enabled event sources: syscall
Fri Nov 3 15:48:07 2023: Opening 'syscall' source with BPF probe. BPF probe path: /root/.falco/falco-bpf.o

Falco is now successfully running on your GKE cluster. The next step is to simulate some suspicious activity and verify that Falco detects it.

Testing Falco

One of Falco’s default rules fires an alert if someone shells into a running container. Follow the steps below to fire off that rule.

Start an Alpine container and have it sleep so it stays running.
```
kubectl run alpine –image alpine – sh -c "sleep infinity"
```
Execute a shell on the Alpine running container.
```
kubectl exec -it alpine -- sh -c "ls -al"
```

Now check the Falco logs to see the alert.

kubectl logs -c falco -n falco -l app.kubernetes.io/name=falco |\
grep "Notice"

You should see something like this:

18:52:06.630209324: Notice A shell was spawned in a container with an attached terminal (evt_type=execve user=root user_uid=0 user_loginuid=-1 process=sh proc_exepath=/bin/busybox parent=runc command=sh -c ls -al terminal=34816 exe_flags=EXE_WRITABLE container_id=e71eac85a570 container_image=docker.io/library/alpine container_image_tag=latest container_name=alpine k8s_ns=default k8s_pod_name=alpine)

Notice all the details the alert provides including the container ID, image, and name, as well as the executed command.

Conclusion

As mentioned at the outset, one of the big advantages of running a managed Kubernetes service is that a lot of the heavy lifting for hardening the cluster has been done for you. However, by using Falco to provide runtime insights into the activity on your cluster you can help ensure that the cluster is being operated responsibly or has not been compromised by any bad actors.

If you’d like to learn more about Falco, head on over to the docs or our GitHub repository. We also have our own channel (#Falco) on the Kubernetes Slack server.

Blog: Falco Weekly 46 - 2023

Fri, 17 Nov 2023 00:00:00 +0000

This is the first of a series of weekly blog post whose aim is to give a quick overview about the development of Falco and its related projects.

What happened in Falco this week?

Let's go through the major changes that happened in various repositories under the falcosecurity organization.

Libs

Lots of cleanups happened in the libs repo; the most outstanding ones being:

udig engine removal (https://github.com/falcosecurity/libs/pull/1485)
dropped legacy metadata clients for k8s and mesos (https://github.com/falcosecurity/libs/pull/1478)
cleaned up proc callback handling code (https://github.com/falcosecurity/libs/pull/1471)

Please, note that the removal of the legacy k8s client is part of a bigger effort to entirely rewrite it as a plugin, with a more future proof architecture and language.
See the tracking issue: https://github.com/falcosecurity/libs/issues/987.

All of these cleanups account for ~26k loc removed!! :rocket:

Moreover, some fixes landed:

removed some more Undefined Behavior warnings from integer copies (https://github.com/falcosecurity/libs/pull/1481)
solved win32 linking issues with zlib (https://github.com/falcosecurity/libs/pull/1484)
prevent libbpf stats from being collected with no bpf stats (https://github.com/falcosecurity/libs/pull/1487)

Finally, some new features were merged:

libraries will now be properly installed under CMAKE_INSTALL_LIBDIR (https://github.com/falcosecurity/libs/pull/1101)
added ppc64le experimental support for modern bpf driver (https://github.com/falcosecurity/libs/pull/1475)
upgraded openssl to 3.1.4 (https://github.com/falcosecurity/libs/pull/1488)

Also, we now have a target release date and a tracking issue for libs 0.14 and next driver release: https://github.com/falcosecurity/libs/issues/1482.

Falco

Now Falco builds and runs on win32 and osx too! https://github.com/falcosecurity/falco/pull/2889 While Falco won't ship for these platforms, we will now have proper CI for them.

Following the huge round of cleanups in libs, k8s and mesos related configs and options were removed: https://github.com/falcosecurity/falco/pull/2914. Also, another small cleanup relative to the legacy k8saudit implementantion (not the plugin one!) was merged: https://github.com/falcosecurity/falco/pull/2913.

Falcoctl

While the code for the new driver-loader feature for falcoctl is being reviewed (part of the effort to drop falco-driver-loader script (https://github.com/falcosecurity/falcoctl/issues/327 and https://github.com/falcosecurity/falco/issues/2675), some features landed too:

fetch config layer for a specific platform (https://github.com/falcosecurity/falcoctl/pull/349)
added a new artifact manifest command (https://github.com/falcosecurity/falcoctl/pull/351)

Others

A new repo, k8s-metacollector, was donated to the falcosecurity.
It is a self-contained module that fetched metadata from kubernetes API server and dispatches them to Falco instances via gRPC.
A new plugin is being developed to receive those metadata from gRPC, and will be shipped with Falco 0.37.

Driverkit gained support for SUSE Linux Enterprise: https://github.com/falcosecurity/driverkit/pull/304.

Let's meet 🤝

We meet every week in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Aldo, Andrea, Federico

Blog: Introducing the new Falco training course, by CNCF, Linux Foundation, and Sysdig

Mon, 06 Nov 2023 00:00:00 +0000

Detecting Cloud Runtime Threats with Falco (LFS254) is the new Falco training course created by CNCF, Linux Foundation, and Sysdig. We're very excited about this new immersive course designed to enhance your expertise in securing cloud-native applications through hands-on learning.

Detecting Cloud Runtime Threats with Falco (LFS254) is a 20-hour course focused on runtime security. It covers what is runtime security and how Falco is a powerful tool designed to detect anomalous activity in applications. From Falco's history and design principles to its architecture, to how it addresses cloud security challenges.

This course is designed for IT professionals, security analysts, DevOps engineers, and anyone interested in cloud security.

Why?

In a rapidly evolving digital landscape with a surge in cloud adoption, the importance of comprehending and deploying robust security solutions, such as Falco, cannot be overstated. Regrettably, cloud-native technologies, particularly cloud-native security, are relatively novel, and there exists a gap in knowledge and expertise for addressing these emerging challenges. Our mission is to bridge this knowledge gap and empower individuals to tackle cloud and container security complexities effectively. Through accessible training, we aspire to contribute to narrowing the talent deficit in these pivotal domains.

How?

In this course, you'll embark on a journey of securing cloud-native environments. The course breaks down complex concepts, making them accessible and actionable. Its self-paced nature provides the flexibility to learn at your own rhythm, accommodating your personal and professional commitments. This structure allows you to digest intricate concepts and apply them bit by bit, ensuring a deeper and more lasting comprehension.

Course Structure

The course begins with an introduction to Falco, encompassing its history, design principles, and its broader role in cloud security. It then delves into the core components of Falco, explaining its architectural design and walking you through the setup and operation of Falco. Moving forward, the course explores the significance of the system call data source in host security, offering insights into the nature of system calls, observation techniques, and best practices for efficient data collection. It further showcases Falco's versatility by examining its utilization of diverse data sources such as Github, Cloudtrail, and Kubernetes Audit logs through its Plugin Framework.

The course also thoroughly covers conditions and fields, delving into the realm of Falco default rules and their integration with security frameworks. It then provides comprehensive guidance on customizing Falco rules to align with specific requirements. The course also addresses Falco outputs and introduces Falcosidekick as a valuable output management and customization tool.

Finally, the course guides you through Falco's configuration process and fine-tuning strategies. It concludes by streamlining the process of writing new Falco rules, presenting a development methodology, along with key considerations to bear in mind when crafting rules.

Sneek Peek

In one of the course exercises, we explain the Log4j vulnerability and the Log4Shell exploit. We detail each step of the attack, allowing you to simulate it in the lab environment.

Then, we walk you through how to write a new rule to detect this type of attack in Falco.

Enroll Now

Ready to embark on this transformative journey? Visit the course page to enroll and step into the world of cloud-native security mastery with Falco.

Blog: Introducing Falco 0.36.2

Fri, 27 Oct 2023 00:00:00 +0000

Today we announce the release of Falco 0.36.2 🦅!

Fixes

Falco's 0.36.2 release is a small patch addressing a few bugs. It includes the following:

Fixed a possible segfault caused by uninitialized variable in libsinsp::next() method call. (https://github.com/falcosecurity/falco/issues/2878)
Improved supported program type detection for modern BPF; this ensures we can actually be sure that our BPF program type is unsupported when returning an error to the user. (https://github.com/falcosecurity/libs/pull/1404)
Fixed a subtle bug in rawarg filtercheck for non-string types. (https://github.com/falcosecurity/libs/pull/1428)
Fixed an uninitialized variable in the libscap bpf engine that lead to stdin getting closed while Falco soft restarted. (https://github.com/falcosecurity/libs/issues/1448)

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.36.2, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

What next? 🔮

The community is active on many topics and we hope to deliver great features and many stability fixes once again during the next release cycle!

The old falco-driver-loader script is showing its age and it's time to work on a more maintainable solution. falcoctl is a great candidate to host everything driver related, implement new features and make our lives easier when we need to install Falco drivers on a new machine.
Our rule framework is brand new and we forsee many improvements and active development work on it.
The latest Falco versions brought many improvements to the plugin framework; we wish to use those to create a more scalable Kubernetes client plugin that will be able to withstand much heavier loads and will be easier to maintain.

And many, many, more enhancements!

To get a weekly reminder of all the great stuff happening in the Falco lands, make sure to join the #falco channel on the Kubernetes Slack!

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions:

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Federico

Blog: Integrate Runtime Security into Your Environment with Falcosidekick

Tue, 24 Oct 2023 00:00:00 +0000

If you’re looking to integrate runtime security into your existing environment, Falco is an obvious choice. Falco is a Cloud Native Computing Foundation backed open source project that provides real-time threat detection for cloud, container, and Kubernetes workloads. With over 80 million downloads Falco has been adopted by some of the largest companies in the world.

However, what many Falco users discover early on is that Falco’s default event output is rather limited. Out of the box, Falco can only send output to five different endpoints: syslog, stdout, stderr, and gRPC or HTTPS endpoints.

While these outputs might be enough to get you started, most practitioners want to integrate Falco with the tooling they already use. This is where Falcosidekick comes in.

Falcosidekick is a companion (i.e. a side-kick ;)) project for Falco that allows Falco events to be forwarded to 60 different services (with more being added all the time) allowing practitioners to monitor and react to Falco events with the tools they are already using.

For example, if you’d like to receive immediate notifications of suspicious activity you can forward Falco events to chat programs such as Slack or Telegram, alerting platforms like PagerDuty or AlertManager, or, of course, email. In order to minimize noise, you can expressly set the level on which to notify, for example, warning-level events might be delivered via email, while critical or higher-level events are sent via chat or directed to your alerting platform.

If you want to programmatically address certain events, Falcosidekick integrates with a bunch of different services including functions as a service platforms like AWS Lambda, GCP Cloud Run and Cloud Functions, or Knative. Alerts can also be sent to message queues like Amazon SNS, Apache Kafka, or RabbitMQ. These integrations offer almost endless possibilities for building out response systems for events.

For instance, let’s say you’re running Falco on your Kubernetes cluster, and Falco discovers that someone ran a program that wasn’t part of a container’s base image. In this case, you might choose to have Falcosidekick send that event over to AWS Lambda. Lambda would receive the alert and execute a function to apply a label to the pod that houses the suspect container resulting in the pod being taken out of service and a new one spawned to replace it.

Falcosidekick can also send events to traditional monitoring and logging platforms such as Elasticsearch, Prometheus, Grafana, or Datadog (and many many others). These platforms can be used to aggregate and process alerts en masse so you can spot trends or anomalies.

In addition to looking at the actual Falco events, Falcosidekick also provides metrics on its own performance. There are also service endpoints to check if Falscosidekick is running and its general health.

Falcosidkick features its own web-based user interface to get a quick view into your Falco environment. This UI offers reporting on Falcosidick’s performance metrics as well as a feed for events reported by the connected Falco instances.

If you want to archive Falco events, Falcosidekick can send them to Redis or various object storage services like AWS S3 or Google Cloud Storage.

Falcosidekick runs as a daemon and can be easily installed via multiple mechanisms including as a Docker container, or via Helm. With Helm, you can either install Falcosidekick when you install Falco or add it to your environment later. From an architectural standpoint, Sidekick is deployed to two pods to help ensure high availability and a single Sidekick instance can receive events from multiple Falco environments.

With so many different options, Falcosidekick opens up a myriad of possibilities for integrating Falco’s runtime security capabilities into your existing tooling. If you’d like to try Falcosidekick yourself, check out Thomas’ blog post which covers how to integrate Falcosidekick with Slack. And, if you’d like to get involved with Falco or Falcosidekick come chat with us over on the #Falco channel in the Kubernetes Slack or check out the Github repos.

Blog: How we Sign and Verify Falco Plugins and Rules

Wed, 18 Oct 2023 00:00:00 +0000

Falco v0.36.0 and the Software Supply Chain (SSC) security

The latest stable Falco release, v0.36.0, alongside falcoctl 0.6.1 and the 0.7.0 Helm chart introduced new features and improvements to the security of Falco's software supply chain artifacts. Falco's two main downloadable artifacts are plugins and rule sets. They're shipped in the OCI specification format and distributed through the official Falcosecurity OCI repositories.

Software supply chain attacks aim at injecting malicious code into software components, to compromise downstream users. These types of attacks are among the primary threats in today's threat landscape.

In particular, attackers abuse trust relationships existing between the different open-source stakeholders. The increase in attacks on open-source software throughout the last few years demonstrates that attackers consider them a viable means for spreading malware.

SSC safeguards

Securing the software supply chain may seem daunting at first glance, but there are a lot of safeguards that can be put in action. And there are ways to categorize them, and ways to prioritize them.

Safeguards against supply chain attacks can be classified by control type: directive, preventive, detective, corrective, and recovery. But there ain't no such thing as a free lunch. Besides safeguard classifications, the utility-to-cost ratio can also be an important factor in deciding where to start in improving the supply chain security of software, and can be pretty easy to measure it.

There are cheap preventive safeguards that can be implemented in open source projects especially, where stakeholders platea can be pretty wide considering the contributions. For example, branch protection rules are usually simple per-code repository configurations in providers (e.g. GitHub) and alongside pull request-based flows enforcing code review quorum, are also standard best practices nowadays. The same applies to reproducible builds, dependency pinning, build steps isolation, MFA authentication to repository providers, etc.

On the other side, safeguards with a high utility value can require a considerable amount of effort to be implemented end-to-end (producer and consumer sides).

MITRE and OpenSSF increased their efforts to improve SSC security recently in general and in the open source, by providing frameworks and tools that increase the utility-to-cost ratio. MITRE for example proposes an end-to-end framework to preserve the integrity of the software supply chain. OpenSSF develops the SLSA framework that groups several security best practices for open-source projects.

Moreover, the in-toto Attestation Framework defines a fixed, lightweight Statement that communicates some information about the execution of the software supply chain, such as if the source was reviewed, or if it went through a SLSA conformant build process.

Because new attacks are being introduced almost every day, It's important to be constantly evolving your SSC security tools and processes. These safeguards should be placed as early as possible in the SSC ("shift left"). Additionally, they need to be continuously maintained in order to keep pace with attacker's efforts.

Thanks to important projects like Sigstore's cosign, software artifact signatures are one of the SSC safeguards with a high utility-to-cost ratio. Due to these factors, the Falco Software Supply Chain Working group decided to make implementing these signatures a goal for the Falco project"

Artifacts distributed as OCI

Since January 2023 we started distributing Falco plugins and rules as OCI artifacts. If you are not familiar with the concept, think about OCI artifacts as a content-addressable unit of data that can be tagged, stored and indexed in container registries alongside your container images but can hold any type of file. You can find the source and the data for Falco's artifacts in the plugins and rules repositories respectively. We are using GitHub Container Registry to publish them so you can find them in the "Package" section of each repository.

Packaging Falco's rules and plugins as container artifacts brings several advantages, such as:

We can create multiple tags, both fixed and floating, for each artifact. Keeping track of the latest version is easy as well as the latest version for each major and minor release.
Distribution is handled with a standard protocol implemented by many different container registries, which means that creating your own rules registry, either public or private, is easy and you can use your existing container distribution infrastructure (or any ready-to-use registry service as offered by all major cloud providers) without installing and maintaining any custom server.
There are standard and developing mechanisms and specifications to store artifact metadata, like link between artifacts for references to signature or Software Bill Of Materials (SBOM), so we do not need to reinvent the wheel.

In falcoctl we have used the CNCF ORAS library to manage downloading and uploading artifacts to the container registry.

Cosign keyless signing and verification

Another very interesting thing you can do with OCI artifacts is now sign them thanks to cosign! Cosign supports a very interesting mechanism called keyless signing. This allows us to perform artifact signature and verification without the need for creating, maintaining and securing long-lived and opaque signing keys.

Moreover, cosign supports OCI v1.1 being able to link security artifacts through OCI references when supported by the registry.

Implementation

Release pipelines for signed plugins and rules

Plugins and rules that we sign are packaged as OCI artifacts.

As for the diagram above, there are two distinct distribution pipelines: one is to update the general Falcoctl index and one is to distribute the single artifact, whether plugin or rule set, to the OCI registry.

Container registry update

At the same time, in the release pipeline, an OCI artifact is built with a specific build tool for each artifact updated, and a signature is generated and distributed to the registry for each one. The signature is generated with the help of Cosign using the keyless mode, which leverages OIDC with GitHub as the identity provider.

In-depth: as the pipeline is a non-interactive environment, cosign automatically uses the OIDC device flow to verify the Github action identity token and generates a certificate as proof of the identity of the user that is signing the artifact. Cosign generates a transparency log object that contains the hash of the artifact, the previously generated certificate's public key, and the signature. This object can be verified against the signature, certificate, and artifact pulled from the OCI registry.

For example, you can check with the cosign CLI that a valid signature for the plugin cloudtrail is available:

$ cosign tree ghcr.io/falcosecurity/plugins/plugin/cloudtrail:latest
📦 Supply Chain Security Related artifacts for an image: ghcr.io/falcosecurity/plugins/plugin/cloudtrail:latest
└── 🔐 Signatures for an image tag: ghcr.io/falcosecurity/plugins/plugin/cloudtrail:sha256-ec47c7448d455b6bb84f39d5c400c2fa207ab5fbe8c21327c6c1d06a3f1eecb4.sig
└── 🍒 sha256:f8aae8441b2d41419ae99d4979d6eab1543d81a269e4721fdbff0d837571aa4d

And you can inspect with the oras CLI its manifest, by taking the tag that contains the digest content identifier of the artifact signed (i.e. ghcr.io/falcosecurity/plugins/plugin/cloudtrail:sha256-ec47c7448d455b6bb84f39d5c400c2fa207ab5fbe8c21327c6c1d06a3f1eecb4.sig):

$ oras manifest fetch ghcr.io/falcosecurity/plugins/plugin/cloudtrail:sha256-ec47c7448d455b6bb84f39d5c400c2fa207ab5fbe8c21327c6c1d06a3f1eecb4.sig | jq
{
"schemaVersion": 2,
"mediaType": "application/vnd.oci.image.manifest.v1+json",
"config": {
"mediaType": "application/vnd.oci.image.config.v1+json",
"size": 242,
"digest": "sha256:ef254edd0dd0ae2b9b6ef0471c915682c0ba31858fe45bc817a951cad715fe8d"
},
"layers": [
{
"mediaType": "application/vnd.dev.cosign.simplesigning.v1+json",
"size": 263,
"digest": "sha256:f8aae8441b2d41419ae99d4979d6eab1543d81a269e4721fdbff0d837571aa4d",
"annotations": {
"dev.cosignproject.cosign/signature": "MEUCIGvgIIu9JX5vVCZ5YHsoVe3wpVoIianJvzkCZUIClVlAAiEAyx7GZztNDzCJN5Fjg8vwq0pv7PH3TrseIXN1ou35Jmw=",
"dev.sigstore.cosign/bundle": "{\"SignedEntryTimestamp\":\"MEYCIQDsAPhTbHc7w+LHpipbx0HSEIlmjjbAszF5ChHRO1jyKgIhAJrDmeLtzO72fGD1WHabY8Mwz7Qph9CzDg+EMkC9JR7A\",\"Payload\":{\"body\":REDACTED}}",
"dev.sigstore.cosign/certificate": "-----BEGIN CERTIFICATE-----\nREDACTED\n-----END CERTIFICATE-----\n",
"dev.sigstore.cosign/chain": "-----BEGIN CERTIFICATE-----\nREDACTED=\n-----END CERTIFICATE-----\n-----BEGIN CERTIFICATE-----\nREDACTED\n-----END CERTIFICATE-----"
}
}
]
}

You can see that it contains a layer of type application/vnd.dev.cosign.simplesigning.v1+json. Also, its annotation metadata helps cosign to authenticate the signature against the Sigstore's Rekor transparency log.

Index update

Each artifact type has its own registry metadata file. The general falcoctl index.yaml, which refers to all artifact types, is updated when a modification to the specific artifact type metadata occurs. In order to verify a keyless signature we need to know in advance some additional metadata about the type of signature and how to verify its authenticity. We are adding this data to the registry metadata so that it can be published to the index by the pipeline. We will see this content in the next section.

Transparent signature verification in `falcoctl`

We have seen that every artifact now has signature data available. This is nice, but to complete this feature we need to make sure that the signature data is automatically checked when we download an artifact. In the same way, package signatures are verified when you run apt-get install, falcoctl needs to do the same to the OCI artifacts it downloads.

When designing this feature we decided not to change the existing usage of falcoctl. For example, you may already know that when you decide to install a new apt repository you need to also import its public key. When installing Falco artifacts with falcoctl, we can avoid this step and make verification transparent, thanks to keyless signing by trusting the Sigstore Fulcio CA. In the section above we introduced the concept of index files. Every time you use falcoctl, there is an index file that tells details about artifacts distributed by the Falco organization. For instance, the cloudtrail plugin has an entry with several pieces of metadata, including the OCI artifact coordinates: ghcr.io/falcosecurity/plugins/plugin/cloudtrail. Of course, anyone can distribute and install their own index file for their plugins and rules.

What we have recently added is the signature part:

signature:
 cosign:
 certificate-oidc-issuer: https://token.actions.githubusercontent.com
 certificate-identity-regexp: https://github.com/falcosecurity/plugins/

This section indicates how to check the signature for this artifact. This way the index file contains information on how to download and how to verify the artifact.

Upon request to install the artifact falcoctl is going to check the index file, resolve the name, identify the digest from the registry, pull the signature, and validate it by using cosign as a library. Only once the signature is verified, falcoctl will allow the installation of the plugin with that specific digest. If the signature data does not match the installation will be halted with an error.

You can try it yourself to make sure that the signature works!

sudo docker run -it --entrypoint sh falcosecurity/falcoctl:0.6.1
# falcoctl artifact install cloudtrail:0.9 --rulesfiles-dir=/tmp --plugins-dir=/tmp

But there is more! If you have cosign installed you can independently verify the signature without using falcoctl or any index file:

$ cosign verify ghcr.io/falcosecurity/plugins/plugin/cloudtrail:0.9.0 --certificate-oidc-issuer=https://token.actions.githubusercontent.com --certificate-identity-regexp=https://github.com/falcosecurity/plugins/
Verification for ghcr.io/falcosecurity/plugins/plugin/cloudtrail:0.9.0 --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The code-signing certificate was verified using trusted certificate authority certificates
[...]

This is one of the main advantages of relying on widely adopted container artifacts signing technologies of the cloud native world. Authentication of the artifacts is decoupled from the producer, and consumers can always choose the method they prefer.

Security Analysis

With the above scheme, we made steps forward and implemented best practices towards better supply chain security in the Falco ecosystem. We would like to conclude our discussion with the main takeaways for a security practitioner.

Safeguards against registry tampering, secure by default

Making use of container registries to distribute artifacts is convenient, and allows for easy replication of the data across multiple regions or providers. But even if one of those registries is at risk of being compromised , clients will be protected from potentially malicious applications. If an artifact is replaced in the registry without going through the appropriate approval and open source release pipeline, clients will identify that the signature does not match and automatically refuse to install any compromised package. This is only possible because cosign signature information is distributed in a file that is separate from the artifact itself. With falcoctl 0.6.1 this feature is on by default and requires no extra action from users.

Independent signature verification

It is now possible for all Falco rules and plugins users to independently verify the authenticity of every artifact they download. We implemented this feature staying true to the open nature of Falco, so by design, you do not need any Falco-specific tool (like falcoctl) to verify your artifact and you can do it manually with cosign to implement even tighter controls.

Attack surface

As security practitioners, we should always analyze the attack surface of the security schemes we design and update our threat model accordingly. In this case, we can identify that:

If the source repositories are compromised, so will the packages. Of course, the open-source repositories act as the single source of truth for our plugins and rules. If an attacker manages to merge malicious code and release a malicious plugin they will be able to compromise users once they update or install a new version. Release pipelines are the only component that can generate a proper signature for the artifacts.
If the index file is compromised, so will the packages. The index file maps artifact names (e.g. cloudtrail) to their location in the container registry (e.g. ghcr.io/falcosecurity/plugins/plugin/cloudtrail). If that entry is modified to point to an attacker-controlled registry, possibly alongside an attacker-controlled signature, clients will download a potentially malicious artifact instead of the official one.
At this point in time, the entire infrastructure is hosted by GitHub, making it a potential single point of failure and an entity that the Falco project completely trusts. However, even if parts of the infrastructure (container registry, CDN for index distribution, ...) were not, GitHub release pipelines would still have permissions or access tokens to operate those locations.

Let's keep in touch

Security is a never ending effort, and supply chain security is no exception. Falco maintainers and community are committed to keep improving all security aspects of the Falco project.

We'll keep you up to date with the next steps and remember, if you want to be involved you're more than welcome, don't hesitate to reach out to the community and the Falco Software Supply Chain Security Working Group in the #falco-sscs-wg Slack channel in the CNCF workspace.

Blog: Introducing Falco 0.36.1

Mon, 16 Oct 2023 00:00:00 +0000

Today we announce the release of Falco 0.36.1 🦅!

Fixes

Falco's 0.36.1 release is a small patch aimed at protecting our uses by addressing a few minor bugs. It includes the following:

Address a HIGH severity vulnerability in libcurl CVE-2023-38545, bumping the library to the patched version 8.4.0. You can find more details in the section below.
The legacy eBPF probe can now handle systems with CPU hotplug enabled, opening the right number of kernel buffers. (https://github.com/falcosecurity/falco/issues/2843)
Remove a no longer useful experimental Falco config outputs_queue.recovery. This was introduced in Falco 0.36.0 as an experiment.
Fix a possible segfault caused by a faulty implementation of timer_delete. (https://github.com/falcosecurity/falco/issues/2850)

Thanks to everyone in the community for helping us in spotting these annoying bugs 🐛! You make Falco successful 🦅!

Thanks as always to the Falco maintainers for their support and effort during the entire release process.

Vulnerability in libcurl

A HIGH severity vulnerability in libcurl, CVE-2023-38545, was disclosed alongside a patched version (8.4.0). We would like to answer the main question you might have about it: Does it affect Falco?

According to the excellent in-depth description of the bug, this can only be triggered if both conditions below are true:

A SOCKS5 HTTP(S) proxy has been configured. This happens if you have set the standard environment variables that control proxy connections, such as http_proxy/https_proxy/no_proxy or libcurl-specific ones as indicated in the advisory or the libcurl documentation.
An attacker controls the server that Falco is connecting to, namely the server configured to receive http_output or a custom prebuilt driver repository server, and the SOCKS5 proxy is "slow enough" to allow the attack to happen.

While it may be rare that users have an exploitable environment, it's still a possibility. For this reason, Falco maintainers decided to ship this patch release 🦅

Try it! 🏎️

As usual, in case you just want to try out the stable Falco 0.36.1, you can install its packages following the process outlined in the docs:

Prefer to use a container image? No problem at all! 🐳

You can read more about running Falco with Docker in the docs.

You can also find the Falcosecurity container images on the public AWS ECR gallery:

What next? 🔮

The community is active on many topics and we hope to deliver great features and many stability fixes once again during the next release cycle!

The old falco-driver-loader script is showing its age and it's time to work on a more maintainable solution. falcoctl is a great candidate to host everything driver related, implement new features and make our lives easier when we need to install Falco drivers on a new machine.
Lately we have expanded the syscall coverage that Falco can provide. We wish to improve these efforts across all drivers with even more 32 bit syscalls.
Our rule framework is brand new and we forsee many improvements and active development work on it.
The latest Falco versions brought many improvements to the plugin framework; we wish to use those to create a more scalable Kubernetes client plugin that will be able to withstand much heavier loads and will be easier to maintain.

And many, many, more enhancements!

Let's meet 🤝

We meet every Wednesday in our community calls, if you want to know the latest and the greatest you should join us there!

If you have any questions

Join the #falco channel on the Kubernetes Slack
Join the Falco mailing list

Thanks to all the amazing contributors!

Cheers 🎊

Andrea, Luca

Blog: Linux Introspection - From BPF to Wireshark to Falco

Wed, 11 Oct 2023 00:00:00 +0000

Falco, an open source innovation, was conceived with the vision of crafting a flexible and robust rules engine atop the Sysdig libraries. This initiative aimed to furnish a potent tool for the detection of aberrant behaviors and intrusions within modern applications, akin to the Snort paradigm but tailored to the realm of system calls and finely tuned for cloud environments.

Nevertheless, it's important to recognize that Falco and Wireshark represent distinct facets of this evolutionary process. Falco offers ongoing surveillance akin to Snort, while Wireshark specializes in interactive endpoint network traffic analysis.

The Need for Modern System Introspection

Part of this journey has been the emergence of cloud native apps. From the early days of BPF (Berkley Packet Filter) and libpcap (a portable C/C++ library for network traffic capture), which laid the foundation for network packet analysis, to the familiar graphical user interface of Wireshark, our understanding of network data has undergone profound changes. This article embarks on a journey through this transformation, shedding light on how tcpdump and libpcap sparked an explosion of packet-based analysis and runtime security tools exemplified by Wireshark and Snort.

Wireshark, Snort, Nmap, Kismet, ngrep, and a bunch of other tools started at around the same time and are all evolutionary branches of tcpdump and libpcap.

However, as cloud computing continues to reshape the technological landscape, traditional network packet analysis tools have found themselves grappling with an evolving challenge: the cloud itself. Cloud native applications have ushered in a new era of complexity and dynamism, rendering many existing visibility solutions obsolete. This shift necessitated a fresh perspective on network monitoring, leading to the birth of Falco, a tool poised to be the Snort of the cloud.

Starting the story with Network Packet Analysis

During the late 1990s Internet boom, the demand for computer networks skyrocketed, leading to an increased need for monitoring, troubleshooting, and securing these networks. Regrettably, the available network visibility tools of that era were prohibitively expensive for many operators, leaving them grappling with a lack of insights.

Consequently, teams worldwide embarked on a mission to address this predicament. Their efforts revolved around expanding existing operating systems to incorporate packet capture capabilities, essentially transforming off-the-shelf computer workstations into devices capable of residing on a network and capturing all inbound and outbound data packets from other workstations. One such solution was the Berkeley Packet Filter (BPF), crafted to extend the functionality of the BSD (Berkeley Software Distribution) operating system kernel.

For Linux users, the term 'eBPF' may ring a bell – a virtual machine renowned for securely executing arbitrary code within the Linux kernel. Remarkably, eBPF has evolved into a powerful and flexible technology over the years. However, its origins trace back to a modest programmable packet capture and filtering module designed for BSD Unix.

The BPF team introduced a game-changing library known as 'libpcap,' which enabled any program to capture raw network packets. It was developed in order to make tcpdump more useful. For instance, it gave the ability to filter packets. Since then, a bunch of spin-off networking projects would emerge on the scene. In 1998, a GUI-based open source protocol analyzer named 'Ethereal' (later renamed Wireshark) was introduced, eventually becoming the gold standard in packet analysis that persists to this day. \

What unites 'tcpdump,' Wireshark, and numerous other popular networking tools is their ability to access a data source that is rich, accurate, and reliable, all collected in a nonintrusive manner: raw network packets. This fundamental concept will be central to our discussion moving forward.

The evolution of Packet-Based Intrusion Detection Systems

Introspection tools, such as tcpdump and Wireshark, naturally emerged as the initial applications harnessing the capabilities of the BPF packet capture stack. However, as time progressed, innovative applications for packet data began to surface. Enter Snort, an open source, packet-based runtime security tool that shares common ground with Falco. Much like Falco, Snort operates as a rule engine, processing packets acquired from network traffic. Like its cloud native counterpart, Snort boasts an extensive library of pre-configured rules designed to identify threats and unwarranted activities by scrutinizing packet content, protocols, and payload data. The success of Snort served as a catalyst for the development of similar tools, including Suricata and Zeek.

What truly empowers tools like Snort is their proficiency in assessing the security of networks and applications in real time, even as these applications run. This real-time focus proves invaluable by delivering immediate protection with a unique emphasis on runtime behavior, enabling the detection of threats rooted in vulnerabilities that may remain undisclosed.

The issue with Network Packet Capture in the Cloud

The utilization of network packets as a foundational data source has spawned a thriving ecosystem. Nonetheless, several emerging trends have gradually eroded the viability of packets as an unequivocal source of information.

First, the task of comprehensively collecting packets has grown increasingly complex, especially within environments such as the cloud, where access to routers and network infrastructure is constrained. Second, the proliferation of encryption and network virtualization has posed formidable challenges in extracting valuable insights from network traffic. Lastly, the ascent of containerization and orchestrators like Kubernetes has rendered infrastructures more elastic while concurrently complicating the reliable collection of network data.

Once again, a dynamic new ecosystem was unfolding, yet the means to effectively troubleshoot and secure it remained elusive.

System Calls are the New Network Packets

Before the emergence of Falco, an open source tool known as 'Sysdig Inspect' was crafted with a primary focus on the collection of packet data within cloud native ecosystems. This was achieved through the capture of system calls, often referred to as syscalls, originating from the kernel of the operating system.

Syscalls, as a data source, offer a richness that surpasses that of mere network packets. They encompass a wide spectrum of activities, extending beyond network data to encompass file I/O operations, command executions, interprocess communication, and more. Syscalls stand out as an ideal data source for cloud native environments as they can be harnessed from the kernel, catering to both containerized environments and cloud instances. Moreover, the process of collecting syscalls is characterized by its simplicity, efficiency, and non-invasiveness.

The architecture of Sysdig comprised a kernel capture probe, making use of either the default, loadable kernel module or leveraging eBPF. To facilitate the development of capture programs, Sysdig offered a suite of libraries, enabling seamless integration with modern cloud native technologies such as Kubernetes and various orchestrators. This versatility addressed the shortcomings observed in environments where traditional solutions like Snort and Wireshark fell short. Additionally, Sysdig provided a command-line tool replete with decoding and filtering functionalities, tailored to accommodate the prevalent network packet workflows essential in cloud environments, where the ease of filtering and scriptability of trace files is paramount.

Falco - the evolution of Wireshark to the Cloud

Drawing from our comprehension of how Snort introduced a rule-based engine for scrutinizing network traffic to identify suspicious activity, an evolution that implemented Wireshark's network introspection, and how Sysdig expanded the scope of visibility within cloud native environments by delving into system calls, effectively departing from sole reliance on Wireshark's libpcap framework. It logically followed that an Intrusion Detection System (IDS) solution would emerge, featuring a sophisticated rule-based engine tailored for cloud native workloads while harnessing the capabilities of eBPF and the kernel's system call architecture.

Falco's rule engine drew inspiration from Snort's design but operated within a far more expansive and versatile dataset, seamlessly integrated with the Sysdig libraries. While its default ruleset may be more concise than Snort's, Falco empowers users to craft intricate rules that trigger in real-time based on arbitrary contextual factors. These factors encompass a wide array of scenarios, including access to sensitive data, mode transitions, unexpected network connections, socket alterations, compliance breaches, and more. Given its capacity to monitor all activities on a server or node through system calls, Falco functions as a real-time intrusion detection tool, mirroring Wireshark's role in providing real-time network analysis for endpoints.

Falco for Cloud Native Security

In the journey from the early days of BPF to the widespread adoption of Wireshark, we've witnessed the remarkable evolution of system introspection tools, each one contributing to the ever-expanding landscape of cybersecurity. However, as cloud native computing and microservices architectures become the new norm, a new champion has emerged: Falco. Falco represents the cutting edge of intrusion detection, specifically designed to tackle the intricacies and challenges posed by cloud native hosts and workloads. With its real-time behavioral monitoring, container awareness, and comprehensive rule sets, Falco stands as a testament to the adaptability and innovation in the world of cybersecurity. As the digital landscape continues to evolve, Falco is the tool of choice for those who prioritize the security and integrity of their cloud native environments. It's not just a system introspection tool; it's the future of protecting what matters most in this rapidly changing world of technology.

If you want to try out Falco, check out our Getting Started documentation. Join our community at #falco channel within Kubernetes Slack.

Falco – The Falco blog

Blog: Introducing Prempti: Falco meets AI coding agents

Agents are a black box at runtime

How Prempti works

Two modes: Monitor and Guardrails

Writing rules: Familiar territory

Rule authoring with Claude Code

Let's be honest about limitations

Getting started

Explore together with us

Blog: Introducing Falco Operator 0.2.0

What's new? TL;DR

The road to production readiness

Major features and improvements

Ecosystem components

ConfigMap support for rules and configuration

Structured API types

Redesigned OCI artifact API

Reference tracking with finalizers

Enhanced observability

Update strategy support

Server-Side Apply

Breaking changes ⚠️

Summary of breaking changes

A Helm chart is on its way

Meet us at KubeCon

Try it out

Full stack quickstart

Step by step

Stay connected

Blog: Falco at KubeCon Europe 2026 — See You in Amsterdam! 🐦

Sneak peek

Project lightning talk

Sysdig-led workshop

Conference talk

Booth demo

Thank you!

Blog: Hey Falco Flock! 🐦 Let's Soar Into 2026

👉 Take the survey here

Blog: Introducing Falco 0.43.0

What's new? TL;DR

Latest updates

Deprecations

Legacy eBPF probe deprecation

gVisor deprecation

gRPC output and server deprecation

GPG key rotation

Container plugin improvements

Falcoctl tweaks and improvements

follow polling interval increase to 1 week

Dependency resolution improvements

Support for cosign v3

Key fixes

evt.arg.filename field reintroduction

Falcoctl signature verification fixes

Signature verification fix for full reference artifacts

Signature verification fix for authenticated registries

Breaking changes and deprecations

Bump drivers minimum required kernel version to 3.10

Deprecation warnings

Try it out

Stay connected

Blog: GPG Key Rotation for Falco Packages (2026)

The Rotation Plan

Phase 1: Soft Launch (Dec 12, 2025)

Phase 2: Hard Cut-Over (Jan 12–17, 2026)

Action Items for Users

New Users

Existing Users

Summary

Blog: Introducing Falco 0.42.0

What's new? TL;DR

Major features and improvements

Capture recording feature

Drop enter initiative

Plugin event schema versioning

Thread table auto-purging configuration

Static fields

Breaking changes and deprecations ⚠️

Event direction and evt.dir deprecation

`follow` polling interval increase to 1 week

`evt.arg.filename` field reintroduction

Bump drivers minimum required kernel version to `3.10`

Event direction and `evt.dir` deprecation