Falco – Falco Outputs

Docs: Output Channels

Mon, 01 Jan 0001 00:00:00 +0000

Standard Output

When configured to send alerts via standard output, a line is printed for each alert.

Here is an example:

stdout_output:
 enabled: true

10:20:05.408091526: Warning Sensitive file opened for reading by non-trusted program (user=root command=cat /etc/shadow file=/etc/shadow)

Standard output is useful when using Fluentd or Logstash to capture logs from containers. Alerts can then be stored in Elasticsearch, and dashboards can be created to visualize the alerts. For more information, read this blog post.

Standard Output buffering

If the logs are inspected by tailing container logs (e.g. kubectl logs -f in Kubernetes) it might look like events can take a long time to appear, sometimes longer than 15 minutes. This is not an issue with Falco but is simply a side effect of the system output buffering.

However, if realtime update of these logs is necessary it can be forced with the -U/--unbuffered command line option which will ensure the output is flushed for every event at the cost of higher CPU usage.

File Output

When configured to send alerts to a file, a message is written to the file for each alert. The configuration is very similar to the Standard Output format:

file_output:
 enabled: true
 keep_alive: false
 filename: ./events.txt

When the field keep_alive is set to false (default value), for each single alert:

the file is opened for appending
the single alert is written
the file is closed

If keep_alive is set to true, the file is opened before the first alert, and kept open for all subsequent alerts. Output is buffered and will be flushed only on close. (This can be changed with the --unbuffered command line option).

Notice that, regardless keep_alive settings, Falco neither rotates nor truncates the output file. If you'd like to use a program like logrotate to rotate the output file, an example logrotate config is available here.

As of Falco 0.10.0, Falco will close and reopen its file output when signaled with SIGUSR1. The logrotate example above depends on it.

Syslog Output

When configured to send alerts to syslog, a syslog message is sent for each alert. The actual format depends on your syslog daemon, but here's a simple configuration example:

syslog_output:
 enabled: true

And its respective entry in the syslog service:

Jun 7 10:20:05 ubuntu falco: Sensitive file opened for reading by non-trusted program (user=root command=cat /etc/shadow file=/etc/shadow)

Syslog messages are sent with a facility of LOG_USER.
The rule's priority is used as the priority of the syslog message.

Program Output

When configured to send alerts to a program, Falco normally starts the program for each alert and writes its contents to the program's standard input. You can only configure a single program output (e.g. route alerts to a single program) at a time.

Here you can find an example of how to configure the program_output inside the falco.yaml file:

program_output:
 enabled: true
 keep_alive: false
 program: mail -s "Falco Notification" someone@example.com

If the program cannot normally accept an input from standard input, xargs can be used to pass the Falco events with an argument. For example:

program_output:
 enabled: true
 keep_alive: false
 program: "xargs -I {} aws --region ${region} sns publish --topic-arn ${falco_sns_arn} --message {}"

When keep_alive is set to false (default value), for each alert Falco will run the program mail -s ... and write the alert to the program. The program is run via a shell, so it's possible to specify a command pipeline if you wish to add additional formatting.

If keep_alive is set to true, before the first alert Falco will spawn the program and write the alert. The program pipe will be kept open for subsequent alerts. Output is buffered and will be flushed only on close. (This can be changed with the --unbuffered command line option).

Controlling the program output

The program spawned by Falco is in the same process group as Falco and will receive all signals that Falco receives. If you want to, say, ignore SIGTERM to allow for a clean shutdown in the face of buffered outputs, you must override the signal handler yourself.
As of Falco 0.10.0, Falco will close and reopen its file output when signaled with SIGUSR1.

Example 1: Posting to a Slack Incoming Webhook

If you'd like to send Falco notifications to a slack channel, here's the required configuration to massage the JSON output to a form required for the slack webhook endpoint:

# Whether to output events in json or text
json_output: true
…
program_output:
 enabled: true
 program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX"

Example 2: Sending Alerts to Network Channel

If you'd like to send a stream of alerts over a network connection, here's an example:

# Whether to output events in json or text
json_output: true
…
program_output:
 enabled: true
 keep_alive: true
 program: "nc host.example.com 1234"

Note the use of keep_alive: true to keep the network connection persistent.

HTTP/HTTPS Output

If you'd like to send alerts to an HTTP(S) endpoint, you can use the http_output option:

json_output: true
...
http_output:
 enabled: true
 url: http://some.url/some/path/

Currently, only unencrypted HTTP endpoints and valid HTTPS endpoints are supported (i.e., invalid or self-signed certificates are not supported).

JSON Output

For all output channels, you can switch to JSON output either in the configuration file or on the command line. For each alert, Falco will print a JSON object, on a single line, containing the following properties:

time: the time of the alert, in ISO8601 format.
rule: the rule that resulted in the alert.
priority: the priority of the rule that generated the alert.
output: the formatted output string for the alert.
hostname: the name of the host running Falco (can be the hostname inside the container).
tags: the list of tags associated with the rule.
output_fields: for each templated value in the output expression, the value of that field from the event that triggered the alert.

Notice that, besides the ones included automatically, you can also include additional fields to output_fields through append_output settings in the configuration.

Here's an example:

{"hostname":"falco-xczjd","output":"13:44:05.478445995: Critical A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=kubecon container=ee97d9c4186f shell=sh parent=runc cmdline=sh -c clear; (bash || ash || sh) terminal=34816 container_id=ee97d9c4186f image=docker.io/library/alpine)","priority":"Critical","rule":"Terminal shell in container","source":"syscall","tags":["container","mitre_execution","shell"],"time":"2023-05-25T13:44:05.478445995Z", "output_fields": {"container.id":"ee97d9c4186f","container.image.repository":"docker.io/library/alpine","evt.time":1685022245478445995,"k8s.ns.name":"default","k8s.pod.name":"kubecon","proc.cmdline":"sh -c clear; (bash || ash || sh)","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}

Here's the same output, pretty-printed:

{
 "hostname": "falco-xczjd",
 "output": "13:44:05.478445995: Critical A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=kubecon container=ee97d9c4186f shell=sh parent=runc cmdline=sh -c clear; (bash || ash || sh) terminal=34816 container_id=ee97d9c4186f image=docker.io/library/alpine)",
 "priority": "Critical",
 "rule": "Terminal shell in container",
 "source": "syscall",
 "tags": [
 "container",
 "mitre_execution",
 "shell"
 ],
 "time": "2023-05-25T13:44:05.478445995Z",
 "output_fields": {
 "container.id": "ee97d9c4186f",
 "container.image.repository": "docker.io/library/alpine",
 "evt.time": 1685022245478445995,
 "k8s.ns.name": "default",
 "k8s.pod.name": "kubecon",
 "proc.cmdline": "sh -c clear; (bash || ash || sh)",
 "proc.name": "sh",
 "proc.pname": "runc",
 "proc.tty": 34816,
 "user.loginuid": -1,
 "user.name": "root"
 }
}

gRPC Output

The gRPC Output as well as the embedded gRPC server have been deprecated in Falco 0.43.0 and will be removed in a future release. Until removal and since Falco 0.43.0, using any of them will result in a warning informing the user about the deprecation. Users are encouraged to leverage another output and/or Falcosidekick, as the usage will result in an error after the removal.

If you'd like to send alerts to an external program connected via gRPC API, you need to enable both the grpc and grpc_output options as described under the gRPC Configuration section.

Docs: Alert Formatting

Mon, 01 Jan 0001 00:00:00 +0000

Previous guides introduced the Output Fields of Falco Rules and provided Guidelines on how to use them. This section highlights additional global formatting options for your deployment, complementing the information previously provided.

Adding the same output field to multiple rules by manually editing rule files can be tedious. Fortunately, Falco provides several ways to simplify this process:

Using the append_output configuration option in falco.yaml to add output text or fields to a subset of loaded rules
Adding an override to a specific rule to replace its output

Appending Extra Output and Fields with `append_output`

The append_output option can be specified in the falco.yaml configuration file. You can use it to add extra output to rules specified by source, tag, name, or to all rules unconditionally. The append_output section is a list of items that are applied in the order they appear.

Example:

append_output:
 - match:
 source: syscall
 extra_output: "on CPU %evt.cpu"
 extra_fields:
 - home_directory: "${HOME}"
 - evt.hostname

In this example:

Every rule with the syscall source will have on CPU %evt.cpu appended at the end of the regular output line.
The rule will also include the additional fields (home_directory and evt.hostname) in the JSON output under output_fields. These extra fields do not appear in the regular (text) output.
Environment variables (like $HOME) expansion is supported in the configuration file, so for extra_fields as well.

Matching Rules

The match section allows you to filter which rules are modified:

source: filters rules by source (e.g., syscall or plugin names)
rule: filters by the complete rule name
tags: filters by a list of tags (all listed tags must be present)

If multiple conditions are specified under match, all must be met for the entry to apply. If no conditions are specified—or match is omitted—then the entry applies to all rules.

Adding an Override to a Specific Rule

Note that append_output only adds output to an existing rule; it does not remove or replace existing fields. To remove or replace output fields, you can add another rule file (loaded after the original) that uses an override. For example:

- rule: Read sensitive file trusted after startup
 output: A file (user=%user.name command=%proc.cmdline file=%fd.name) was read after startup
 override:
 output: replace

Suggested Output Fields

By default, Falco can also include "suggested" fields from plugins implementing the extraction capabilities. This is especially useful if certain plugins mark some fields as recommended for output. Those fields will appear automatically in your alerts.

Below is an example configuration entry that enables suggested output fields unconditionally for any source:

append_output:
 - suggested_output: true # Enable the use of extractor plugins' suggested fields for all matching sources.

When suggested_output is set to true, any extractor plugin that provides "suggested" fields will add them to the output in the form plugin_field_name=$plugin.field_name.

Command-Line Usage

You can also specify this option on the command line via the -o flag, for example:

falco ... \
 -o 'append_output[]={"match": {"source": "syscall"}, "extra_output": "on CPU %evt.cpu", "extra_fields": ["evt.hostname"]}'

Docs: Alerts Forwarding

Mon, 01 Jan 0001 00:00:00 +0000

Falco alerts can easily be forwarded to third-party systems. Their JSON format allows them to be easily consumed for storage, analysis and reaction.

Falcosidekick

Falcosidekick is a proxy forwarder that acts as a central point for any fleet of Falco instances, using their HTTP outputs to send alerts.

It supports forwarding alerts to various outputs such as chat platforms, alerting systems, logs, storage services, and streaming systems.

Falcosidekick can also add custom fields to the alerts, filter them by priority and expose a Prometheus metrics endpoint.

The full documentation and the project repository are here.

Falcosidekick can be deployed with Falco in Kubernetes clusters with the official Falco Helm chart.

Its configuration can be made through a yaml file and/or env vars.

Outputs

Follow the links to know what are the settings of each output.

The available outputs in Falcosidekick are:

Chat

Metrics / Observability

Datadog
Influxdb
StatsD (for monitoring of falcosidekick)
DogStatsD (for monitoring of falcosidekick)
Prometheus (for both events and monitoring of falcosidekick)
Wavefront
Spyderbat
TimescaleDB
Dynatrace
OTEL Metrics (for both events and monitoring of falcosidekick)

Alerting

Logs

Object Storage

FaaS / Serverless

Message queue / Streaming

Email

SMTP

Database

Redis

Web

SIEM

AWS Security Lake

Workflow

n8n

Traces

OTEL Traces

Response engine

Falco Talon

Other

Policy Report

Installation in Kubernetes with Helm

See the available Helm values to configure Falcosidekick.

helm install falco falcosecurity/falco \
-n falco --create-namespace \
--set falcosidekick.enabled=true \
--set tty=true

Installation in Docker

Use the env vars to configure Falcosidekick.

docker run -d -p 2801:2801 -e SLACK_WEBHOOKURL=XXXX falcosecurity/falcosidekick:2.27.0

Installation on the host

Adapt the version and the architecture to your environment. You can find all the releases here.

sudo mkdir -p /etc/falcosidekick
wget https://github.com/falcosecurity/falcosidekick/releases/download/2.27.0/falcosidekick_2.27.0_linux_amd64.tar.gz && sudo tar -C /usr/local/bin/ -xzf falcosidekick_2.27.0_linux_amd64.tar.gz

See the example config file to create your own in /etc/falcosidekick/config.yaml.

To enable and start the service, you can use a systemd unit /etc/systemd/system/falcosidekick.service like this one:

[Unit]
Description=Falcosidekick
After=network.target
StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
ExecStart=/usr/local/bin/falcosidekick -c /etc/falcosidekick/config.yaml
EOF

systemctl enable falcosidekick
systemctl start falcosidekick

Falcosidekick UI

Falcosidekick comes with its own interface to visualize the events and get statistics.

Installation in Kubernetes with Helm

You can install the UI at the same moment as Falcosidekick by adding the argument --set falcosidekick.webui.enabled=true.

helm install falco falcosecurity/falco \
-n falco --create-namespace \
--set falcosidekick.enabled=true \
--set falcosidekick.webui.enabled=true \
--set tty=true

Then create a port-forward to access it: kubectl port-forward svc falco-falcosidekick-ui 2802:2802 -n falco. The default credentials are admin/admin.

The full documentation and the repository of the project are here.

Falco – Falco Outputs

Docs: Output Channels

Standard Output

Standard Output buffering

File Output

Syslog Output

Program Output

Controlling the program output

Example 1: Posting to a Slack Incoming Webhook

Example 2: Sending Alerts to Network Channel

HTTP/HTTPS Output

JSON Output

gRPC Output

Docs: Alert Formatting

Appending Extra Output and Fields with append_output

Matching Rules

Adding an Override to a Specific Rule

Suggested Output Fields

Command-Line Usage

Docs: Alerts Forwarding

Falcosidekick

Outputs

Installation in Kubernetes with Helm

Installation in Docker

Installation on the host

Falcosidekick UI

Installation in Kubernetes with Helm

Appending Extra Output and Fields with `append_output`