Blog: Monitoring your EKS clusters audit logs

Tue, 30 May 2023 00:00:00 +0000

This blog post is an update of a post of November 2022

At the beginning of the year 2022, Falco introduced a game changing feature, the Falco Plugins. They allow Falco to monitor and trigger alerts for any kind of event. Since the launch of the new plugin framework the Falco community has collaborated to create plugins for Github, AWS CloudTrail and Okta. A plugin has also replaced the way Falco consumes the Audit Logs generated by a K8s API server through a dedicated plugin. With these plugins, Falco covers more in depth the aspects of your infrastructure and allows you to use a single syntax for rules.

For months (okay, maybe years...), our adopters have asked us for a way to monitor K8s Audit Logs. The previous implementation used an internal web server to receive the logs from the Kubernetes API, although it was functional, it was a very manual process to install and manage clusters. This method didn't support clusters managed by cloud providers, such as EKS, AKS, or GKE as they had to capture the Audit Logs for their own usage and then add them to their log aggregators.

This situation is now solved thanks to the plugin framework and we're proud to announce the first release of the plugin for EKS Audit Logs: k8saudit-eks!!!

How it works

AWS captures the Audit Logs and exposes them in the CloudWatch Logs service. We have made available libs to create a clean session with the AWS API and pull logs from the relevant Cloudwatch Logs Stream. You can reuse these libs for any plugin you'd like to create for any Amazon service.

Usage

The configuration for the usage of the plugin is:

plugins:
 - name: k8saudit-eks
 library_path: libk8saudit-eks.so
 init_config:
 region: "{REGION}"
 profile: "{AWS_PROFILE}"
 shift: 10
 polling_interval: 10
 use_async: false
 buffer_size: 500
 open_params: "{CLUSTER_NAME}"
 - name: json
 library_path: libjson.so
 init_config: ""

load_plugins: [k8saudit-eks, json]

With:

init_config:
- profile: The Profile to use to create the session, env var AWS_PROFILE if present
- region: The Region of your EKS cluster, env var AWS_REGION is used if present
- use_async: If true then async extraction optimization is enabled (Default: true)
- polling_interval: Polling Interval in seconds (default: 5s)
- shift: Time shift in past in seconds (default: 1s)
- buffer_size: Buffer Size (default: 200)
open_params: A string which contains the name of your EKS Cluster (required).

If you run Falco inside an EKS cluster with a setup of an OIDC provider, the profile and region parameters can be omitted in favor of a service account + IAM Role (see the official docs).

IAM

Whatever the method you use for the authentication to the AWS API, you need to set up this minimal policy to your user/role:

{
 "Version":"2012-10-17",
 "Statement":[
 {
 "Sid":"ReadAccessToCloudWatchLogs",
 "Effect":"Allow",
 "Action":[
 "logs:Describe*",
 "logs:FilterLogEvents",
 "logs:Get*",
 "logs:List*"
 ],
 "Resource":[
 "arn:aws:logs:${REGION}:${ACCOUNT_ID}:log-group:/aws/eks/${CLUSTER_NAME}/cluster:*"
 ]
 }
 ]
}

With:

REGION: The Region of your EKS cluster
ACCOUNT_ID: The ID of the account running the EKS cluster
CLUSTER_NAME: The name of your cluster EKS cluster (same value than in your plugin configuration)

Default Rules

A good thing about Kubernetes is that it brings standards into our industry. Despite a few differences, the cluster works in the same way and produces the same format of logs. This helps us enormously. By creating the k8saudit plugin we declared the fields to extract, as well as some default rules, which we can reuse for any plugin that consumes the same Audit Logs. It is a time saver for both, developers and adopters.

You can find the proposed default rules here.

Installation

For the installation, we'll use falcoctl. We can use it as a CLI tool for a locale installation or as a sidecar for Kubernetes. See this blog post for more information.

Local installation

In this example, we'll see how to install the k8saudit-eks in a local host.

First, add the index of artifacts for falcoctl:

sudo falcoctl index add falcosecurity https://falcosecurity.github.io/falcoctl/index.yaml

Install the latest version of the plugins:

sudo falcoctl artifact install k8saudit-eks

sudo falcoctl artifact install json

As the output notices, the plugins will be now available in /usr/share/plugins:

 INFO Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO Resolving dependencies ...
 INFO Installing the following artifacts: [ghcr.io/falcosecurity/plugins/plugin/k8saudit-eks:latest]
 INFO Preparing to pull "ghcr.io/falcosecurity/plugins/plugin/k8saudit-eks:latest"
 INFO Pulling b2daf90a878e: ############################################# 100%
 INFO Pulling 60b0846d1e18: ############################################# 100%
 INFO Pulling 12c7f0f5f00e: ############################################# 100%
 INFO Artifact successfully installed in "/usr/share/plugins"

 INFO Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO Resolving dependencies ...
 INFO Installing the following artifacts: [ghcr.io/falcosecurity/plugins/plugin/json:latest]
 INFO Preparing to pull "ghcr.io/falcosecurity/plugins/plugin/json:latest"
 INFO Pulling 087dba2e76d6: ############################################# 100%
 INFO Pulling c4cb35bb528a: ############################################# 100%
 INFO Pulling 9b50ea237bc6: ############################################# 100%
 INFO Artifact successfully installed in "/usr/share/plugins"

After installing the k8saudit-eks plugin, we also need to install the rules that it uses. (Those rules are the same as the k8saudit plugin rules.):

sudo falcoctl artifact install k8saudit-rules --resolve-deps=false

And we get the rules in /etc/falco:

 INFO Reading all configured index files from "/root/.config/falcoctl/indexes.yaml"
 INFO Installing the following artifacts: [ghcr.io/falcosecurity/plugins/ruleset/k8saudit:latest]
 INFO Preparing to pull "ghcr.io/falcosecurity/plugins/ruleset/k8saudit:latest"
 INFO Pulling b8c6b1def3eb: ############################################# 100%
 INFO Pulling 6b6f3231f2b2: ############################################# 100%
 INFO Pulling 91b2b7a9944d: ############################################# 100%
 INFO Artifact successfully installed in "/etc/falco"

By default, the installation of a ruleset comes with the associated plugins. In this blog, we have installed them on our own, this is why --resolve-deps is set to false.

We now have the plugins and rules in our system. The last step is to configure Falco to use them. Modify your /etc/yaml to make it looks like the following (with the relevant values for REGION, ACCOUNT_ID, and CLUSTER_NAME):

rules_files:
 ...
 - /etc/k8s_audit_rules.yaml

plugins:
 - name: k8saudit-eks
 library_path: libk8saudit-eks.so
 init_config:
 region: "{REGION}"
 profile: "{AWS_PROFILE}"
 shift: 10
 polling_interval: 10
 use_async: false
 buffer_size: 500
 open_params: "{CLUSTER_NAME}"
 - name: json
 library_path: libjson.so
 init_config: ""

load_plugins: ["k8saudit-eks","json"]
...

You can now start Falco ( sudo /usr/bin/falco --disable-source syscall) and monitor the audit logs of your EKS cluster:

Wed May 10 14:07:18 2023: Falco version: 0.34.1 (x86_64)
Wed May 10 14:07:18 2023: Falco initialized with configuration file: /etc/yaml
Wed May 10 14:07:18 2023: Loading plugin 'k8saudit-eks' from file /usr/share/plugins/libk8saudit-eks.so
Wed May 10 14:07:18 2023: Loading plugin 'json' from file /usr/share/plugins/libjson.so
Wed May 10 14:07:18 2023: Loading rules from file /etc/k8s_audit_rules.yaml
Wed May 10 14:07:18 2023: Loading rules from file /etc/rules.d/override-k8saudit.yaml
Wed May 10 14:07:18 2023: Starting health webserver with threadiness 4, listening on port 8765
Wed May 10 14:07:18 2023: Enabled event sources: k8s_audit
Wed May 10 14:07:18 2023: Opening capture with plugin 'k8saudit-eks'

In this example, we also disable the collection of the syscalls with the option --disable-source syscall. The k8saudit-eks plugin works with a pull model, you should have only one Falco instance collecting the audit logs to avoid any duplication of alerts. We dedicate that instance to that role.

And get alerts:

14:09:17.699479000: Informational K8s Serviceaccount Created (user=system:node:ip-192-168-34-135.ec2.internal serviceaccount=default ns=default resource=serviceaccounts resp=201 decision=allow reason=)
14:09:24.046826000: Notice Attach/Exec to pod (user=kubernetes-admin pod=kubecon resource=pods ns=default action=exec command=sh)

Kubernetes installation

Next, we will see how to install the k8saudit-eks plugin with the falcoctl tool in a Kubernetes setup. We'll use the official Helm chart with the following values.yaml file:

serviceAccount:
 create: true
 annotations:
 - eks.amazonaws.com/role-arn: {ROLE_ARN}

falco:
 rules_files:
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit-eks
 library_path: libk8saudit-eks.so
 init_config:
 shift: 10
 polling_interval: 10
 use_async: false
 buffer_size: 500
 open_params: "{CLUSTER_NAME}"
 - name: json
 library_path: libjson.so
 init_config: ""
 load_plugins: [k8saudit-eks, json]

falcosidekick:
 enabled: true

driver:
 enabled: false
collectors:
 enabled: false

controller:
 kind: deployment
 deployment:
 replicas: 1

falcoctl:
 indexes:
 - name: falcosecurity
 url: https://falcosecurity.github.io/falcoctl/index.yaml
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 artifact:
 allowedTypes:
 - plugin
 - rulesfile
 install:
 resolveDeps: false
 refs: [k8saudit-rules:0, k8saudit-eks:0, json:0]
 follow:
 refs: [k8saudit-rules:0]

Let's look into this configuration section by section:

serviceAccount:
 create: true
 annotations:
 - eks.amazonaws.com/role-arn: {ROLE_ARN}

With EKS, we can use an OIDC provider and annotations on the service accounts to allow access to the AWS API ( see the official docs). This section allows you to set which ROLE_ARN to use.

falco:
 rules_files:
 - /etc/falco/k8s_audit_rules.yaml
 - /etc/falco/rules.d
 plugins:
 - name: k8saudit-eks
 library_path: libk8saudit-eks.so
 init_config:
 shift: 10
 polling_interval: 10
 use_async: false
 buffer_size: 500
 open_params: "{CLUSTER_NAME}"
 - name: json
 library_path: libjson.so
 init_config: ""
 load_plugins: [k8saudit-eks, json]

As the local installation, this section sets the used plugins.

driver:
 enabled: false
collectors:
 enabled: false

controller:
 kind: deployment
 deployment:
 replicas: 1

To collect and monitor all the syscalls, Falco must be deployed as a DaemontSet in your cluster, to have one instance per Kernel. In this example, we're deploying Falco with the k8saudit-eks plugin, which relies on a pull model. To avoid several Falco pods to collect the same audit logs and duplicate the alerts, we MUST install Falco with the k8saudit-eks plugin only once. The Helm chart allows to use a one replica deployment and to disable the syscalls and K8s metdadata collections, this is the purpose of this section.

falcoctl:
 indexes:
 - name: falcosecurity
 url: https://falcosecurity.github.io/falcoctl/index.yaml
 artifact:
 install:
 enabled: true
 follow:
 enabled: true
 config:
 artifact:
 allowedTypes:
 - plugin
 - rulesfile
 install:
 resolveDeps: false
 refs: [k8saudit-rules:0, k8saudit-eks:0, json:0]
 follow:
 refs: [k8saudit-rules:0]

The last section sets up falcoctl. It will install the requested plugins and rules and track new versions of the rules.

Now you can run your helm install command:

helm install falco-k8saudit-eks -n falco falcosecurity/falco -f ./values.yml

And see the result:

$ kubectl get pods -n falco

NAME READY STATUS RESTARTS AGE
falco-k8saudit-eks-96d7f6f99-vwjgc 2/2 Running 0 70m

$ kubectl logs falco-k8saudit-eks-96d7f6f99-vwjgc -n falco -c falco
Wed May 10 14:07:18 2023: Falco version: 0.34.1 (x86_64)
Wed May 10 14:07:18 2023: Falco initialized with configuration file: /etc/falco/falco.yaml
Wed May 10 14:07:18 2023: Loading plugin 'k8saudit-eks' from file /usr/share/falco/plugins/libk8saudit-eks.so
Wed May 10 14:07:18 2023: Loading plugin 'json' from file /usr/share/falco/plugins/libjson.so
Wed May 10 14:07:18 2023: Loading rules from file /etc/falco/k8s_audit_rules.yaml
Wed May 10 14:07:18 2023: Loading rules from file /etc/falco/rules.d/override-k8saudit.yaml
Wed May 10 14:07:18 2023: Starting health webserver with threadiness 4, listening on port 8765
Wed May 10 14:07:18 2023: Enabled event sources: k8s_audit
Wed May 10 14:07:18 2023: Opening capture with plugin 'k8saudit-eks'
{"hostname":"falco-k8saudit-eks-96d7f6f99-vwjgc","output":"14:09:17.699479000: Informational K8s Serviceaccount Created (user=system:node:ip-192-168-34-135.ec2.internal serviceaccount=default ns=default resource=serviceaccounts resp=201 decision=allow reason=)","priority":"Informational","rule":"K8s Serviceaccount Created","source":"k8s_audit","tags":["k8s"],"time":"2023-05-10T14:09:17.699479000Z", "output_fields": {"evt.time":1683727757699479000,"ka.auth.decision":"allow","ka.auth.reason":"","ka.response.code":"201","ka.target.name":"default","ka.target.namespace":"default","ka.target.resource":"serviceaccounts","ka.user.name":"system:node:ip-192-168-34-135.ec2.internal"}}
{"hostname":"falco-k8saudit-eks-96d7f6f99-vwjgc","output":"14:09:24.046826000: Notice Attach/Exec to pod (user=kubernetes-admin pod=kubecon resource=pods ns=default action=exec command=sh)","priority":"Notice","rule":"Attach/Exec Pod","source":"k8s_audit","tags":["k8s"],"time":"2023-05-10T14:09:24.046826000Z", "output_fields": {"evt.time":1683727764046826000,"ka.target.name":"kubecon","ka.target.namespace":"default","ka.target.resource":"pods","ka.target.subresource":"exec","ka.uri.param[command]":"sh","ka.user.name":"kubernetes-admin"}}

Disclaimer

Our tests noticed some latencies between the presence of the logs in the CloudWatch Logs Stream and their evaluation by Falco. This is more visible with highly requested API servers. The solution is to adapt the size of your nodes where Falco runs, considering a minimal size of 2xlarge as a safe option.

Conclusion

With this first Plugin for a managed K8s solution, we hope to open the door for new contributions from the community for other flavors like GKE and AKS. If you need to create a plugin for another AWS service, take also a look at the libs we created to help the developers.

You can find us in the Falco community. Please feel free to reach out to us for any questions, suggestions, or even for a friendly chat!

If you would like to find out more about Falco:

Get started in Falco.org
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Falco on AWS Cloud

Wed, 30 Nov 2022 00:00:00 +0000

It's Amazon Web Services' largest user conference this week, re:Invent, which is a good time to highlight the ways you can use Falco in the AWS Cloud for runtime security. In this article we'll review what's new, and take a look at installation, plugins, and integrations for AWS.

Support for Amazon Security Lake

We're pleased to announce that Falcosidekick will shortly be available with preview integration for Amazon Security Lake, a new service that optimizes and centralizes security data from cloud, on-premises, and custom sources into a purpose-built data lake.

Falcosidekick is designed to forward Falco events into other services: the new integration exports security events using the Open Cybersecurity Schema Framework (OCSF) format, an open industry standard, and sends them directly to Amazon Security Lake. This makes it easier to normalize and combine Falco events with other security data sources. You can check out the integration in the next version of Falcosidekick, 2.27.0.

Installation and drivers

You can find Falco and Falcosidekick as container images through the Amazon ECR Registry:

Falco image.
Falcosidekick image.

Additionally, the Falco project publishes pre-built driver modules for AWS kernels, whether you are using the kernel module driver or the eBPF probe. These can be fetched using falco-driver-loader.

Review the available drivers:

The prebuilt modules are available for both x86_64 and aarch64 architectures.

Plugins

Falco plugins let you use event sources other than kernel syscalls. Falco has two plugins specific to AWS.

The CloudTrail plugin can read AWS CloudTrail logs and emit events for each CloudTrail log entry. It includes out-of-the-box rules that can be used to identify potential threats in CloudTrail logs, including:
- Console logins that do not use multi-factor authentication.
- Disabling multi-factor authentication for users.
- Disabling encryption for S3 buckets.
Falco has extended its capability to read Kubernetes audit logs through a plugin for CloudWatch, where it can read the EKS audit logs. Read more about configuration and usage.

Falcosidekick integrations

Falcosidekick lets you forward events from Falco into a variety of different services, including many on AWS.

Cloudwatch Logs: emit events into a CloudWatch log stream.
S3: add events in JSON format to an S3 bucket.
Lambda: invoke a Lambda function in response to a Falco event.
SQS: send a message into an SQS queue.
SNS: create a push notification to apps or people.
Kinesis: send Falco events as streaming data.
Amazon Security Lake: add Falco events to a security data lake.

Conclusion

Falco offers a wide variety of support for runtime security on the AWS cloud. As we are an open source project, we welcome contributions and feedback! Read more about running Falco on AWS from this AWS Security blog post, Continuous runtime security monitoring with AWS Security Hub and Falco.

You can find us in the Falco community. Please feel free to reach out to us for any questions, suggestions, or even for a friendly chat!

If you would like to find out more about Falco:

Get started in Falco.org
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter

Falco – Cloud

Blog: Monitoring your EKS clusters audit logs

How it works

Usage

IAM

Default Rules

Installation

Local installation

Kubernetes installation

Disclaimer

Conclusion

Blog: Falco on AWS Cloud

Support for Amazon Security Lake

Installation and drivers

Plugins

Falcosidekick integrations

Conclusion