Falco – Falcosidekick

Blog: Falcosidekick 2.31.0

Tue, 04 Feb 2025 00:00:00 +0000

The year 2025 is well started now. We saw a few days ago the first release of Falco for the year. It's to let fly out a new version of Falcosidekick, the 2.31.0.

New output

This release comes with a new output only, the last pillar of the observability with [OpenTelemetry].(https://opentelemetry.io/) that missing in Falcosidekick.

OTLP Metrics

You can now forward the Falco Events to the OpenTelemetery collector or any received understanding the protocol.

New features

Here's a non exhaustive list of the great features and enhancements which come with this new release:

Better logger

It was a ToDo for a while (even years), but it's now completed. The log system used by Falcosidekick has been replaced, without any breaking change for the users, but opening the door to more enhancements in the future.

More default labels for Loki

The log lines forwarded to Loki contain now by default the source namespace and pod name, if present in the alert. It will allow to filter more easily the events you want to display in your dashboards. Thanks to @afreyermuth98.

Payload format for Loki

Some users asked for the possibility to forward the Falco alerts in their JSON format to Loki. You can now use the setting loki.format for.

NATS/STAN subject

The template for the subject where to push the messages for NATS/STAN was hardcoded, it can now be overridden with nats/stan.subjecttemplate. See the example config file.

Fixes

Fix the missing templated fields as labels in Loki payload (PR#1091)
Fix the creation error of a ClusterPolicyReport (PR#1100)
Fix the missing custom headers for HTTP requests for Loki (PR#1107 thanks to @lsroe)
Fix the wrong key format of custom fields for Prometheus (PR#1110 thanks to @rubensf)

Conclusion

You can find the full changelog here.

The respective Helm charts are already updated and allow you to test by yourself all these great new features. Just issue the helm repo update; helm upgrade --reuse-values -n falco command to do so.

Once again, thanks to all the adopters and contributors who helped and contributed to this project all these years. We would never have reached this success without you.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falco Talon project docs.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Falcosidekick 2.30.0

Wed, 04 Dec 2024 00:00:00 +0000

A few days after a new release of Falco Talon, our response engine, it's time for our favorite proxy forwarder to do the same.

New outputs

A new release means new integrations. Thanks to our contributors for their helps.

Webex

Notify your team on Webex with the integration developed by @k0rventen.

OTLP Metrics

The adoption of Open Telemetry is bigger and bigger in the Cloud Native ecosystem, @ekoops introduced the OTLP Metrics in Falcosidekick.

Datalog Logs

The Falco alerts can be forwarded to Datadog as events for a while in Falcosidekick, you can now use their Logs service thanks to @yohboy.

New features

Here's a non exhaustive list of the great features and enhancements which come with this new release:

x3 throughput

@alekmaus spotted a bottleneck with the http client used to forward the events to the outputs. His fix increases up to 300% the throughput!!!

Better integration with Elasticsearch

@alekmaus worked hard to improve the integration with Elasticsearch. In addition improvments for the clients, new settings have been introduced, like the possibility to specify an ingest pipeline or an api key, to enable batching and compression. See the docs to know them all.

Better consistency for the Prometheus metrics

Falco recently integrated a direct endpoint to expose metrics in the Prometheus format. After a lot of discussions between the maintainers and the community, a convention has been chosen for the names of the metrics. This release adapts the metrics exposed by Falcosidekick to follow this convention and have a consistency accross the different components of the ecosystem.

Breaking changes: The renaming of the metrics might impact the queries for your alerts and dashboards.

Multi hosts for AlertManager

You can now specify a list of servers for the AlertManager output, which is a requirement when it's deployed in HA mode.

Fixes

The contributors fixed several bugs, here's a non exhaustive list of the more important ones:

Fix PolicyReports created in the same namespace than the previous event (PR#978)
Fix the missing customFields/extraFields in the Elasticsearch payload (PR#1033)
Fix the incorrect key name for CloudEvent spec attribute (PR#1051)

Conclusion

You can find the full changelog here.

Once again, thanks to all the adopters and contributors who helped and contributed to this project all these years. We would never have reached this success without you.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falco Talon project docs.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Falcosidekick 2.29.0

Tue, 02 Jul 2024 00:00:00 +0000

Almost 1 year without a release of Falcosidekick, but version 2.29.0 is finally here. Thanks to all contributors for their patience, you made amazing contributions and we're happy to finally have them available for all users.

Like for every releases, a small recap about its adoption. Falcosidekick continues to be adopted, even if the rate is not as high as before, but we're sure it will explode once again with this new fresh version.

Once more, Falcosidekick expands Falco's integrability with a lot of new outputs. That and the introduction of many new features has been possible thanks to the hard work of the community. You can find a comprehensive list of these in the changelog.

New outputs

More and more systems are integrated as outputs in Falcosidekick, more and more often directly by the companies themselves and not their end users. It shows Falco and Falcosidekick are seen as major components in the security fields, and trusted as de facto standards.

Dynatrace

Mario Kahlhofer, aka @blu3r4y, from Dynatrace, integrated the well known observability and security platform he works for. You can even read his blog post about, to discover how to correlate the Falco events with their APM agent events.

Sumologic

Carlo Mencarelli, aka @mencarellic, did the exporter of the Falco events to Sumologic, the SaaS platform for your logs.

OTLP Traces

It started as an internal hackaton at Grafana Labs and became a real integration thanks to JuanJo Ciarlante (@jjo). You can now export the Falco event as traces, to have an automatic correlation between the detected events.

[!WARNING] It works only for the syscall related events.

Quickwit

After a demo of Falco at a CNCF Meetup, the Quickwit team wanted to add their product as a new output for Falcosidekick, and they did it. You can now easily index your Falco events in their search engine thanks to the work of Idriss Neumann (@idrissneumann).

Falco Talon

New born in the Falco ecosystem, trying to complete the last missing piece: the reaction. You can now forward the Falco events to Falco Talon, a tailor made no-code response engine for Falco. The project is still in alpha stage, but moves quickly. Stay tuned.

New features

Aside from new outputs, we introduced very important and useful new features. Let's do a recap of them.

Revamp of the Policy Report output

The Policy Report feature in Kubernetes evolved since its integration in Falcosidekick, it was the time to do some clean up. The report now contains more information, and their displays in the Policy Reporter UI is better.

New outputFieldFormat setting

Some systems perform deduplication of the events, for example the on-call platforms. They use the content of the output to do so, but the current format starting with a timestamp prevents the process to run as expected. A new setting outputFieldFormat is now available allows to "format" the output field of the Falco payload before forwarding it to the outputs.

The default format received from Falco is : <timestamp>: <priority> <output> which corresponds to this:

14:37:27.505989596: Warning Detected ptrace PTRACE_ATTACH attempt (proc_pcmdline=%proc.pcmdline evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty exe_flags=%evt.arg.flags %container.info)

By removing the <timestamp> and <priority>, you get:

Detected ptrace PTRACE_ATTACH attempt (proc_pcmdline=%proc.pcmdline evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty exe_flags=%evt.arg.flags %container.info)

If you use the settings customFields and templatedFields of Falcosidekick to inject new elements in the output_fields, it's also possible to have them in the output with the tokens <custom_fields> and <templated_fields>.

Alternative endpoints for AWS S3

Some projects like Minio are S3-compliant, you can now use them as target for the AWS S3 output by changing the endpoint to use. Thanks to @gysel for this feature.

Split of the docs

The main README of the project became really huge over the years, with all those available outputs. We did a big refactor and you can now find one file per output, with more details about the configuration, the default values and some tips. The docs are here, and any help is welcome to make them even better.

Fixes

The contributors fixed several bugs, here's a non exhaustive list of the more important ones:

Fix missing root CA for the Kafka output (thanks to @claviola)
Fix bug with the extension source in the CloudEvent output
Fix panics in the Prometheus output when hostname field is missing
Fix locks in the Loki output (thanks to @bsod90)
Fix mTLS client verification failures due to missing ClientCAs (thanks to @jgmartinez)
Fix wrong env vars for pagerduty output
Remove hard settings for usernames in Mattermost and Rocketchat
Fix multi lines json in the error lines (thanks to @idrissneumann)
Fix duplicated custom headers in clients
Fix the labels for the AlertManager output (thanks to @Umaaz)

Conclusion

You can find the full changelog here.

Once again, thanks to all the adopters and contributors who helped and contributed to this project all these years. We would never have reached this success without you.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falco Talon project docs.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Falcosidekick 2.28.0

Fri, 28 Jul 2023 00:00:00 +0000

It's summertime, it's hot, and many people are on vacation, but the Falco community is still there. Six months after the release of Falcosidekick's latest upgrade, version 2.28.0 becomes officially available.

The number of pulls of the official Falcosidekick image from Docker Hub has also just reached 15M, which we consider mind-blowing. It took it 3 years to reach the first 5M pulls and now it's needed less than six months to do it again. Awesome!

New outputs

With every new release, the number of integrations of Falcosidekick increases. We have reached the number of 58 available outputs already, and we hope the threshold of 60 will be left behind very soon.

Redis

Redis is a well-known in-memory database with many years of adoption on its path. It is now possible for Falcosidekick to use it as an output destination, thanks to the contributions of pandyamarut.

Telegram, the instant messaging platform, is becoming more and more used by companies for notifications, and thanks to zufardhiyaulhaq, it can receive Falco alerts too.

N8N

Do you want to extend the possibilities or just avoid developing a script to react to Falco events? Here comes n8n.

Grafana OnCall

At the last KubeCon, we met with some of the Grafana maintainers. We discussed the integration of Falco using Falcosidekick within the OnCall project. It's done now.

OpenObserve

OpenObserve is a young but promising full stack observability platform.

New features

Aside from new outputs, we introduced very important and useful new features. Let's do a recap of them.

Use different methods for the Webhook output

Since its implementation, the Webhook output has only used the HTTP method POST. Now, you can choose between the POST and PUT methods to send your data, extending the catalog of possible REST APIs to use it with.

webhook:
 method: "POST" # HTTP method: POST or PUT (default: POST)

Replace the brackets in the payload

Some Falco fields refer to lists and reflect that their keys contain brackets, like proc.args[0], proc.args[1], etc. Unfortunately, some outputs may refuse payloads with brackets in keys. For this reason, we introduced the possibility of replacing them with any other chosen character:

bracketreplacer: "_" # if not empty, the brackets in keys of Output Fields are replaced

Set custom headers for Loki, Elasticsearch and Grafana outputs

If you want to protect your private instances of Loki, Grafana, or Elasticsearch you may need to specify custom headers. This new feature allows you to do so.

elasticsearch:
 customHeaders: # Custom headers to add in POST. Useful for Authentication
 key: value

Match the priority with the severity for the AlertManager output

AlertManager is a pretty common software at companies also using Prometheus. Until now, the mapping between the Priority of Falco events and the Severity of AlertManager was already predefined. You can now define it depending on your needs thanks to Lowaiz.

alertmanager:
 customseveritymap: "" # comma separated list of tuple composed of a ':' separated Falco priority and Alertmanager severity that is used to override the severity label associated to the priority level of falco event. Example: debug:value_1,critical:value2. Default mapping: emergency:critical,alert:critical,critical:critical,error:warning,warning:warning,notice:information,informational:information,debug:information. (default: "")

Set thresholds for the dropped events for the AlertManager output

Another contribution from Lowaiz: You can now configure a set of thresholds to start dropping the events.

alertmanager:
 # dropeventdefaultpriority: "" # default priority of dropped events, values are emergency|alert|critical|error|warning|notice|informational|debug (default: "critical")
 dropeventthresholds: # comma separated list of priority re-evaluation thresholds of dropped events composed of a ':' separated integer threshold and string priority. Example: `10000:critical, 100:warning, 1:informational` (default: `"10000:critical, 1000:critical, 100:critical, 10:warning, 1:warning"`)

Better integration with TimescaleDB

We thank hileef for improving the integration with TimescaleDB.

User roleARN and externalID for the AWS outputs

Under some situations, you may want Falcosidekick to assume a role, possibly from another account. You can do it with the new pair of settings rolearn and externalid:

aws:
 rolearn: "" # aws role to assume (optional if you use EC2 Instance Profile)
 externalid: "" # external id for the role to assume (optional if you use EC2 Instance Profile)

Set the region for the PagerDuty output

Falcosidekick allows you to select between one of the two different regions' PagerDuty offers now.

pagerduty:
 region: "us" # Pagerduty Region, can be 'us' or 'eu' (default: us)

Allow TLS for the SMTP output

It is now possible to communicate with an SMTP server using TLS.

smtp:
 tls: false # Use TLS connection (true/false). Default: true

Set attributes to GCP PubSUb messages

GCP PubSub accepts attributes in its messages. You can specify yours, thanks to annadorottya.

gcp:
 customAttributes: # Custom attributes to add to the Pub/Sub messages
 key: value

More options for TLS and mTLS

These are the most relevant changes of this release. To improve security, Falcosidekick can now listen using HTTPS with TLS. You can also be more specific with the keys and certificates for the mTLS for the outputs.

mutualtlsfilespath: "/etc/certs" # folder which will used to store client.crt, client.key and ca.crt files for mutual tls for outputs, will be deprecated in the future (default: "/etc/certs")
 # certfile: "/etc/certs/client/client.crt" # client certification file
 keyfile: "/etc/certs/client/client.key" # client key
 cacertfile: "/etc/certs/client/ca.crt" # for server certification
tlsserver:
 # certfile: "/etc/certs/server/server.crt" # server certification file
 keyfile: "/etc/certs/server/server.key" # server key

The mutualtlsfilespath setting is kept for now for backward compatibility but it will be remove in future

In some edge cases, you may need some endpoints to listen in HTTP only. You can specifically define them together with the associated port:

tlsserver:
 notlsport: 2810 # port to serve http server serving selected endpoints (default: 2810)
 notlspaths: # if not empty, a separate http server will be deployed for the specified endpoints
 - "/metrics"
 - "/healthz"

Thanks to annadorottya for her impressive work on this functionality.

Autocreate the topic for the Kafka output

When Falcosidekick doesn't detect the topic, it can create it automatically. This feature is not enabled by default.

kafka:
 topiccreation: false # auto create the topic if it doesn't exist (default: false)

Support multiple bootstrap servers and TLS for the Kafka output

To get better resiliency, you can now specify several bootstrap servers and even communicate with them with TLS, thanks to ibice.

kafka:
 tls: false # Use TLS for the connections (default: false)

Fixes

We're not going to go into detail about all the corrections made in this version - you can find the full list here. Thanks to everyone who reported issues and to those who have corrected them.

The most important have been:

Fix breaking brackets in AWS SNS messages.
Fix setting name for the table of TimescaleDB output (thanks to alika).
Fix the cardinality issue with Prometheus labels.
Fix panic when asserting output fields that are nil.
Fix URL generation for Spyderbat output (thanks to bc-sb).
Fix nil values in Spyderbat output (thanks to spider-guy).
Fix duplicated headers in SMTP output (thanks to apsega).

Conclusion

The respective Helm charts are also updated and allow you to test for yourself all these great new features. Just issue the helm repo update; helm upgrade --reuse-values -n falco command to do so.

Falcosidekick is now mentioned in the official Falco docs. It's a shy beginning, but more details will come shortly.

Once again, thanks to all the adopters and contributors who helped and contributed to this project all these years. We would never have reached this success without you.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falcosidekick UI project on GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Falcosidekick 2.27.0 and Falcosidekick-UI 2.1.0

Tue, 10 Jan 2023 00:00:00 +0000

So many good things happened for Falcosidekick and Falcosidekick UI this year. It's still incredible these projects became so beloved and useful for the community. To all contributors, promotors and users, a big big thank you.

The new year is there, it's time to release new versions and reach 10 million Docker pulls for Falcosidekick.

Falcosidekick v2.27.0

What a huge release! Never has a previous release gotten so many new features and outputs. You can read the full changelog here.

New outputs

This release brings a lot of new outputs thanks to our amazing contributors. Here you have a list of them.

Yandex Data Stream

Yandex is a Russian cloud provider that provides various services such as Data Streams. With this new output, we can connect Falco to one more cloud providers. Thank you, preved911.

MQTT and Node-Red

IoT is a whole new world for Falco. With these 2 new outputs, Falco can make its first steps in this ecosystem and we are sure more will come in 2023. Stay tuned.

Zincsearch

Do you want a full-text indexer lighter than Elasticsearch? Take a look at Zincsearch.

Gotify

By using Gotify and the new dedicated output, you can now push Falco events to your Android phone.

Spyderbat

Are you a user of Spyderbat and want to extend its sources of events? Now you can thank spyder-kyle.

Tekton

Do you remember the blog post how to create a Response Engine for Falco with Tekton? The proposed solution used the generic Webhook output. From now on, Tekton can use a dedicated one.

TimescaleDB

TimescaleDB is an OSS database made for time-series data, thanks to jagretti Falcosidekick can insert into it the Falco events.

AWS Security Lake

At re:Invent 2023, AWS announced a new data lake made for security data: AWS Security Lake. We worked with AWS teams to have Falco as a source partner from day one, making it the first OSS project that can be used with that service and strengthening once more the integration with the AWS ecosystem.

New features

The list of new outputs is already quite long, but the list of enhancements is even longer. The full list is here, but let's have a look at the major changes.

SASL auth mechanisms for SMTP and Kafka outputs

Improving security is our duty to all, and one key element is the authentication method. Thanks to Lowaiz, both SMTP and Kafka outputs can now use the benefits of SASL Auth mechanisms.

Environment variables for custom labels and templated labels

The ability to inject custom fields in the payloads is an important feature of Falcosidekick. The only drawback was these fields were previously static. That limitation is over. Now, you can use environment variables in your custom fields. A new kind of custom fields has become available: `templated fields.' They allow the reuse of the present fields to generate new ones following with Go template:

templatedfields: # templated fields are added to falco events and metrics, it uses Go template + output_fields values
 Dkey: '{{ or (index . "k8s.ns.labels.foo") "bar" }}'

Hostname field

Since Falco 0.33, a new field is present in Falco events: hostname. Falcosidekick and all its current outputs are up to date and ready for it. Once again, thanks to Lowaiz.

Loki format and Grafana Cloud

The Loki format has been upgraded and credentials can be set. It allows you to use Grafana Cloud as a target.

K8S Policy Reports are binded to the namespaces

Policy Reports in K8S are still prototypes but Falcosidekick is already able to create them. Some improvements have been made to determine the target resource and the report is now created in the same namespace as the source pod.

More headers in SMTP payload

To avoid being flagged as spam by some anti-spam systems, some headers like From, To and Date have been added to the emails created by Falcosidekick.

CEF format Syslog

For the Syslog output, you can choose between json and CEF as formats. It makes easier the integration with some services like Microsoft Sentinel or Splunk.

Fixes

Even if we do our best to avoid them, the community has lately faced some bugs that we have fixed in this release.

The most important one was a race condition when headers were added to the POST requests. Adopters with high rates of requests were occasionally facing authentication failures or missing headers. bc-sb solved this with a temporary solution, but we'll improve it in the future (Falcosidekick v3? Who knows...).

Falcosidekick UI v2.1.0

The new features for Falcosidekick UI, although lower in number, are still big improvements. The full changelog is here.

Env vars for settings

All settings to configure Falcosidekick UI can be passed as either CLI arguments or as env vars. Run falcosidekick-ui --help for more details.

New logs

The logs were too verbose for production contexts. Now it's configurable via a log-level option:

-l string
 Log level: "debug", "info", "warning", "error" (default "info", environment "FALCOSIDEKICK_UI_LOGLEVEL")

Auto refresh

Long-term adopters may remember the dashboard in Falcosidekick UI v1 was auto-refreshed. This feature is back, for all widgets, independently of the page.

Authentication

This is a major new feature. The interface is now protected by the Basic Auth method. More methods will be added in the future:

Set the FALCOSIDEKICK_UI_USER env var to define the credentials.

Info page

The info page has been rewritten for a nicer look & feel.

Hostname

As for Falcosidekick, Falcosidekick UI supports the display of the new hostname field.

TTL for keys

Falcosidekick UI can store a huge amount of events, leading to filling the disk of the Redis database. A TTL for the entries can be set to avoid this situation.

-t int
 TTL for keys (default "0", environment "FALCOSIDEKICK_UI_TTL")

Conclusion

The respective Helm charts are already updated to allow you to test on your own all these great new features. Run a helm upgrade --reuse-values -n falco to do so.

Once again, thanks to all adopters and contributors who helped and contributed for years to create pieces of software useful to everybody. We hope 2023 will be amazing for Falco and its ecosystem.

As usual, if you have any feedback or need help, you can find us at any of the following locations.

Get started in Falco.org
Check out the Falcosidekick project on GitHub.
Check out the Falcosidekick UI project on GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Falcosidekick 2.25.0 and Falcosidekick 2.0.0

Wed, 01 Jun 2022 00:00:00 +0000

A few days ago was the KubeCon EU in Valencia, Spain. The moment to meet contributors who made what Falcosidekick is now was a really enjoyable time and I hope we'll do it again in the future. One week before, two new major versions of Falcosidekick and Falcosidekick-Ui were released. Let's see what's new.

Falcosidekick

Almost 10 months without a new release for Falcosidekick, the version 2.25.0, and what a huge release is. For curious people, the full changelog can be found here.

New outputs

Each new release brings more outputs, thanks to the community. Here's the list of new ones:

Policy Report

With some CRD, you can now create reports in your Kubernetes clusters, the feature is often used for Security or Compliance, but anything is technically possible. For more details about how to use this output, read the documentation from @anushkamittal20 who implemented it for her project for Linux Foundation Mentorship Program.

apiVersion: wgpolicyk8s.io/v1alpha2
kind: ClusterPolicyReport
metadata:
 creationTimestamp: "2022-05-23T13:57:40Z"
 generation: 110
 name: falco-cluster-policy-report-4c9eac68
 resourceVersion: "71090"
 uid: ed8f0659-74d5-488c-90f8-d7b0622738cf
results:
- category: SI - System and Information Integrity
 message: Cluster Role Binding to cluster-admin role (user=%ka.user.name subject=%ka.req.binding.subjects)
 policy: Attach to cluster-admin Role
 properties:
 ka.req.binding: '%ka.req.binding'
 ka.user.name: '%ka.user.name'
 result: fail
 severity: high
 source: Falco
 timestamp:
 nanos: 98821031
 seconds: 40
- category: SI - System and Information Integrity
 message: Created Role/ClusterRole with write privileges (user=%ka.user.name role=%ka.target.name
 rules=%ka.req.role.rules)
 policy: ClusterRole With Write Privileges Created
 properties:
 ka.req.role: '%ka.req.role'
 ka.target.name: '%ka.target.name'
 ka.user.name: '%ka.user.name'
 result: fail
 severity: high
 source: Falco
 timestamp:
 nanos: 103148849
 seconds: 42

The reports can also be displayed with Policy Reporter UI, created by @fjogeleit another member of the Falco community.

Syslog

Years after its creation, Syslog remains a solid solution for managing the log files, especially if you're running Falcosidekick or else directly at the host level. With this new version, a Syslog server can be directly used as the target for the events, allowing you to send them in a secure place. Thanks to @bdluca.

AWS Kinesis

Do you want to ingest thousands of events from Falco and be able to run data analysis on them? You can do so smoothly with the new AWS Kinesis output, bringing a new integration of Falco with AWS Ecosystem. We would be delighted to know any use case with analytics the community could create now. Thanks to @gauravgahlot.

Zoho Cliq

Your DevOps/SRE/SecOps team uses Zoho Cliq for their communication? Allow them to receive nice notifications with this new output for Falcosidekick. Thanks to @averni.

Enhancements

Getting new features is nice, but we can also improve the existing ones, here's a list of major changes of this 2.25.0 release.

Compiled ans signed binaries

Until then, if you wanted binaries of Falcosidekick, you had to build them by yourself or use the provided Docker image. Now, each release will contain the compiled binaries for amd64 and arm64. The security is not forgotten, all artifacts are signed with Cosign.

Tags and Source

In January, Falco 0.31.0 brought its new Plugin system, the source field of events becoming more important. This new release of Falcosidekick updates all the outputs to deal with source and tags events. Your Response Engines can now be much clever than even.

IRSA

IRSA, aka Iam Role for Service Accounts, is the method provided by AWS for linking a Kubernetes Service Account with an IAM Role, allowing the Pod to easily use a Service. Falcosidekick is now able to use this mechanism for its outputs for AWS Services, no need to add Access and Secret Keys in your values.yaml. The UX is much better. Thanks to @VariableExp0rt.

Falcosidekick UI

I created the first version v0 of Falcosidekick-UI to have something more graphical for my talks, with the help of @fjogeleit we created a nice v1 that has been finally used more and more by people, becoming a famous product in the community. It was time to have a better version with some requested features:

a database (Redis) for a long term storage of events
an API for counting or searching the events
filters are kept as query strings, allowing to share links

All details to use this new version, v2.0.0, are described in the README.

Here's some screenshots:

Deployments

The Helm charts are already up to date, you can upgrade your deployments with:

helm upgrade falcosidekick falcosecurity/falcosidekick --set webui.enabled=true -n falco

helm upgrade falco falcosecurity/falco \
 --set falcosidekick.enabled=true \
 --set falcosdekick.webui.enabled=true -n falco

Enjoy

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 9: Falcosidekick + Fission

Wed, 01 Sep 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7 : Falcosidekick + Cloud Functions

Kubernetes Response Engine, Part 8: Falcosidekick + Flux v2

The earlier posts in this series, show how to use Kubeless, Argo, Knative, and others to trigger a resource after getting input from Falcosidekick. Recently, Falcosidekick received a new output type support for Fission.

In this blog post, we will cover using Falcosidekick and Fission to detect and delete a compromised pod in a Kubernetes cluster. We will briefly talk about Fission in this blog, however, you can check the complete documentation here.

Prerequisites

We need tools with the following minimum versions to achieve this demo:

Minikube v1.19.0
Helm v3.5.4
kubectl v1.21.0
fission-cli v1.13.1

Provision local Kubernetes Cluster

There are various ways to provision a local Kubernetes cluster such as, KinD, k3s, k0s, Minikube, etc. We are going to use Minikube in this walkthrough.

Let's get provisioned a local Kubernetes cluster:

$ minikube start --cpus 3 --memory 8192 --vm-driver virtualbox
😄 minikube v1.19.0 on Darwin 10.15.7
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating virtualbox VM (CPUs=3, Memory=8192MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.4 ...
 ▪ Generating certificates and keys ...
 ▪ Booting up control plane ...
 ▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
 ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Install Fission

Fission is a fast, open source serverless framework for Kubernetes with a focus on developer productivity and high performance. Fission operates on just the code: Docker and Kubernetes are abstracted away under normal operation, though you can use both to extend Fission if you want to.

Follow the official documentation for deploying Fission to Kubernetes.

Here we will be using Helm to install Fission:

$ export FISSION_NAMESPACE="fission"
$ kubectl create namespace $FISSION_NAMESPACE
$ kubectl create -k "github.com/fission/fission/crds/v1?ref=1.13.1"
$ helm repo add fission-charts https://fission.github.io/fission-charts/
$ helm repo update
$ helm install --version 1.13.1 --namespace $FISSION_NAMESPACE fission fission-charts/fission-all
NAME: fission
LAST DEPLOYED: Wed Jul 21 18:03:44 2021
NAMESPACE: fission
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Install the client CLI.

Mac:
 $ curl -Lo fission https://github.com/fission/fission/releases/download/1.13.1/fission-1.13.1-darwin-amd64 && chmod +x fission && sudo mv fission /usr/local/bin/

Linux:
 $ curl -Lo fission https://github.com/fission/fission/releases/download/1.13.1/fission-1.13.1-linux-amd64 && chmod +x fission && sudo mv fission /usr/local/bin/

Windows:
 For Windows, you can use the linux binary on WSL. Or you can download this windows executable: https://github.com/fission/fission/releases/download/1.13.1/fission-1.13.1-windows-amd64.exe

2. You're ready to use Fission!

 # Create an environment
 $ fission env create --name nodejs --image fission/node-env

 # Get a hello world
 $ curl https://raw.githubusercontent.com/fission/examples/master/nodejs/hello.js > hello.js

 # Register this function with Fission
 $ fission function create --name hello --env nodejs --code hello.js

 # Run this function
 $ fission function test --name hello
 Hello, world!

Check if everything is working before moving onto the next step:

$ kubectl get pods --namespace fission
NAME READY STATUS RESTARTS AGE
buildermgr-5698c89fff-rk9z6 1/1 Running 0 7m20s
controller-5dcb44bcd6-vq9hb 1/1 Running 0 7m20s
executor-6b6d6469d6-2xrlk 1/1 Running 0 7m20s
fission-kube-state-metrics-5fc9bd6684-7ffwp 1/1 Running 0 7m20s
fission-prometheus-alertmanager-65f5574885-tlrz6 2/2 Running 0 7m20s
fission-prometheus-node-exporter-jd9w6 1/1 Running 0 7m20s
fission-prometheus-node-exporter-jpzn8 1/1 Running 0 7m20s
fission-prometheus-node-exporter-rb25l 1/1 Running 0 7m20s
fission-prometheus-pushgateway-54c87b5796-28x2h 1/1 Running 0 7m20s
fission-prometheus-server-9d64c74b4-ld97h 2/2 Running 0 7m20s
influxdb-59649c8c6-5vx54 1/1 Running 0 7m20s
kubewatcher-6996fccc6b-5vbvx 1/1 Running 0 7m20s
logger-6kdw4 2/2 Running 0 7m20s
logger-nmw9t 2/2 Running 0 7m20s
logger-zkrq9 2/2 Running 0 7m20s
mqtrigger-keda-7584989c48-n5w6g 1/1 Running 0 7m20s
mqtrigger-nats-streaming-664c55c979-t9gp5 1/1 Running 0 7m19s
nats-streaming-6c6d7c6fbf-ft468 1/1 Running 0 7m20s
router-5c5c6cbb87-989pc 1/1 Running 0 7m20s
storagesvc-57ccf58976-qcr4d 1/1 Running 0 7m19s
timer-794b89579b-6kxwx 1/1 Running 0 7m20s

Install Falco + Falcosidekick

Firstly, we'll create the namespace that will host both Falco and Falcosidekick:

$ kubectl create namespace falco

We add the helm repo:

$ helm repo add falcosecurity https://falcosecurity.github.io/charts
"falcosecurity" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
...Successfully got an update from the "openfaas" chart repository
Update Complete. ⎈Happy Helming!⎈

In a real project, you should get the whole chart with helm pull falcosecurity/falco --untar and then configure the values.yaml. For this tutorial, will try to keep thing as easy as possible and set configs directly by passing arguments to helm install command line:

$ helm upgrade --install falco falcosecurity/falco --namespace falco \
--set falcosidekick.enabled=true \
--set falcosidekick.config.fission.function="falco-pod-delete"

You should get this output:

Release "falco" does not exist. Installing it now.
NAME: falco
LAST DEPLOYED: Tue Apr 13 10:49:49 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.


No further action should be required.

And you can see your new Falco and Falcosidekickpods:

$ kubectl get pods --namespace falco
NAME READY STATUS RESTARTS AGE
falco-falcosidekick-f77c58899-gd467 1/1 Running 0 51s
falco-falcosidekick-f77c58899-hfsjx 1/1 Running 0 51s
falco-hg2wm 1/1 Running 0 51s

The argument falcosidekick.enabled=true sets the following settings in Falco for you:

--set falco.jsonOutput=true \
--set falco.httpOutput.enabled=true \
--set falco.httpOutput.url=http://falco-falcosidekick:2801

The arguments --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falco-falcosidekick:2801 are there to configure the format of events and the URL where Falco will send them. As Falco and Falcosidekick will be in the same namespace, it can directly use the name of the service (falco-falcosidekick) above Falcosidekick pods.

We check the logs:

$ kubectl logs deployment/falco-falcosidekick -n falco
Found 2 pods, using pod/falco-falcosidekick-f77c58899-gd467
2021/07/21 12:52:02 [INFO] : Enabled Outputs : [Fission]
2021/07/21 12:52:02 [INFO] : Falco Sidekick is up and listening on :2801

Fission is displayed as enabled output, everything is good 👍.

Install our Fission Function

Our really basic function will receive events from Falco, thanks to Falcosidekick, check if the triggered rule is Terminal Shell in container, extract the namespace and pod name from the fields of events and delete the according pod:

Basically, the process is:

 +----------+ +---------------+ +----------+
 | Falco +-----------------> Falcosidekick +--------------------> Fission |
 +----^-----+ sends event +---------------+ triggers +-----+----+
 | |
detects a shell | |
 | |
 +----+-------+ deletes |
 | Pwned Pod <----------------------------------------------------------+
 +------------+

Let's get the function and other artifacts:

$ git clone https://github.com/fission/examples.git && cd examples/sample/falco

The function we are going to deploy basically receives events for an infected pod from the Falcosidekick and deletes it immediately. Before deploying the function we need some permissions to delete Pod. We create a ServiceAccount with rights to delete a Pod in any namespace, and we'll associate it to our function:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: falco-pod-delete
 namespace: fission-function
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role
rules:
- apiGroups: [""]
 resources: ["configmaps", "secrets"]
 verbs: ["get"]
- apiGroups: [""]
 resources: ["pods"]
 verbs: ["get", "list", "delete"]
- apiGroups: [""]
 resources: ["events"]
 verbs: ["*"]
- apiGroups: ["fission.io"]
 resources: ["packages"]
 verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role-binding
roleRef:
 kind: ClusterRole
 name: falco-pod-delete-cluster-role
 apiGroup: rbac.authorization.k8s.io
subjects:
 - kind: ServiceAccount
 name: falco-pod-delete
 namespace: fission-function

Let's create the service account with:

$ kubectl apply -f sa-falco-pod-delete.yaml

The falco-pod-delete/handler.go contains our function:

package main

import (
 "context"
 "encoding/json"
 "io/ioutil"
 "log"
 "net/http"
 "time"

 metaV1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/kubernetes"
 "k8s.io/client-go/rest"
)

var kubeClient *kubernetes.Clientset

func init() {
 // creates the in-cluster config
 config, err := rest.InClusterConfig()
 if err != nil {
 panic(err.Error())
 }
 // creates the clientset
 kubeClient, err = kubernetes.NewForConfig(config)
 if err != nil {
 panic(err.Error())
 }
}

type Alert struct {
 Output string `json:"output"`
 Priority string `json:"priority"`
 Rule string `json:"rule"`
 Time time.Time `json:"time"`
 OutputFields struct {
 ContainerID string `json:"container.id"`
 ContainerImageRepository interface{} `json:"container.image.repository"`
 ContainerImageTag interface{} `json:"container.image.tag"`
 EvtTime int64 `json:"evt.time"`
 FdName string `json:"fd.name"`
 K8SNsName string `json:"k8s.ns.name"`
 K8SPodName string `json:"k8s.pod.name"`
 ProcCmdline string `json:"proc.cmdline"`
 } `json:"output_fields"`
}

var criticalNamespaces = []string{"kube-system", "kube-public", "kube-node-lease", "falco", "fission", "fission-function"}

func Handler(w http.ResponseWriter, r *http.Request) {
 var alert Alert

 if r.Body != nil {
 defer r.Body.Close()

 body, _ := ioutil.ReadAll(r.Body)
 json.Unmarshal(body, &alert)

 log.Printf("\n[ INFO ] Alert : %v\n", alert)

 podName := alert.OutputFields.K8SPodName
 namespace := alert.OutputFields.K8SNsName

 var critical bool
 for _, ns := range criticalNamespaces {
 if ns == namespace {
 critical = true
 break
 }
 }

 if !critical {
 _, err := kubeClient.CoreV1().Pods(namespace).Get(context.Background(), podName, metaV1.GetOptions{})
 if err != nil {
 log.Printf("\n[ WARN ] Failed to get pod '%s' in '%s' namespace", podName, namespace)
 return
 }

 log.Printf("Deleting pod %s from namespace %s", podName, namespace)
 err = kubeClient.CoreV1().Pods(namespace).Delete(context.Background(), podName, metaV1.DeleteOptions{})
 if err != nil {
 log.Printf("\n[ ERROR ] Failed to delete pod: %v", err)
 w.WriteHeader(http.StatusInternalServerError)
 w.Write([]byte(err.Error()))
 }
 }
 }

 w.WriteHeader(http.StatusOK)
 w.Write([]byte("OK"))
}

A fission function requires an environment/runtime to run. The yaml definitions for the runtime, the function and the router are available under the specs directory.

Now, we are ready to deploy our falco-pod-delete function using the specs:

$ fission spec apply
DeployUID: edc80e3e-7d1e-448c-aba8-c8cd75b3a1eb
Resources:
 * 1 Functions
 * 1 Environments
 * 1 Packages
 * 1 Http Triggers
 * 0 MessageQueue Triggers
 * 0 Time Triggers
 * 0 Kube Watchers
 * 1 ArchiveUploadSpec
Validation Successful
Spec doesn't belong to Git Tree.
1 function created: falco-pod-delete
1 HTTPTrigger created: falco-pod-delete
1 environment created: go
1 package created: falco-pod-delete-d18f6a0b-e5a1-4275-9471-38d684ac4dfe

Check if the package was built successfully for our fission function before moving to the next step:

$ fission pkg list
NAME BUILD_STATUS ENV LASTUPDATEDAT
falco-pod-delete-d18f6a0b-e5a1-4275-9471-38d684ac4dfe succeeded go 21 Jul 21 08:26 IST

Test our function

We start by creating a dumb pod:

$ kubectl run alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"
pod/alpine created

$ kubectl get pods --namespace default
AME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 11s

Let's run a shell command inside and see what happens:

$ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"

19:27:21 up 50 min, load average: 0.11, 0.12, 0.11

As expected we got the result of our command, but, if we get the status of the pod we retrieve:

$ kubectl get pods --namespace default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 103s

💥 It has been terminated 💥

We can now check the logs of components.

For Falco:

$ kubectl logs daemonset/falco --namespace falco
..
{"output":"10:36:32.750441241: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=cbd3133ccac6 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=cbd3133ccac6 image=alpine) k8s.ns=default k8s.pod=alpine container=cbd3133ccac6","priority":"Notice","rule":"Terminal shell in container","time":"2021-07-21T10:36:32.750441241Z", "output_fields": {"container.id":"cbd3133ccac6","container.image.repository":"alpine","evt.time":1626863792750441241,"k8s.ns.name":"default","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}
{"output":"10:37:09.101509967: Notice Unexpected connection to K8s API Server from container (command=fetcher -secret-dir /secrets -cfgmap-dir /configs -jaeger-collector-endpoint /userfunc k8s.ns=fission-function k8s.pod=poolmgr-go-default-516098-5bdbf8c8f5-g8gvc container=281c99ea33c2 image=fission/fetcher:1.13.1 connection=192.168.43.223:39526->10.100.0.1:443) k8s.ns=fission-function k8s.pod=poolmgr-go-default-516098-5bdbf8c8f5-g8gvc container=281c99ea33c2","priority":"Notice","rule":"Contact K8S API Server From Container","time":"2021-07-21T10:37:09.101509967Z", "output_fields": {"container.id":"281c99ea33c2","container.image.repository":"fission/fetcher","container.image.tag":"1.13.1","evt.time":1626863829101509967,"fd.name":"192.168.43.223:39526->10.100.0.1:443","k8s.ns.name":"fission-function","k8s.pod.name":"poolmgr-go-default-516098-5bdbf8c8f5-g8gvc","proc.cmdline":"fetcher -secret-dir /secrets -cfgmap-dir /configs -jaeger-collector-endpoint /userfunc"}}
..

For Falcosidekick:

$ kubectl logs deployment/falco-falcosidekick --namespace falco
..
2021/07/21 10:37:13 [INFO] : Fission - Post OK (200)
2021/07/21 10:36:32 [INFO] : Fission - Function Response : OK
2021/07/21 10:36:32 [INFO] : Fission - Call Function "falco-pod-delete" OK
..

For falco-delete-pod function:

$ fission function logs -f --name falco-pod-delete
..
[2021-07-21 10:47:27.206605532 +0000 UTC] 2021/07/21 10:47:27 Deleting pod alpine from namespace default
..

Conclusion

With this really simple example, we got another way to create a Response Engine with amazing pieces of software from the Open Source world. We only scratched the surface of possibilities, so don't hesitate to share with us your comments, ideas and successes.

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 8: Falcosidekick + Flux v2

Tue, 31 Aug 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7 : Falcosidekick + Cloud Functions

Today, we will set up another KRE (Kubernetes Response Engine) based on Flux (version 2). If you don't know about Flux (version 2), let me explain it in a few words. Flux (version 2) is a tool for keeping Kubernetes clusters in sync with configuration sources (such as Git repositories) and automating updates to the configuration when new code is available to deploy.

To learn more about Flux (version 2), see Flux Documentation.

Flux (version 2) might look like a GitOps tool. It is, in reality, another GitOps tool in that it watches Github repositories for configuration changes and keeps the current state and the desired state always in sync. It does that on top of Kubernetes by using a bunch of CRs (Custom Resources). However, unlike Knative Eventing, Tekton Triggers, and Argo Events, Flux (version 2) does not support an eventing system to forward events from one point to another.

To set up KRE with Flux (version 2), we will create a small project that listens to events and updates the GitHub repository, which Flux (version 2) monitors to alter the desired state. For example, get the event of a pwned pod, then change its replicas to zero within the deployment YAML file.

The reference architecture given below illustrates the content of this blog.

Table of Contents

Prerequisites

minikube v1.21.0
helm v3.6.2+gee407bd
kubectl v1.21.1
ko v0.8.3
flux v0.16.0
gcloud v347.0.0

Tutorial

Provision local Kubernetes Cluster

$ minikube config view
- cpus: 3
- driver: virtualbox
- memory: 8192

minikube start

Install Flux V2 - GitOps Toolkit

I highly recommended that you check out getting started page of Flux (version 2). It gives you detailed installation instructions for Flux (version 2).

For Flux (version 2) to create a GitHub repository for its resources, we must define the token and username information. Then, Flux (version 2) installed in a GitOps way, and Flux (version 2) will push its manifest to the repository.

export GITHUB_USER=<username>
export GITHUB_TOKEN=<token>

Once the necessary environment variables are defined, we can install Flux (version 2). The following command will create and push its manifests to the repository, then install Flux components.

flux bootstrap github \
 --owner=$GITHUB_USER \
 --repository=fleet-infra \
 --branch=main \
 --path=./clusters/my-cluster \
 --personal

After the installation is complete, the next step is creating the alpine GitRepository and Kustomization CRD's (Custom Resource Definitions). For more information, see components.

Apply the CRD files as follows:

$ kubectl apply -f https://raw.githubusercontent.com/developer-guy/kubernetes-response-engine-based-on-flux-v2-gitops-toolkit/master/alpine-gitsource.yaml
gitrepository.source.toolkit.fluxcd.io/alpine created

$ kubectl apply -f https://raw.githubusercontent.com/developer-guy/kubernetes-response-engine-based-on-flux-v2-gitops-toolkit/master/alpine-kustomization.yaml
kustomization.kustomize.toolkit.fluxcd.io/alpine created

Alternatively, we can use the Flux CLI.

To create GitRepository:

flux create source git alpine \
 --url https://github.com/developer-guy/desired-state-repository \
 --branch master \
 --interval 30s

To create Kustomization:

flux create kustomization alpine \
 --source alpine \
 --path "./" \
 --prune true \
 --validation client \
 --interval 5m

Install falco-event-listener

As the name suggests, this program will listen to an event in the form of CloudEvents. This CloudEvents forwarded from Falcosidekick, a simple daemon for enhancing available outputs for Falco. After the successful receipt of the event, falco-event-listener will update the YAML definition to scale its replicas to zero based on the pieces of information given in the event.

In most basic form, the architecture of the demo is:

Falco w/HTTP --> Falcosidekick w/CloudEvent --> falco-event-listener w/HTTP --> GitHub

To learn more about CloudEvents, see cloudevents.io.

First, let us clone the falco-event-listener repository.

git clone https://github.com/developer-guy/falco-event-listener
cd falco-event-listener

Before installing this project, we have to do a couple of things.

We have to create a GitHub PAT (Personal Access Token) to be able to update the desired state configurations which are in the GitHub repository after we detect a malicious behavior related to our pod has detected.

As you can see in the above arguments, we should pass GITHUB_TOKEN as an argument to our CLI application. The best option is storing GITHUB_TOKEN in a Kubernetes Secret and using Secret as environment variables.

kubectl create namespace falco
kubectl create secret generic github-secret \
 --from-literal GITHUB_TOKEN=$GITHUB_TOKEN \
 --namespace falco

You might notice that we are using some URLs (--notify-url) within the flags of the project. Let me explain that why we need it a bit. As you might know, we are trying to set up some remediation engine here, so we have to react to those events thrown by the Falcosidekick as soon as possible. We defined an interval while creating a GitRepository, which means that Flux (version 2) will wait at least that long to sync configurations, so we have to notify Flux (version 2) controllers about changes once we edited the desired state by a process whose name is falco-event-listener. To notify the Flux (version 2) controllers about changes in Git or Helm repositories, we can set up webhooks and trigger a cluster reconciliation every time a source changes. For more detail, please see.

There are different kinds of webhook receivers in Flux (version 2), but we'll use the generic one in this guide.

const (
 GenericReceiver string = "generic"
 GenericHMACReceiver string = "generic-hmac"
 GitHubReceiver string = "github"
 GitLabReceiver string = "gitlab"
 BitbucketReceiver string = "bitbucket"
 HarborReceiver string = "harbor"
 DockerHubReceiver string = "dockerhub"
 QuayReceiver string = "quay"
 GCRReceiver string = "gcr"
 NexusReceiver string = "nexus"
 ACRReceiver string = "acr"
)

Let us to create a Receiver, to do that we have to a couple of things again:

$ TOKEN=$(head -c 12 /dev/urandom | sha256sum | cut -d ' ' -f1)
$ echo $TOKEN
0babd54d2b64d6d6fcd10a663cb6195773e968ba6642ca8c1a8a54df7b52efd0

$ kubectl -n flux-system create secret generic webhook-token \
--from-literal=token=$TOKEN
secret/webhook-token created

$ cat << EOF | kubectl apply -f -
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Receiver
metadata:
 name: generic-receiver
 namespace: flux-system
spec:
 type: generic
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: source.toolkit.fluxcd.io/v1beta1
 kind: GitRepository
 name: alpine
 namespace: flux-system
EOF
receiver.notification.toolkit.fluxcd.io/generic-receiver created

$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
 name: receiver
 namespace: flux-system
spec:
 type: ClusterIP
 selector:
 app: notification-controller
 ports:
 - name: http
 port: 80
 protocol: TCP
 targetPort: 9292
EOF

This program is basically a CLI application and it uses the following arguments as:

args:
 - "--owner"
 - "developer-guy" # this is the owner of the desired state repository
 - "--repository"
 - "desired-state-repository" # this is the repository which we store desired state configurations
 - "--file"
 - "alpine.yaml" # this is the file that we are going to update
 - "--github-token"
 - "$(GITHUB_TOKEN)"
 - "--notify-url"
 - "http://receiver.flux-system/$(WEBHOOK_URL)"

Let us run our project on the Kubernetes cluster. To do that, we'll be using the ko tool. ko, created by Google, is a simple, fast container image builder for Go applications. For more information, see the official repository of the project.

We'll use Container Registry as an image repository service provided by the Google Cloud to store, manage, and secure our container images. Alternatively, we could also use DockerHub, quay.io, and so on.

export WEBHOOK_URL=$(kubectl get receivers -n flux-system generic-receiver -ojsonpath='{.status.url}')
export PROJECT_ID=$(gcloud config get-value project)
export KO_DOCKER_REPO=gcr.io/$PROJECT_ID
envsubst < falcoeventlistener.yaml | ko apply -f -

If you are using a private container registry, don't forget to create a registry secret to pull and push images. You can follow the following guide to achieve this:
https://colinwilson.uk/2020/07/09/using-google-container-registry-with-kubernetes/

If everything works as expected, we should see an output as given below:

$ kubectl get pods --namespace falco
NAME READY STATUS RESTARTS AGE
falco-event-listener 1/1 Running 0 59s

The last step we have to do is installing Falco and Falcosidekick with configuring Falcodekick to forward events to our application.

Installing Falco and Falcosidekick

For an up-to-date and detailed guide to installing Falco and Falcosidekick, see falcosidekick).

Let us enable CloudEvents support of Falcosidekick.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm upgrade --install falco falcosecurity/falco \
 --namespace falco \
 --set falcosidekick.enabled=true \
 --set falcosidekick.webui.enabled=true \
 --set falcosidekick.config.cloudevents.address=http://falco-event-listener

Verify that everything is working as expected:

$ kubectl get pods --namespace falco
NAME READY STATUS RESTARTS AGE
falco-event-listener 1/1 Running 0 5m19s
falco-falcosidekick-5854669c76-ddvrv 1/1 Running 0 2m16s
falco-falcosidekick-5854669c76-rdlqn 1/1 Running 0 2m16s
falco-falcosidekick-ui-7c5fc8dd54-q4qh9 1/1 Running 0 2m16s
falco-vkl4f 1/1 Running 0 2m16s

Test

To test this, we have to connect a shell within the container.

Let's list the pods that we already have.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
alpine-deployment-77789455d6-m7flp 1/1 Running 15 3h6m
alpine-deployment-77789455d6-v7fkw 1/1 Running 15 3h6m
podinfo-6df788c7b8-gs5qb 1/1 Running 0 3h28m
podinfo-6df788c7b8-sfxvd 1/1 Running 0 3h28m

Now, run the following command to connect a shell.

$ kubectl exec -ti alpine-deployment-77789455d6-m7flp -- sh -c "uptime"
19:35:58 up 3:04, load average: 3.12, 1.91, 1.22

Once you run the command above, Falco will detect that malicious behavior and send it to the Falcosidekick via HTTP.

You should see an output in the Falco logs as given below:

{"output":"19:35:58.532086161: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine-deployment-77789455d6-m7flp container=788861c3cf83 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=788861c3cf83 image=alpine) k8s.ns=default k8s.pod=alpine-deployment-77789455d6-m7flp container=788861c3cf83","priority":"Notice","rule":"Terminal shell in container","time":"2021-06-13T19:35:58.532086161Z", "output_fields": {"container.id":"788861c3cf83","container.image.repository":"alpine","evt.time":1623612958532086161,"k8s.ns.name":"default","k8s.pod.name":"alpine-deployment-77789455d6-m7flp","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}

and the similar output below in the Falcosidekick logs:

falco-falcosidekick-5854669c76-ddvrv falcosidekick 2021/06/13 19:51:45 [INFO] : CloudEvents - Send OK

and the similar output below in the falco-event-listener logs:

falco-event-listener falco-event-listener 2021/06/13 19:35:59 resp.Status=200 OK
falco-event-listener falco-event-listener 2021/06/13 19:35:59 resp.StatusCode=200
falco-event-listener falco-event-listener 2021/06/13 19:35:59 [Terminal shell in container] scaled down to zero alpine-deployment-77789455d6-m7flp from default because 19:35:58.532086161: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine-deployment-77789455d6-m7flp container=788861c3cf83 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=788861c3cf83 image=alpine) k8s.ns=default k8s.pod=alpine-deployment-77789455d6-m7flp container=788861c3cf83

You should also notice that a new commit is available in the desired-state-repository as given below:

After the commit, Flux (version 2) will detect the change and sync the current state of the cluster with the desired state in the GitHub repository so that Flux (version 2) will terminate the alpine deployment pods.

$ kubectl get pods -l app=alpine
No resources found in default namespace.

Blog: Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

Tue, 29 Jun 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Earlier in this series, we have seen how to use Argo, Tekton, and Knative to trigger a resource after getting input from Falcosidekick. Recently, Falcosidekick received a new output type support for Cloud Functions.

In this part, let us learn how we can use Falcosidekick and Cloud Functions to detect and delete a compromised pod.

We will not go through what Cloud Functions is in-depth, however, you can always find a good overview about it in the official documentation.

Here is the high-level overview architecture that shows what we want to achieve at the end of the day:

You can find all the related code and resources in this repository.

Prerequisites

gcloud 342.0.0
kubectl 1.20.5

Tutorial

Provision Google Kubernetes Engine (GKE) Cluster

As the blog title said already, we need to create a GKE cluster with workload identity enabled:

$ GOOGLE_PROJECT_ID=$(gcloud config get-value project)
$ CLUSTER_NAME=falco-falcosidekick-demo
$ gcloud container clusters create $CLUSTER_NAME --workload-pool ${GOOGLE_PROJECT_ID}.svc.id.goog
$ gcloud container clusters get-credentials $CLUSTER_NAME

Configure IAM Service Accounts

We need to create a new Service Account for target $GOOGLE_PROJECT_ID using IAM Binding policies to get access our Cloud Function:

$ SA_ACCOUNT=falco-falcosidekick-sa
$ gcloud iam service-accounts create $SA_ACCOUNT

$ gcloud projects add-iam-policy-binding ${GOOGLE_PROJECT_ID} \
--member="serviceAccount:${SA_ACCOUNT}@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/cloudfunctions.developer"

$ gcloud projects add-iam-policy-binding ${GOOGLE_PROJECT_ID} \
--member="serviceAccount:${SA_ACCOUNT}@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/cloudfunctions.invoker"

In the beginning, we already enabled WorkloadIdentity feature for our GKE Cluster by setting --workload-pool flag. What we need to do here is to add a iam.workloadIdentityUser role for the given Service Account.

$ gcloud iam service-accounts add-iam-policy-binding \
 --role roles/iam.workloadIdentityUser \
 --member "serviceAccount:${GOOGLE_PROJECT_ID}.svc.id.goog[${FALCO_NAMESPACE}/falco-falcosidekick]" \
 ${SA_ACCOUNT}@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com

We need to annotate the falco-falcosidekick resource. So it can grant access for our Cluster. Set up the Falcosidekick SA to impersonate a GCP SA:

$ kubectl annotate serviceaccount \
 --namespace $FALCO_NAMESPACE \
 falco-falcosidekick \
 iam.gke.io/gcp-service-account=${SA_ACCOUNT}@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com

Create Necessary Cluster Role

To limit function role access in the particular cluster, ensure that only SA has limited permissions within a particular namespace by using Role Bindings.

$ kubectl create serviceaccount pod-destroyer
$ kubectl create clusterrole pod-destroyer
 --verb=delete
 --resource=pod # give only pod resource access for delete op 
$ kubectl create clusterrolebinding pod-destroyer
 --clusterrole pod-destroyer
 --serviceaccount default:pod-destroyer

To obtain the Token from secret, we need to get pod-deleter ServiceAccount resource first:

$ POD_DESTROYER_TOKEN=$(kubectl get secrets $(kubectl get serviceaccounts pod-deleter -o json \
 | jq -r '.secrets[0].name') -o json \
 | jq -r '.data.token' \
 | base64 -D)

Add the pod-destroyer user to your KUBECONFIG:

# Generate your KUBECONFIG
$ kubectl config view --minify --flatten > kubeconfig_pod-destroyer.yaml

# Set the token at the end of yaml
$ cat << EOF >> kubeconfig_pod-destroyer.yaml
users:
- name: user.name
 user:
 token: $POD_DESTROYER_TOKEN

We can test it with auth can-i to check if roles are set correctly

$ kubectl auth can-i list deployments # no
$ kubectl auth can-i delete pod # yes
$ kubectl access-matrix # github.com/corneliusweig/rakkess

Create Secret Manager

The main reason Secret Manager get involved our architecture is because we had to find a way out to initialize our kubeclient in our function by getting pod-destroyer's KUBECONFIG.

We need to create a new secrets IAM policy for the SA member to enable Managing Secrets:

$ gcloud secrets add-iam-policy-binding pod-destroyer \
 --role roles/secretmanager.secretAccessor \
 --member serviceAccount:$SA_ACCOUNT@$GOOLE_PROJECT_ID.iam.gserviceaccount.com

Create a new secret, called pod-destroyer:

$ gcloud secrets create pod-destroyer --replication-policy="automatic"

Push the our generated kubeconfig_pod-destroyer.yaml file as a new version:

$ gcloud secrets versions add pod-destroyer --data-file=kubeconfig_pod-destroyer.yaml

Finally, we are ready to deploy our Cloud Run function!

Deploy Google Cloud Function

In this demonstration our function will simply delete the pwned Pod, as we already pointed it out in the architecture diagram.

The Go code

package kill_the_pwned_pod

import (
 secretmanager "cloud.google.com/go/secretmanager/apiv1"
 "context"
 "encoding/json"
 "fmt"
 secretmanagerpb "google.golang.org/genproto/googleapis/cloud/secretmanager/v1"
 "io/ioutil"
 metaV1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/kubernetes"
 "k8s.io/client-go/tools/clientcmd"
 "net/http"
 "os"
 "time"
)

// Alert falco data structure
type Alert struct {
 Output string `json:"output"`
 Priority string `json:"priority"`
 Rule string `json:"rule"`
 Time time.Time `json:"time"`
 OutputFields struct {
 ContainerID string `json:"container.id"`
 ContainerImageRepository interface{} `json:"container.image.repository"`
 ContainerImageTag interface{} `json:"container.image.tag"`
 EvtTime int64 `json:"evt.time"`
 FdName string `json:"fd.name"`
 K8SNsName string `json:"k8s.ns.name"`
 K8SPodName string `json:"k8s.pod.name"`
 ProcCmdline string `json:"proc.cmdline"`
 } `json:"output_fields"`
}

var op Operation

type Operation struct {
 clientSet *kubernetes.Clientset
}

// init initializes new Kubernetes ClientSet with given config
func init() {
 // The resource name of the KUBECONFIG_SECRET_NAME in the format
 // `projects/*/secrets/*/versions/*`
 resource := os.Getenv("KUBECONFIG_SECRET_NAME")
 if len(resource) == 0 {
 panic(fmt.Errorf("$KUBECONFIG_SECRET_NAME env variable did not set"))
 }

 secret, err := GetSecret(resource)
 if err != nil {
 panic(fmt.Errorf("get secret: %q", err))
 }

 kubeCfg, err := clientcmd.NewClientConfigFromBytes(secret)
 if err != nil {
 panic(fmt.Errorf("new client config: %q", err))
 }

 restCfg, err := kubeCfg.ClientConfig()
 if err != nil {
 panic(fmt.Errorf("client config: %q", err))
 }

 cs, err := kubernetes.NewForConfig(restCfg)
 if err != nil {
 panic(fmt.Errorf("unable to initialize config: %q", err))
 }

 op = Operation{clientSet: cs}
}

// KillThePwnedPod will executed for each Falco event
func KillThePwnedPod(w http.ResponseWriter, r *http.Request) {
 body, err := ioutil.ReadAll(r.Body)
 if err != nil {
 http.Error(w, "cannot read body", http.StatusBadRequest)
 return
 }

 var event Alert

 err = json.Unmarshal(body, &event)
 if err != nil {
 http.Error(w, "cannot parse body", http.StatusBadRequest)
 return
 }

 err = op.PodDestroy(event.OutputFields.K8SPodName, event.OutputFields.K8SNsName)
 if err != nil {
 http.Error(w, fmt.Sprintf("cannot delete pod: %q", err), http.StatusInternalServerError)
 return
 }

 w.WriteHeader(http.StatusOK)
}

// PodDestroy destroys the given pod name in the given namespace
func (d *Operation) PodDestroy(name, namespace string) error {
 err := d.clientSet.CoreV1().Pods(namespace).Delete(context.TODO(), name, metaV1.DeleteOptions{})
 if err != nil {
 return fmt.Errorf("unable to delete pod %s: %q", name, err)
 }
 return nil
}

// GetSecret returns the secret data.
func GetSecret(name string) ([]byte, error) {
 ctx := context.Background()

 client, err := secretmanager.NewClient(ctx)
 if err != nil {
 return nil, fmt.Errorf("failed to create secretmanager client: %v", err)
 }
 defer client.Close()

 result, err := client.AccessSecretVersion(ctx, &secretmanagerpb.AccessSecretVersionRequest{
 Name: name,
 })

 if err != nil {
 return nil, fmt.Errorf("failed to access secret version: %v", err)
 }

 return result.Payload.Data, nil
}

If you rather see it in GitHub.

$ git clone https://github.com/Dentrax/k8s-response-engine-gke-functions.git
$ cd kubernetes-response-engine-based-on-gke-and-gcloudfunctions
...

We need to pass extra --service-account flag in order to get access to Secret Manager.

Deploy the function:

$ FUNCTION_NAME=KillThePwnedPod
$ gcloud functions deploy $FUNCTION_NAME \
--runtime go113 --trigger-http \
--service-account $SA_ACCOUNT@$GOOLE_PROJECT_ID.iam.gserviceaccount.com
Allow unauthenticated invocations of new function [KillThePwnedPod]? (y/N)? N
...

Now, get the name of the function:

$ CLOUD_FUNCTION_NAME=$(gcloud functions describe --format=json $FUNCTION_NAME | jq -r '.name')

Install Falco + Falcosidekick

It is time to install Falco, Falcosidekick with Cloud Function output type enabled:

$ export FALCO_NAMESPACE=falco
$ kubectl create namespace $FALCO_NAMESPACE
$ helm upgrade --install falco falcosecurity/falco \
--namespace $FALCO_NAMESPACE \
--set ebpf.enabled=true \
--set falcosidekick.enabled=true \
--set falcosidekick.config.gcp.cloudfunctions.name=${CLOUD_FUNCTION_NAME} \
--set falcosidekick.webui.enabled=true

Test

Try to run a busybox image and execute a command:

$ kubectl run busybox --image=busybox --restart='Never' -- sh -c "sleep 600"

Try to exec into:

$ kubectl exec -it busybox -- sh -c "uptime"

We can check the logs of the Falco, and Falcosidekick to see what happened:

For Falcosidekick:

$ kubectl logs deployment/falco-falcosidekick --namespace falco
2021/06/14 21:01:24 [INFO] : GCPCloudFunctions - Call Cloud Function OK
..

Conclusion

We got another way to create a Response Engine with amazing pieces of software from Open Source world. Of course, it's just the beginning, feel free to share your functions and workflows with the community for starting the creation of a true library of remediation methods.

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 6: Falcosidekick + Cloud Run

Fri, 25 Jun 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

Recently, we added two new output-type support to Falcosidekick, and they are Cloud Functions, and Cloud Run. This blog post will discuss how to set up Kubernetes Response Engine on GKE (Google Kubernetes Engine) by using Cloud Run.

Let's start by explaining a little bit about Cloud Run. Cloud Run is a managed compute platform that enables you to run containers that are invocable via requests or events. Cloud Run is serverless: it abstracts away all infrastructure management, so you can focus on what matters most — building great applications.

For more information, see Cloud Run.

Given below is a reference architecture of what's being explained in this blog.

This demo might be useful for Google Cloud users who might already be using GKE with Falco to protect container runtime against malicious behaviors, and wants to take any action for them with Cloud Run.

Table of Contents

Prerequisites

gcloud 342.0.0
ko 0.8.3

Tutorial

Provision GKE (Google Kubernetes Engine) Cluster

First, let us create a GKE cluster.

$ GKE_CLUSTER_NAME=cloud-run-demo
$ gcloud container clusters create $GKE_CLUSTER_NAME

To learn more about the setup GKE Cluster, see quickstart guide.

Deploy Cloud Run Function

Once GKE is set up, we are ready to deploy Cloud Run. But before doing that, let us examine the responsibility of the Cloud Run function. As you can see in the reference architecture, this function will delete the pwned pods. To be able to do that, Cloud Run should be given appropriate permissions.

There are two approaches to obtain these permissions.

The first approach is creating a Kubernetes Service Account, an appropriate Role with granted permissions to delete pod resource, and a RoleBinding to bind Role to Service Account. Then create the kubeconfig file, package it up with the function code while deploying the Cloud Run function, and use this file to create a Kubernetes client.

To learn more about the kubeconfig files, see kubeconfig.

The second approach is producing a valid ~/.kube/config with a library called google.golang.org/api/ within the function code. We are doing this because the representation of the valid ~/.kube/config file is clientcmd/api/Config in Go.

We'll go with the second approach in this demo. Thanks to Scott Blum and his detailed blog post on this topic. I highly recommend that you check that out.

Let's deploy the function. If you want to take a look at the function code, see the repository.

Note that we're going to use the ko tool to build and push our container image which is created by Google. ko is a simple and fast container image builder for Go applications.

To learn more, see the official repository of the project.

We are also going to use Container Registry as an image repository service provided by the Google Cloud to store, manage, and secure your Docker container images. Alternatively, you can also use DockerHub, quay.io, etc.

$ git clone https://github.com/developer-guy/kubernetes-response-engine-based-on-gke-and-cloud-run.git
$ cd kubernetes-response-engine-based-on-gke-and-cloud-run
$ GKE_CLUSTER_NAME=$(kubectl config view --minify -o jsonpath='{.clusters[].name}{"\n"}')
$ PROJECT_ID=$(gcloud config get-value project)
$ FUNCTION_NAME=pod-deleter
$ export KO_DOCKER_REPO=gcr.io/$PROJECT_ID # Please, change this variable if you are not using Container Registry.
$ gcloud config set run/region us-west1
$ gcloud config set run/platform managed
$ gcloud run deploy $FUNCTION_NAME --image=$(ko publish .) \
 --set-env-vars GKE_CLUSTER_NAME=$GKE_CLUSTER_NAME \
 --set-env-vars PROJECT_ID=$PROJECT_ID
...
Allow unauthenticated invocations to [pod-deleter] (y/N)? N

Deploying container to Cloud Run service [pod-deleter] in project [developerguy-311909] region [us-west1]
✓ Deploying... Done.
 ✓ Creating Revision...
 ✓ Routing traffic...
Done.
Service [pod-deleter] revision [pod-deleter-00002-cej] has been deployed and is serving 100 percent of traffic.
Service URL: https://pod-deleter-uoz6q2wria-uw.a.run.app

Install Falco + Falcosidekick

Now, it is time to set up Falco, Falcosidekick with the Cloud Run output type enabled.

$ kubectl create namespace falco
$ helm repo add falcosecurity https://falcosecurity.github.io/charts
$ helm repo update
$ helm install falco falcosecurity/falco --namespace falco \
--set ebpf.enabled=true \
--set falcosidekick.enabled=true \
--set falcosidekick.webui.enabled=true \
--set falcosidekick.config.gcp.cloudrun.endpoint=$(gcloud run services list --format json | jq -r ".[] | select(.metadata.name==\"$FUNCTION_NAME\") | .status.address.url") \
--set falcosidekick.config.gcp.cloudrun.jwt=$(gcloud auth print-identity-token)

Check the logs to see if Cloud Run output enabled for Falcosidekick.

$ kubectl logs -f falco-falcosidekick-7cd7bc6859-2nd9t --namespace falco
falco-falcosidekick-7cd7bc6859-2nd9t falcosidekick 2021/06/07 16:03:14 [INFO] : Enabled Outputs : [GCPCloudRun WebUI]
falco-falcosidekick-7cd7bc6859-2nd9t falcosidekick 2021/06/07 16:03:14 [INFO] : Falco Sidekick is up and listening on :2801
...

If you see the GCPCloudRun in the list of enabled outputs, you can confirm that everything is working as expected 👍.

Test

Let us start by creating a test pod:

$ kubectl run alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"
pod/alpine created

$ kubectl get pods --namespace default
AME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 11s

Let's run a shell command inside and see what happens:

$ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"
19:27:21 up 50 min, load average: 0.11, 0.12, 0.11

As expected the command returned the output. However, the status of the pod we retrieved is Terminating as follows:

$ kubectl get pods --namespace default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 103s

To investigate further, check the logs of the Cloud Run function from the Google Cloud Console:

Let us check the logs of Falco and Falcosidekick to see what happened.

 $ kubectl logs daemonset/falco --namespace falco
..
{"output":"19:27:21.002873265: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=97c9868ea832 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=97c9868ea832 image=alpine) k8s.ns=default k8s.pod=alpine container=97c9868ea832","priority":"Notice","rule":"Terminal shell in container","time":"2021-04-10T19:27:21.002873265Z", "output_fields": {"container.id":"97c9868ea832","container.image.repository":"alpine","evt.time":1618082841002873265,"k8s.ns.name":"default","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}
...

For Falcosidekick:

$ kubectl logs deployment/falco-falcosidekick --namespace falco
2021/06/07 16:03:15 [INFO] : GCPCloudRun - Post OK (200)
2021/06/07 16:03:15 [INFO] : GCPCloudRun - Function Response : OK
..

Conclusion

We got another way to create a Response Engine with amazing pieces of software from the Open Source world. Of course, it's just the beginning, feel free to share your functions and workflows with the community for creating a library of remediation methods.

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 5: Falcosidekick + Argo

Sun, 23 May 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

The Open Source ecosystem is very vibrant, there are many ways to create a Kubernetes Response Engine based on our dynamic duo, Falco + Falcosidekick. Today, we will use two components of the CNCF project Argo:

Argo Events, will receive events from Falcosidekick and push into it event bus.
Argo Workflow, will listen the event bus and then trigger the workflow if certain criteria are encountered.

Like we did for previous examples with Kubeless, OpenFaas and Knative, we'll address the situation where a shell is spawned in a pod and we want to remediate that by deleting it.

This is how we will set this up:

┌─────────────┐ ┌─────────┐ ┌────────────────┐
│ │ detect │ │ push │ │
│ pwned pod ├───────────► falco ├──────────► falcosidekick ├────┐
│ │ │ │ │ │ │
└──────▲──────┘ └─────────┘ └────────────────┘ │ notify
│ │
│ │
delete │ ┌──────────────┐ ┌───────────────┐ ┌──────▼──────┐
│ │ │ │ │ │ │
└───┤ deletion pod ◄──────────┤ argo workflow │ │ argo events │
│ │ create │ │ │ │
└──────────────┘ └────────────▲──┘ └─┬───────────┘
│ │
trigger │ │ push
│ │
┌─┴─────────────▼──┐
│ bus │
└──────────────────┘

Requirements

We require a kubernetes cluster running at least 1.17 release, helm and kubectl installed in your locale environment.

Installation of Argo Events

We simply follow the official documentation.

kubectl create namespace argo-events
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install-validating-webhook.yaml
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml

Installation of Argo Workflow

Again, the official documentation will help us.

kubectl create namespace argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/install.yaml
kubectl patch -n argo cm workflow-controller-configmap -p '{"data": {"containerRuntimeExecutor": "pns"}}'

The kubectl patch is there for allowing the workflows to run in minikube, kind, etc. See docs about Workflow Executors to learn more about.

After a while, you should have access to Argo Workflow UI through a dport-forward:

kubectl port-forward svc/argo-server -n argo 2746:2746 &

The link is https://localhost:2746 (you can ignore the certificate error, we're in a lab 😉).

Creation of the Event Source

We'll use an Event Source with Webhook type. It will receive Falco events from Falcosidekick and push them then into the Event Bus.

This component is pretty easy to understand. Falcosidekick will have to POST the events to an endpoint /falco of a service opened on port 12000. Easy.

cat <<EOF | kubectl apply -n argo-events -f -
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: webhook-falco
namespace: argo-events
spec:
service:
ports:
- port: 12000
targetPort: 12000
webhook:
# event-source can run multiple HTTP servers. Simply define a unique port to start a new HTTP server
falco-event:
# port to run HTTP server on
port: "12000"
# endpoint to listen to
endpoint: /falco
# HTTP request method to allow. In this case, only POST requests are accepted
method: POST
EOF

As expected, we now have a new service which will listen events from Falcosidekick on port 12000 and endpoint /falco:

kubectl get svc -n argo-events
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
webhook-falco-eventsource-svc ClusterIP 10.43.117.26 <none> 12000/TCP 11m

Creation of the Sensor

In Argo Events architecture, Sensors are responsible for listening to the Event Bus and triggering something should the criteria we set match. In our case, our Sensor:

listen only events for pushed by webhook-falco Event Source
consider only events where the body (in JSON) contains the value Terminal shell in container for field with key rule, we want to match for only this Falco rule in one word.
trigger a workflow based on a template with our event as input

First, create the Service Account which allows our Sensor will.

cat <<EOF | kubectl apply -n argo-events -f -
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: argo-events
name: sensor-terminal-shell-container-sa
---
# Similarly you can use a ClusterRole and ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: sensor-terminal-shell-container-role
namespace: argo-events
rules:
- apiGroups:
- argoproj.io
verbs:
- "*"
resources:
- workflows
- workflowtemplates
- cronworkflows
- clusterworkflowtemplates
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: sensor-terminal-shell-container-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: sensor-terminal-shell-container-role
subjects:
- kind: ServiceAccount
name: sensor-terminal-shell-container-sa
namespace: argo-events
EOF

And now we deploy our Sensor.

cat <<EOF | kubectl apply -n argo-events -f -
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: terminal-shell-container
namespace: argo-events
spec:
template:
serviceAccountName: sensor-terminal-shell-container-sa
dependencies:
- name: falco-event
eventSourceName: webhook-falco
eventName: falco-event
filters:
data:
- path: body.rule
type: string
value:
- "Terminal shell in container"
triggers:
- template:
name: delete-pod-trigger
argoWorkflow:
group: argoproj.io
version: v1alpha1
resource: workflows
operation: submit
parameters:
- src:
dependencyName: falco-event
dest: spec.arguments.parameters.0.value
source:
resource:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: delete-pod-
namespace: argo
spec:
workflowTemplateRef:
name: delete-pod-template
arguments:
parameters:
- name: falco-event
value: {}
EOF

Creation of the Workflow Template

There is one piece missing in our Argo stack, we mentioned a template above, we logically need to create it too, with the service account it needs.

cat <<EOF | kubectl apply -n argo -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: delete-pod-sa
namespace: argo
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: delete-pod-sa-cluster-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "delete", "patch", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: delete-pod-sa-cluster-role-binding
roleRef:
kind: ClusterRole
name: delete-pod-sa-cluster-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: delete-pod-sa
namespace: argo
EOF

cat <<EOF | kubectl apply -n argo -f -
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: delete-pod-template
namespace: argo
spec:
entrypoint: delete-pod
serviceAccountName: delete-pod-sa
arguments:
parameters:
- name: falco-event
value: "{}"
templates:
- name: delete-pod
inputs:
parameters:
- name: falco-event
container:
image: devopps/kubernetes-response-engine-based-on-event-driven-workflow@sha256:22ee203a33fe88f0f99968daebdcea0ca52c8a3d6f7af4c823ed78ac15b7c5db
env:
- name: BODY
value: "{{inputs.parameters.falco-event}}"
EOF

Argo Workflow runs all workflow steps inside their own pods, we'll use for this tutorial a Golang image developped by @developer-guy (who wrote the Part 2 of this series 😄), the sources are there.

At this stage, everything is ready to receive events from Falco and protect our cluster. If you go in Argo Workflow UI you will find the architecture we described at beginning.

Installation of Falco and Falcosidekick

Last but not least, it's time to install our beloved Falco and Falcosidekick and connect them to our shiny new Response Engine.

As with other posts of this series we'll use Helm as conveniant installation method.

kubectl create ns falco
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
--namespace falco \
--set falcosidekick.enabled=true \
--set falcosidekick.config.webhook.address=http://webhook-falco-eventsource-svc.argo-events.svc.cluster.local:12000/falco

Remember the service we "mentioned" earlier? This is it in its FQDN format as an endpoint.

Test our Response Engine

Let's delete pwned pod !

We'll simulate a webshell by executing a shell command into a running pod.

kubectl run alpine -n default --image=alpine --restart='Never' -- sh -c "sleep 6000"
kubectl get pods -n default

NAME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 8s

Run a shell command inside.

kubectl exec -i --tty alpine -n default -- sh -c "uptime"
22:03:23 up 44 min, load average: 0.07, 0.13, 0.19

If you're quick enough, you may see the termination of the pod.

kubectl get pods -n default

NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 8s

And in Argo Workflow UI.

👍

Go a little further with Argo

We can even go further by deploying all components with Argo CD, another project from Argo team. You can find out all you need in this repo.

Here a quick demo of the results with the exact same workflow we just created in this tutorial.

Conclusion

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 4: Falcosidekick + Tekton

Fri, 14 May 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

Falcosidekick + Tekton

Earler in this series we have seen how to use Kubeless, OpenFaas and Knative to trigger a pod after getting input from falcosidekick to delete a compromised pod.

In this part I will showcase how we can use Tekton and not have to add any extra complexity to your cluster by adding a serverless runtime.

I won't go through how Tekton works in depth but, you can find a good overview in the official docs. But here is the crash course:

Tekton is built to be reusable.
The smallest part of tekton is a step, a step can be something like this:
- Run unit tests
- Run linting
In a task you can have multiple steps.
A pipeline consist of one or multiple tasks.
To trigger a pipeline to actually run you need a pipelinerun or a trigger-template.

Tekton also supports eventlisteners that is used to listen for webhooks. Normally these webhooks listen for incoming changes to a git repo, for example a PR. But we will use it to listen for Falco events.

You can find all the yaml and code in my repo.

Prerequisites

As always within Kubernetes we need a few tools, I have used the following versions of Helm, Minikube and kubectl in my setup.

Minikube v1.19.0
Helm v3.4.2
kubectl v1.20.5

Provision local Kubernetes Cluster

I'm sure you can use a kind cluster as well to follow along, but falco complained a bit when I tried and I was too lazy to check out what extra flags I need so I went with minikube.

minikube start --cpus 3 --memory 8192 --vm-driver virtualbox

Install Tekton

Install Tekton pipelines and triggers. When doing this in production I recommend the Tekton operator but for now let us use some pure yaml.

kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml

Within a few seconds you should be able to see a few pods in the tekton-pipelines namespace.

kubectl get pods -n tekton-pipelines
NAME READY STATUS RESTARTS AGE
tekton-pipelines-controller-6b94f5f96-cmf8m 1/1 Running 0 1h
tekton-pipelines-webhook-5bfbbd6475-fmjp4 1/1 Running 0 1h
tekton-triggers-controller-7cbd49fbb8-p4lrz 1/1 Running 0 1h
tekton-triggers-webhook-748fb7778c-w6zxv 1/1 Running 0 1h

If you want a deeper understanding how Tekton triggers work check out the getting-started guide.

Install Falco + Falcosidekick

Create the falco namespace and add the helm repo:

kubectl create namespace falco
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

For simplicity and long term usability let us create a custom values file and start falco.

cat <<'EOF' >> values.yaml
falcosidekick:
 config:
 webhook:
 address: http://el-falco-listener.falcoresponse.svc.cluster.local:8080
 enabled: true

customRules:
 # Applications which are expected to communicate with the Kubernetes API
 rules_user_known_k8s_api_callers.yaml: |-
 - macro: user_known_contact_k8s_api_server_activities
 condition: >
 (container.image.repository = "gcr.io/tekton-releases/github.com/tektoncd/triggers/cmd/eventlistenersink") or
 (container.image.repository = "quay.io/nissessenap/poddeleter")
EOF

# Install falco
helm upgrade --install falco falcosecurity/falco --namespace falco -f values.yaml

Note the customRules and the webhook address.

We haven't setup this webhook address nor is there currently any reason for us to have customRules for eventlistenersink or poddeleter, but it will come. Both the Tekton event listener and my poddeleter does a few kubernetes API calls and we don't want falco generate alarms for our own infrastructure.

You should be able to see falco and falcosidekick pods in the falco namespace:

kubectl get pods --namespace falco

NAME READY STATUS RESTARTS AGE
falco-44p4v 1/1 Running 0 64m
falco-falcosidekick-779b87f446-8zf9m 1/1 Running 0 2h
falco-falcosidekick-779b87f446-fdk55 1/1 Running 0 2h

Protect me Falco

My current setup is rather harsh and will delete any pods that breaks any falco rule. In the future I plan to make both the go code and the tekton setup better and more flexible, hopefully this is something that we can do in the community.

During this demo I will use the Terminal Shell in container since it's very easy to reproduce.

So how does all this work?

We start a random pod and perform a simple exec.
Falco will notice that a pod have broken the rule
Sends an event to Falcosidekick
Sends a webhook to tekton event-listener
Tekton triggers a new pipeline
A task is started with a small go program that deletes the pod

So let us look at some yaml.

The go code

I have adapted the code that Batuhan Apaydın wrote in Falcosidekick + OpenFaas = a Kubernetes Response Engine, Part 2 to listen for json in an environment variable instead of a http request.

Below you can see the code, in short it does the following:

Check for environment variable BODY.
Unmarshal the data according to the Alert struct.
Setups a kubernetes client, by calling setupKubeClient function.
Calls the deletePod with a kubernetes client, the falcoEvent we gotten and a hash map of critical Namespaces.
Check in the event that we got from falcosidekick and see if the pod that triggered the event is in our critical namespaces hash map.
If it is return to the main and shutdown the application.
Else deletes the pod in the namespace specified in the falcosidekick event.

package main

import (
 "context"
 "encoding/json"
 "log"
 "os"
 "time"

 metaV1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/kubernetes"
 "k8s.io/client-go/rest"
)

// Alert falco data structure
type Alert struct {
 Output string `json:"output"`
 Priority string `json:"priority"`
 Rule string `json:"rule"`
 Time time.Time `json:"time"`
 OutputFields struct {
 ContainerID string `json:"container.id"`
 ContainerImageRepository interface{} `json:"container.image.repository"`
 ContainerImageTag interface{} `json:"container.image.tag"`
 EvtTime int64 `json:"evt.time"`
 FdName string `json:"fd.name"`
 K8SNsName string `json:"k8s.ns.name"`
 K8SPodName string `json:"k8s.pod.name"`
 ProcCmdline string `json:"proc.cmdline"`
 } `json:"output_fields"`
}

func main() {
 criticalNamespaces := map[string]bool{
 "kube-system": true,
 "kube-public": true,
 "kube-node-lease": true,
 "falco": true,
 }

 var falcoEvent Alert

 bodyReq := os.Getenv("BODY")
 if bodyReq == "" {
 log.Fatalf("Need to get environment variable BODY")
 }
 bodyReqByte := []byte(bodyReq)
 err := json.Unmarshal(bodyReqByte, &falcoEvent)
 if err != nil {
 log.Fatalf("The data doesent match the struct %v", err)
 }

 kubeClient, err := setupKubeClient()
 if err != nil {
 log.Fatalf("Unable to create in-cluster config: %v", err)
 }

 err = deletePod(kubeClient, falcoEvent, criticalNamespaces)
 if err != nil {
 log.Fatalf("Unable to delete pod due to err %v", err)
 }
}

// setupKubeClient
func setupKubeClient() (*kubernetes.Clientset, error) {
 config, err := rest.InClusterConfig()
 if err != nil {
 return nil, err
 }

 // creates the clientset
 kubeClient, err := kubernetes.NewForConfig(config)
 if err != nil {
 return nil, err
 }
 return kubeClient, nil
}

// deletePod, if not part of the criticalNamespaces the pod will be deleted
func deletePod(kubeClient *kubernetes.Clientset, falcoEvent Alert, criticalNamespaces map[string]bool) error {
 podName := falcoEvent.OutputFields.K8SPodName
 namespace := falcoEvent.OutputFields.K8SNsName
 log.Printf("PodName: %v & Namespace: %v", podName, namespace)

 log.Printf("Rule: %v", falcoEvent.Rule)
 if criticalNamespaces[namespace] {
 log.Printf("The pod %v won't be deleted due to it's part of the critical ns list: %v ", podName, namespace)
 return nil
 }

 log.Printf("Deleting pod %s from namespace %s", podName, namespace)
 err := kubeClient.CoreV1().Pods(namespace).Delete(context.Background(), podName, metaV1.DeleteOptions{})
 if err != nil {
 return err
 }
 return nil
}

If you rather see it in github.

Now that you know what I will make run in your cluster let us take a look at the Tekton yaml.

Tekton pipeline

Create the falcoresponse namespace to do our tests in.

kubectl create ns falcoresponse

Task

So let us start with the smallest part, the task.

cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
 name: pod-delete
 namespace: falcoresponse
spec:
 params:
 - name: falco-event
 description: The entire msg from falco
 steps:
 - name: pod-delete
 image: quay.io/nissessenap/poddeleter@sha256:ae94ec2c9f005573e31e4944d1055a0dd92ee7594e7e7e36a4540a1811977270
 env:
 - name: BODY
 value: \$(params.falco-event)
EOF

The task needs a input variable falco-event.
The step called pod-delete uses the poddeleter image.
Step pod-delete sets the environment BODY from the input parameter called falco-event.

Pipeline

Here you can see the reusability of tekton. This pipeline can easily add more tasks and other pipelines can use the exact same task as this one.

Just like the task this pipeline expects a parameter called falco-event which it sends in to the pod-delete task.

cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
 name: pod-delete-pipeline
 namespace: falcoresponse
spec:
 params:
 - name: falco-event
 description: The entire msg from falco
 tasks:
 - name: run-pod-delete
 taskRef:
 name: pod-delete
 params:
 - name: falco-event
 value: \$(params.falco-event)
EOF

RBAC

We will be using two separate serviceAccounts, one for the event-listener and one for the poddeleter it self.

So let us create these serviceAccounts and give them some access.

Below you can find the event listener RBAC config.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
 name: tekton-triggers-example-sa
 namespace: falcoresponse
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: tekton-triggers-example-minimal
 namespace: falcoresponse
rules:
 # EventListeners need to be able to fetch all namespaced resources
 - apiGroups: ["triggers.tekton.dev"]
 resources:
 ["eventlisteners", "triggerbindings", "triggertemplates", "triggers"]
 verbs: ["get", "list", "watch"]
 - apiGroups: [""]
 # configmaps is needed for updating logging config
 resources: ["configmaps"]
 verbs: ["get", "list", "watch"]
 # Permissions to create resources in associated TriggerTemplates
 - apiGroups: ["tekton.dev"]
 resources: ["pipelineruns", "pipelineresources", "taskruns"]
 verbs: ["create"]
 - apiGroups: [""]
 resources: ["serviceaccounts"]
 verbs: ["impersonate"]
 - apiGroups: ["policy"]
 resources: ["podsecuritypolicies"]
 resourceNames: ["tekton-triggers"]
 verbs: ["use"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: tekton-triggers-example-binding
 namespace: falcoresponse
subjects:
 - kind: ServiceAccount
 name: tekton-triggers-example-sa
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: tekton-triggers-example-minimal
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: tekton-triggers-example-clusterrole
rules:
 # EventListeners need to be able to fetch any clustertriggerbindings
 - apiGroups: ["triggers.tekton.dev"]
 resources: ["clustertriggerbindings", "clusterinterceptors"]
 verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: tekton-triggers-example-clusterbinding
subjects:
 - kind: ServiceAccount
 name: tekton-triggers-example-sa
 namespace: falcoresponse
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: tekton-triggers-example-clusterrole
EOF

And here is the poddeleter serviceAccount:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
 name: falco-pod-delete
 namespace: falcoresponse
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role
rules:
 - apiGroups: [""]
 resources: ["pods"]
 verbs: ["get", "list", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role-binding
roleRef:
 kind: ClusterRole
 name: falco-pod-delete-cluster-role
 apiGroup: rbac.authorization.k8s.io
subjects:
 - kind: ServiceAccount
 name: falco-pod-delete
 namespace: falcoresponse
EOF

Event listener

Finally time to configure the tekton webhook receiver. Just like rest of Tekton the event listener builds on multiple parts.

cat <<EOF | kubectl apply -f -
apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
 name: falco-listener
 namespace: falcoresponse
spec:
 serviceAccountName: tekton-triggers-example-sa
 triggers:
 - name: cel-trig
 bindings:
 - ref: falco-pod-delete-binding
 template:
 ref: falco-pod-delete-trigger-template
EOF

It is possible to expose a event listener using a ingress, this is a rather normal use case if you want github to trigger a pipeline for example.

I cannot stress this enough DO NOT MAKE THE EVENT LISTENER PUBLIC TO THE INTERNET. We haven't added any protection and this task have the power to kill pods in your cluster. Don't give a potential hacker this power!

The event listener is rather complex and can do allot. For example one way to improve this tekton pipeline could be to check for a specific Priority from Falco. This could be done with a cel interceptor and filter on body.Priority.

But for now let us just trigger on everything.

The triggerBinding let us you define what data should be gathered from the incoming webhook. In this case I take the entire request body.

cat <<EOF | kubectl apply -f -
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
 name: falco-pod-delete-binding
 namespace: falcoresponse
spec:
 params:
 - name: falco-event
 value: \$(body)
EOF

We use the TriggerTemplate to call on the pipeline that we defined earlier using the parameter that the TriggerBinding gives us.

cat <<EOF | kubectl apply -f -
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
 name: falco-pod-delete-trigger-template
 namespace: falcoresponse
 annotations:
 triggers.tekton.dev/old-escape-quotes: "true"
spec:
 params:
 - name: falco-event
 description: The entire msg from falco
 resourcetemplates:
 - apiVersion: tekton.dev/v1beta1
 kind: PipelineRun
 metadata:
 generateName: falco-pod-delete-pipeline-run-
 spec:
 serviceAccountName: falco-pod-delete
 pipelineRef:
 name: pod-delete-pipeline
 params:
 - name: falco-event
 value: \$(tt.params.falco-event)
EOF

Notice the annotations, without it the pipeline will never get triggered.

We define the serviceAccount to use in our pipeline/task, point to the pipeline that we should use. And what parameter to send down to the pipeline, notice the tt in front of parma. This is special syntax for TriggerBindings.

The triggerTemplate was the final piece needed and you should see a pod spinning up in the falcoresponse namespace.

kubectl get pods -n falcoresponse
NAME READY STATUS RESTARTS AGE
el-falco-listener-557786f598-zdmw2 1/1 Running 0 2h

Trigger job

Finally it's time to test our setup.

I would recommend that you start a second terminal for this part.

Terminal 1 follow the falco logs:

kubectl logs -f $(kubectl get pods -l app=falco -o jsonpath="{.items[0].metadata.name}" -n falco) -n falco

Terminal 2 let us trigger the Terminal Shell in container falco rule

# Start a alpine pod
kubectl run alpine --namespace falcoresponse --image=alpine --restart='Never' -- sh -c "sleep 600"
# Trigger the rule breaking behavior
kubectl exec -i --tty alpine --namespace falcoresponse -- sh -c "uptime"
# Watch for pods in falcoresponse namespace
kubectl get pods -n falcoresponse -w

In Terminal 1 you should see something like this:

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.28.0, driver version=5c0b863ddade7a45568c0ac97d037422c9efb750
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to load a system falco module, if present
* Success: falco module found and loaded with modprobe
Sun May 2 18:00:10 2021: Falco version 0.28.0 (driver version 5c0b863ddade7a45568c0ac97d037422c9efb750)
Sun May 2 18:00:10 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Sun May 2 18:00:10 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Sun May 2 18:00:10 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:
Sun May 2 18:00:10 2021: Loading rules from file /etc/falco/rules.d/rules_user_known_k8s_api_callers.yaml:
Sun May 2 18:00:10 2021: Starting internal webserver, listening on port 8765
{"output":"20:24:10.361728219: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=falcoresponse k8s.pod=alpine container=6ac7d190134e shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=6ac7d190134e image=alpine) k8s.ns=falcoresponse k8s.pod=alpine container=6ac7d190134e k8s.ns=falcoresponse k8s.pod=alpine container=6ac7d190134e","priority":"Notice","rule":"Terminal shell in container","time":"2021-05-02T20:24:10.361728219Z", "output_fields": {"container.id":"6ac7d190134e","container.image.repository":"alpine","evt.time":1619987050361728219,"k8s.ns.name":"falcoresponse","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}

In Terminal 2 you should see a pod starting and hopefully Complete without any errors and the alpine pod getting killed.

NAME READY STATUS RESTARTS AGE
alpine 0/1 Terminating 0 1m7s
el-falco-listener-557786f598-znzk9 1/1 Running 0 10m
falco-pod-delete-pipeline-run-w2vf8-run-pod-delete-jlxl7--mk44k 0/1 Completed 0 59s

Hurray our "hacked" pod have been killed

If you look in the logs of the task

kubectl logs -f $(kubectl get pods -l tekton.dev/task=pod-delete -o jsonpath="{.items[0].metadata.name}" -n falcoresponse) -n falcoresponse
2021/05/02 18:11:00 PodName: alpine & Namespace: falcoresponse
2021/05/02 18:11:00 Rule: Terminal shell in container
2021/05/02 18:11:00 Deleting pod alpine from namespace falcoresponse

Conclusion

This was a rather simple example on how we can use the power of tekton together with Falco to protect us from bad actors that is trying to take over pods in our cluster.

As noted during this post there are a lot of potential improvements before this is production ready:

The criticalNamespaces in our go code is currently hard-coded and needs to be input variable of some kind.
We need to be able to delete pods depending on priority level, rule or something similar.
To be able to debug pods we might need to shell in to them, we need a way to ignore pods temporary without the pod getting restarted. Probably a annotation to look for in the pod before deleting it.
And probably many other needs that you can come up with.

If you have any ideas/issues come and share them in the falco slack https://kubernetes.slack.com #falco.

Tekton

If you would like to find out more about Tekton:

Get started in tekton.dev.
Check out the Tekton Project in GitHub.
Meet the maintainers on the TektonCD Slack.
Follow @tektoncd on Twitter.

Falco

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 3: Falcosidekick + Knative

Thu, 13 May 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

As the Cloud Native ecosystem grows and the idea that an integrator can browse the offerings and slap them together like an a la carte menu resonates. We call this Thinking Cloud Native.

Falco already produced events, but in the form of a webhook with bespoke payloads, which is fine, unless you would like to integrate into an ecosystem for event routing. To enable this for Falco we had to think about how these events are moved from producer to consumer via something else. Enter: CloudEvents.

What is CloudEvents? It is a specification for translating an event and the metadata onto a specific protocol and back. What? It lets you think about the event in a generic way without it being tied to particular choices the integration is making today, and with minor effort CloudEvents lets that integration change the protocol choice without changing the meaning of the event.

This lossless property of CloudEvents means the integrator is free to choose middleware that also speaks CloudEvents and has its own choices of persistence and protocol, but the consumer of the event need not be aware of these translations that have happened between the producer and consumer.

There are several choices that support CloudEvents today: Serverless.com Event Gateway, Argo, Google Cloud Pub/Sub, Azure Event Grid, and Knative Eventing. A more full list is at the cloudevents/spec repo.

For this blog post, we are going to focus in on Falco+Knative and see what we can do with that a la carte selection.

Falco+Knative

What is Knative? It is two things: Knative Serving and Knative Eventing. Serving provides a container based scale to zero, scale real big functionality; as well as rainbow deploys, auto-TLS, domain mappings, and various knobs to control concurrency and scale traits. Eventing provides a thin abstraction on top of traditional message brokers (think Kafka or AMQP) that lets you compose your application without considering the message persistence choices in the moment (CloudEvents).

From Knative Eventing, we will use two components: Broker and Trigger. A Knative Eventing Broker represents a event delivery and persistence layer, sort of an eventing mesh. A Knative Eventing Trigger works with the Broker to ask that a consumer be involved with a CloudEvent that matches some specified attributes. So the Broker is the stream of events, the Trigger is how you select events out of the stream and get them delivered.

With Falco producing CloudEvents, we can point our alerts from Falco at the Knative Eventing Broker. Then create a Trigger that selects the Falco event we want to react to. But we also need something to consume the event and react!

From Knative Serving, we can leverage a Knative Serving Service (KService). A KService looks like a lot like a Kubernetes deployment, but it is realized on the cluster as an autoscaling and routable component without the need for manually creating additional Kubernetes Services. KService can run any container as long as it is stateless, and the lifecycle is defined only in the context of an active HTTP request.

To tie this up in a picture,

Falco --[via Sidekick]--> Broker --[via Trigger]--> KService

We are free to make the subscriber of the Trigger be anything we want it to be as long as it is routable from the Broker, and it accepts HTTP POSTs. The request will be a CloudEvent in Binary mode, and Falco makes JSON events, so the payload will be the standard JSON Falco is known for. In-fact, we can replace the KService in with a Kubeless function and it will work.

Demo

To demonstrate this, we have prepared a simple example: We will detect root shell creations and delete that pod.

Prerequisites

K3s Cluster

For this blog post, we a will show the demo using k3s using multipass. Here is a cluster creation commands:

multipass launch --name k3s-leader --cpus 2 --mem 2048M --disk 10G

multipass exec k3s-leader -- /bin/bash -c "curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE=644 sh -"

export K3S_IP_SERVER="https://$(multipass info k3s-leader | grep "IPv4" | awk -F' ' '{print $2}'):6443"

multipass exec k3s-leader -- /bin/bash -c "cat /etc/rancher/k3s/k3s.yaml" | sed "s%https://127.0.0.1:6443%${K3S_IP_SERVER}%g" | sed "s/default/k3s/g" > ./k3s.yaml
export KUBECONFIG=./k3s.yaml

You should get this final output:

(╯°□°)╯︵ multipass launch --name k3s-leader --cpus 2 --mem 2048M --disk 10G
Launched: k3s-leader
(╯°□°)╯︵ multipass exec k3s-leader -- /bin/bash -c "curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE=644 sh -"
[INFO] Finding release for channel stable
[INFO] Using v1.20.6+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.20.6+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.20.6+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s

Now we have a bare bones k3s cluster!

Install Knative

To install the rest of Knative into k3s:

# Installs Knative Serving
kubectl apply -f https://github.com/knative/serving/releases/download/v0.22.0/serving-crds.yaml
kubectl wait --for=condition=Established --all crd
kubectl apply -f https://github.com/knative/serving/releases/download/v0.22.0/serving-core.yaml
kubectl apply -f https://github.com/knative/net-kourier/releases/download/v0.22.0/kourier.yaml
kubectl patch configmap/config-network \
 --namespace knative-serving \
 --type merge \
 --patch '{"data":{"ingress.class":"kourier.ingress.networking.knative.dev"}}'

# Installs Knative Eventing
kubectl apply -f https://github.com/knative/eventing/releases/download/v0.22.0/eventing-crds.yaml
kubectl wait --for=condition=Established --all crd
kubectl apply -f https://github.com/knative/eventing/releases/download/v0.22.0/eventing-core.yaml
kubectl apply -f https://github.com/knative/eventing/releases/download/v0.22.0/in-memory-channel.yaml
kubectl apply -f https://github.com/knative/eventing/releases/download/v0.22.0/mt-channel-broker.yaml

# Creates a default Broker
kubectl create -f - <<EOF
apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
 name: default
 namespace: default
EOF

See also knative.dev install instructions for installing these into your own cluster.

Falco/Falcosidekick/sidekick UI

We'll use helm to install Falco ,Falcosidekick and Falcosidekick UI.

First, add the falcosecurity helm repo:

helm repo add falcosecurity https://falcosecurity.github.io/charts

helm install falco falcosecurity/falco \
 --create-namespace --namespace falco \
 --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true \
 --set falcosidekick.config.cloudevents.address=http://broker-ingress.knative-eventing.svc.cluster.local/default/default

You should get this output:

NAME: falco
LAST DEPLOYED: Thu Jan 14 23:43:46 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.
No further action should be required.

And you can see your new Falco,Falco Sidekick,Falco Sidekick UI pods:

kubectl get pods -n falco
NAME READY STATUS RESTARTS AGE
falco-jh75c 1/1 Running 0 1d
falco-falcosidekick-554b8859d5-v9xkg 1/1 Running 0 1d
falco-falcosidekick-554b8859d5-x2zkk 1/1 Running 0 1d
falco-falcosidekick-ui-5d747688f9-g96x5 1/1 Running 11 1d

The arguments --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true enables Falcosidekick and the UI as the below shows:

You can now test it with a typical port-forwarding:

kubectl port-forward svc/falco-falcosidekick-ui -n falco 2802:2802

Drop demo

Install the demo with:

kubectl apply -f https://github.com/n3wscott/falco-drop/releases/download/v0.1.0/release.yaml

This will install a Knative Service that will consume the Falco events sent by falcosidekick (to the broker), some RBAC to enable that service to delete pods, and a Knative Trigger to register this consumer for events from the Knative Eventing Broker.

Consumer KService

The simplified go code in use is like the following:

func main() {
 ...setup context...
 kc := kubeclient.Get(ctx)

 // Make a CloudEvents Client.
 c, _ := cloudevents.NewDefaultClient(p)

 // StartReceiver is blocking, it will deliver events to the inline function.
 c.StartReceiver(ctx, func(event cloudevents.Event) {
 // Filter based on source and type.
 if event.Source() == "falco.org" && event.Type() == "falco.rule.output.v1" {

 // Extract the Falco event Payload
 payload := &FalcoPayload{}
 if err := event.DataAs(payload); err != nil {
 return
 }

 // Only react to "Terminal shell in container" triggered rules.
 if payload.Rule == "Terminal shell in container" {
 if err := kc.CoreV1().Pods(payload.Fields.Namespace).Delete(ctx, payload.Fields.Pod, metav1.DeleteOptions{}); err != nil {
 log.Println("failed to delete pod from event:", err)
 return
 }
 log.Printf("[%s] deleted %s from %s because %s\n", payload.Rule, payload.Fields.Pod, payload.Fields.Namespace, payload.Output)
 }
 }
 })
}

The full implementation can be found in the falco-drop repo.

Pro-tip: if you are developing in Go for Kubernetes, take a look at ko. ko enables containerizing go applications without needing a Dockerfile.

Even though the Trigger only delivers events that match the Trigger filter, it is a good idea to validate the event that the function is receiving, which is why we are validating again in the above code (trust, but verify).

Eventing Triggers

The Trigger configures the Broker for a subscriber to be invoked when the Broker ingresses an event that matches the spec.filter settings.

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: drop
 namespace: default
spec:
 broker: default
 filter:
 attributes:
 source: falco.org
 type: falco.rule.output.v1
 subscriber:
 ref:
 apiVersion: serving.knative.dev/v1
 kind: Service
 name: drop

Note: the kind: Service, name: drop resource is the Knative Service we created above.

Here we are requesting that the broker only deliver events that have the attributes (CloudEvent attributes) of source=falco.org and type=falco.rule.output.v1. These events are delivered to our subscriber KService.

Want to learn how that spec.subscriber.ref works?! It is duck typing and you can learn more, but tl;dr: it is basically doing this (except fancy),

kubectl get ksvc drop -o jsonpath='{.status.address.url}'

Test

First we will create a pod that we can execute code on later:

kubectl run alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"

You should see two pods runing, drop-00001-* and a alpine:

(╯°□°)╯︵ kubectl get pods
NAME READY STATUS RESTARTS AGE
drop-00001-deployment-6b4c5d8bb-m8q4z 2/2 Running 0 4m9s
alpine 1/1 Running 0 39s

Next, we will execute a command in that alpine pod:

kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"

The alpine pod will be terminated by the drop function once the events are processed:

(╯°□°)╯︵ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"
pod/alpine created
(╯°□°)╯︵ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"
 19:29:29 up 17 min, load average: 0.90, 0.85, 0.59
(╯°□°)╯︵ kubectl get pods
NAME READY STATUS RESTARTS AGE
drop-00001-deployment-6b4c5d8bb-m8q4z 2/2 Running 0 10m
alpine 1/1 Terminating 0 5s

Or simply start a hanging shell:

kubectl run alpine-alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"
kubectl exec -i --tty alpine-hang --namespace default -- sh

And the shell will be closed:

(╯°□°)╯︵ kubectl exec -i --tty alpine-hang --namespace default -- sh
/ # command terminated with exit code 137

The event that the drop function is reacting to is a CloudEvent that looks something like this:

Context Attributes,
specversion: 1.0
type: falco.rule.output.v1
source: falco.org
id: f7628198-3822-4c98-ac3f-71770e272a16
time: 2021-01-11T23:46:19.82302759Z
datacontenttype: application/json
Extensions,
foo: bar
priority: Notice
rule: Terminal shell in container
Data,
{
"output": "23:46:19.823027590: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=f29b261f8831 shell=bash parent=runc cmdline=bash -il terminal=34816 container_id=f29b261f8831 image=mysql) k8s.ns=default k8s.pod=mysql-db-7d59548d75-wh44s container=f29b261f8831",
"priority": "Notice",
"rule": "Terminal shell in container",
"time": "2021-01-11T23:46:19.82302759Z",
"output_fields": {
"container.id": "f29b261f8831",
"container.image.repository": "mysql",
"evt.time": 1610408779823027700,
"k8s.ns.name": "default",
"k8s.pod.name": "alpine",
"proc.cmdline": "bash -il",
"proc.name": "bash",
"proc.pname": "runc",
"proc.tty": 34816,
"user.loginuid": -1,
"user.name": "root"
}
}

The KService consumes this event and simply deletes the pod. You can also see this activity in the falcosidekick UI.

Conclusion

Thinking Cloud Native is a mindset of picking the right tool for the job and assembling these tools into something greater than their parts. Falco is a great tool for detection and alerts, it gets really interesting once we can react to those events in ways we never imagined, because integrators are creative and innovative.

What will you build?

Knative

If you would like to find out more about Knative:

Get started in knative.dev.
Check out the Knative Project in GitHub.
Meet the maintainers on the Knative Slack.
Follow @KnativeProject on Twitter.

Falco

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 2: Falcosidekick + OpenFaas

Sun, 11 Apr 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

We recently talked about a concept called "Kubernetes Response Engine", and we achieved this by using Falco

Falcosidekick + Kubeless. But as you might guess, Falcosidekick project is evolving day after day, which means new outputs are added. With the release 2.22.0, we are proud to support OpenFaaS as a new output for Falcosidekick. This allows us to achieve the same concept, "Kubernetes Response Engine", but this time by using "OpenFaaS" instead of "Kubeless".

In this blog post, we will explain the basic concepts for integrating your own Response Engine into K8S with the stack Falco + Falcosidekick + OpenFaaS.

Prerequisites

We need tools with the following minimum versions to achieve this demo:

Minikube v1.19.0
Helm v3.5.3
kubectl v1.21.0
arkade v0.7.13
faas-cli v0.13.9

Provision local Kubernetes Cluster

There are various ways to provision a local Kubernetes cluster such as, KinD, k3s, k0s, Minikube etc. We are going to use Minikube in this walkthrough.

Let's get provisioned a local Kubernetes cluster:

$ minikube start --cpus 3 --memory 8192 --vm-driver virtualbox
😄 minikube v1.19.0 on Darwin 10.15.7
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating virtualbox VM (CPUs=3, Memory=8192MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.4 ...
 ▪ Generating certificates and keys ...
 ▪ Booting up control plane ...
 ▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
 ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Install OpenFaaS

OpenFaaS can be deployed into a variety of container orchestrators like Kubernetes, OpenShift, Docker Swarm or into a single host with faasd.

Follow the official documentation for deploying OpenFaaS to Kubernetes.

The fastest option is the tool called arkade to deploy OpenFaaS:

$ arkade install openfaas
Using Kubeconfig: /Users/batuhan.apaydin/.kube/config
Client: x86_64, Darwin
2021/04/10 21:39:29 User dir established as: /Users/batuhan.apaydin/.arkade/
"openfaas" already exists with the same configuration, skipping

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
...Successfully got an update from the "openfaas" chart repository
Update Complete. ⎈Happy Helming!⎈

VALUES values.yaml
Command: /Users/batuhan.apaydin/.arkade/bin/helm [upgrade --install openfaas openfaas/openfaas --namespace openfaas --values /var/folders/pf/6h9t0mnd4d342ncgpjq_3zl80000gp/T/charts/openfaas/values.yaml --set queueWorker.replicas=1 --set queueWorker.maxInflight=1 --set clusterRole=false --set operator.create=false --set faasnetes.imagePullPolicy=Always --set basicAuthPlugin.replicas=1 --set gateway.replicas=1 --set gateway.directFunctions=false --set openfaasImagePullPolicy=IfNotPresent --set ingressOperator.create=false --set basic_auth=true --set serviceType=NodePort]
Release "openfaas" does not exist. Installing it now.
NAME: openfaas
LAST DEPLOYED: Sat Apr 10 21:39:37 2021
NAMESPACE: openfaas
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that openfaas has started, run:

 kubectl -n openfaas get deployments -l "release=openfaas, app=openfaas"
=======================================================================
= OpenFaaS has been installed. =
=======================================================================

# Get the faas-cli
curl -SLsf https://cli.openfaas.com | sudo sh

# Forward the gateway to your machine
kubectl rollout status -n openfaas deploy/gateway
kubectl port-forward -n openfaas svc/gateway 8080:8080 &

# If basic auth is enabled, you can now log into your gateway:
PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
echo -n $PASSWORD | faas-cli login --username admin --password-stdin

faas-cli store deploy figlet
faas-cli list

# For Raspberry Pi
faas-cli store list \
 --platform armhf

faas-cli store deploy figlet \
 --platform armhf

# Find out more at:
# https://github.com/openfaas/faas

Thanks for using arkade!

Check if everything is working before moving onto the next step:

$ kubectl get pods --namespace openfaas
NAME READY STATUS RESTARTS AGE
alertmanager-74f9b48464-7gvrj 1/1 Running 0 2m13s
basic-auth-plugin-54bbd886f5-fclgn 1/1 Running 0 2m13s
gateway-6f8f5d5c87-tbxns 2/2 Running 0 2m13s
nats-695bf7587-hcbc2 1/1 Running 0 2m13s
prometheus-577c65f58c-4nvm7 1/1 Running 0 2m13s
queue-worker-b45b85966-g7kpt 1/1 Running 0 2m13s

Now, it is time to deploy our function. The function we are going to deploy basically receives events for an infected pod from the Falcosidekick and deletes it immediately. Before deploying the function we need some permissions to delete Pod. We create a ServiceAccount with right to delete a Pod in any namespace, and we'll associate it to our function:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
 name: falco-pod-delete
 namespace: openfaas-fn
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role
rules:
 - apiGroups: [""]
 resources: ["pods"]
 verbs: ["get", "list", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role-binding
roleRef:
 kind: ClusterRole
 name: falco-pod-delete-cluster-role
 apiGroup: rbac.authorization.k8s.io
subjects:
 - kind: ServiceAccount
 name: falco-pod-delete
 namespace: openfaas-fn
EOF

Now, we are ready to deploy our falco-pod-delete function, log in into OpenFaaS Gateway first:

$ kubectl port-forward -n openfaas svc/gateway 8080:8080 &
$ PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
$echo -n $PASSWORD | faas-cli login --username admin --password-stdin
Calling the OpenFaaS server to validate the credentials...
credentials saved for admin http://127.0.0.1:8080

Install Falco + Falcosidekick

Firstly, we'll create the namespace that will host both Falco and Falcosidekick:

$ kubectl create namespace falco

We add the helm repo:

$ helm repo add falcosecurity https://falcosecurity.github.io/charts
"falcosecurity" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
...Successfully got an update from the "openfaas" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm upgrade --install falco falcosecurity/falco --namespace falco \
--set falcosidekick.enabled=true \
--set falcosidekick.config.openfaas.functionname="falco-pod-delete"

You should get this output:

Release "falco" does not exist. Installing it now.
NAME: falco
LAST DEPLOYED: Tue Apr 13 10:49:49 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.


No further action should be required.

And you can see your new Falco and Falcosidekickpods:

$ kubectl get pods --namespace falco
NAME READY STATUS RESTARTS AGE
falco-falcosidekick-7779579477-mwsb4 1/1 Running 0 67s
falco-falcosidekick-7779579477-n5v89 1/1 Running 0 67s
falco-p97rw 1/1 Running 0 67s

The argument falcosidekick.enabled=true sets the following settings in Falco for you:

--set falco.jsonOutput=true \
--set falco.httpOutput.enabled=true \
--set falco.httpOutput.url=http://falco-falcosidekick:2801

We check the logs:

$ kubectl logs deployment/falco-falcosidekick -n falco
Found 2 pods, using pod/falcosidekick-5c696d7fd8-9bnnj
2021/04/10 19:21:55 [INFO] : Enabled Outputs : [OpenFaaS]
2021/04/10 19:21:55 [INFO] : Falco Sidekick is up and listening on :2801

OpenFaaS is displayed as enabled output, everything is good 👍.

Install our OpenFaaS function

Our really basic function will receive events from Falco thanks to Falcosidekick, check if the triggered rule is * Terminal Shell in container* , extract the namespace and pod name from the fields of events and delete the according pod:

Basically, the process is:

 +----------+ +---------------+ +----------+
 | Falco +-----------------> Falcosidekick +--------------------> OpenFaaS |
 +----^-----+ sends event +---------------+ triggers +-----+----+
 | |
detects a shell | |
 | |
 +----+-------+ deletes |
 | Pwned Pod <----------------------------------------------------------+
 +------------+

Let's create the function and deploy it:

$ faas-cli template store pull golang-middleware
Fetch templates from repository: https://github.com/openfaas/golang-http-template at
2021/04/10 21:56:34 Attempting to expand templates from https://github.com/openfaas/golang-http-template
2021/04/10 21:56:35 Fetched 2 template(s) : [golang-http golang-middleware] from https://github.com/openfaas/golang-http-template
$ tree -L 2 .
.
└── template
├── golang-http
└── golang-middleware
# Don't forget to set your docker id in the prefix section, mine is devopps.
$ faas-cli new falco-pod-delete --lang golang-middleware --prefix devopps
faas-cli new falco-pod-delete --lang golang-middleware --prefix devopps
Folder: falco-pod-delete created.
___ _____ ____
/ _ \ _ __ ___ _ __ | ___|_ _ __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) | __/ | | | _| (_| | (_| |___) |
\___/| .__/ \___|_| |_|_| \__,_|\__,_|____/
|_|
Function created in folder: falco-pod-delete
Stack file written: falco-pod-delete.yml
Notes:
You have created a new function which uses Golang 1.13.
To include third-party dependencies, use Go modules and use
"--build-arg GO111MODULE=on" with faas-cli build or configure this
via your stack.yml file.
See more: https://docs.openfaas.com/cli/templates/
For detailed examples:
https://github.com/openfaas-incubator/golang-http-template
$ tree -L 2 .
.
├── falco-pod-delete
│ └── handler.go
├── falco-pod-delete.yml
└── template
├── golang-http
└── golang-middleware

First, replace the falco-pod-delete.yml with the following content:

version: 1.0
provider:
 name: openfaas
 gateway: http://127.0.0.1:8080
functions:
 falco-pod-delete:
 lang: golang-middleware
 handler: ./falco-pod-delete
 image: devopps/falco-pod-delete:latest # be careful this line, it should be your docker id.
 annotations:
 com.openfaas.serviceaccount: falco-pod-delete
 build_args:
 GO111MODULE: on

Once you have edited it, let's continue with the code, create a go.mod.

$ cd falco-pod-delete
$ go mod init falco-pod-delete
go: creating new go.mod: module falco-pod-delete
go: to add module requirements and sums:
go mod tidy

Then, replace the handler.go with the following content:

package function

import (
 "context"
 "encoding/json"
 "io/ioutil"
 "log"
 "net/http"
 "time"

 metaV1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/kubernetes"
 "k8s.io/client-go/rest"
)

var kubeClient *kubernetes.Clientset

func init() {

 // creates the in-cluster config
 config, err := rest.InClusterConfig()
 if err != nil {
 panic(err.Error())
 }
 // creates the clientset
 kubeClient, err = kubernetes.NewForConfig(config)
 if err != nil {
 panic(err.Error())
 }
}

type Alert struct {
 Output string `json:"output"`
 Priority string `json:"priority"`
 Rule string `json:"rule"`
 Time time.Time `json:"time"`
 OutputFields struct {
 ContainerID string `json:"container.id"`
 ContainerImageRepository interface{} `json:"container.image.repository"`
 ContainerImageTag interface{} `json:"container.image.tag"`
 EvtTime int64 `json:"evt.time"`
 FdName string `json:"fd.name"`
 K8SNsName string `json:"k8s.ns.name"`
 K8SPodName string `json:"k8s.pod.name"`
 ProcCmdline string `json:"proc.cmdline"`
 } `json:"output_fields"`
}

var CriticalNamespaces = []string{"kube-system", "kube-public", "kube-node-lease", "falco", "openfaas", "openfaas-fn"}

func Handle(w http.ResponseWriter, r *http.Request) {
 var alert Alert

 if r.Body != nil {
 defer r.Body.Close()

 body, _ := ioutil.ReadAll(r.Body)

 json.Unmarshal(body, &alert)

 podName := alert.OutputFields.K8SPodName
 namespace := alert.OutputFields.K8SNsName

 var critical bool
 for _ , ns := range CriticalNamespaces {
 if ns == namespace {
 critical = true
 break
 }
 }

 if !critical {
 log.Printf("Deleting pod %s from namespace %s", podName, namespace)
 kubeClient.CoreV1().Pods(namespace).Delete(context.Background(), podName, metaV1.DeleteOptions{})
 }
 }

 w.WriteHeader(http.StatusOK)
 w.Write([]byte("OK"))
}

After that, update your Go Modules by doing go mod tidy:

$ go mod tidy
go: finding module for package k8s.io/client-go/rest
go: finding module for package k8s.io/apimachinery/pkg/apis/meta/v1
go: finding module for package k8s.io/client-go/kubernetes
go: downloading k8s.io/client-go v0.21.0
go: downloading k8s.io/apimachinery v0.21.0
go: found k8s.io/apimachinery/pkg/apis/meta/v1 in k8s.io/apimachinery v0.21.0
go: found k8s.io/client-go/kubernetes in k8s.io/client-go v0.21.0
go: found k8s.io/client-go/rest in k8s.io/client-go v0.21.0
go: downloading k8s.io/api v0.21.0
go: downloading golang.org/x/net v0.0.0-20210224082022-3d97a244fca7
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.1.0
go: downloading golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d

Now, you should be able to build, push and deploy your function with faas-cli:

$ cd ..
$ faas-cli up -f falco-pod-delete.yml
[0] > Building falco-pod-delete.
Clearing temporary build folder: ./build/falco-pod-delete/
Preparing: ./falco-pod-delete/ build/falco-pod-delete/function
Building: devopps/falco-pod-delete:latest with golang-middleware template. Please wait..
#1 [internal] load build definition from Dockerfile
#1 sha256:8cd765381aabb90df3bcfbc06f4d175af37d66b85125d463585abc1fc878b94b
#1 transferring dockerfile: 1.81kB done
#1 DONE 0.0s
...

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Image: devopps/falco-pod-delete:latest built.
[0] < Building falco-pod-delete done in 1.31s.
[0] Worker done.

Total build time: 1.31s

[0] > Pushing falco-pod-delete [devopps/falco-pod-delete:latest].
The push refers to repository [docker.io/devopps/falco-pod-delete]
8096edd09fbc: Layer already exists
464d68aca3d9: Layer already exists
e4766ea46ad0: Layer already exists
5f70bf18a086: Layer already exists
a823d50a5b72: Layer already exists
060f21486264: Layer already exists
8ea3b23f387b: Layer already exists
latest: digest: sha256:f94abba203232b97cb2873ef5d60eec31b52d492f3d3ee106d6a9877bf131d95 size: 1782
[0] < Pushing falco-pod-delete [devopps/falco-pod-delete:latest] done.
[0] Worker done.

Deploying: falco-pod-delete.

Deployed. 202 Accepted.
URL: http://127.0.0.1:8080/function/falco-pod-delete

Check if everything is working before moving to the next step:

$ kubectl get pods --namespace openfaas-fn
NAME READY STATUS RESTARTS AGE
falco-pod-delete-7dc9f5fbb8-gbfk7 1/1 Running 0 27s

Test our function

We start by creating a dumb pod:

$ kubectl run alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"
pod/alpine created

$ kubectl get pods --namespace default
AME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 11s

Let's run a shell command inside and see what happens:

$ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"

19:27:21 up 50 min, load average: 0.11, 0.12, 0.11

As expected we got the result of our command, but, if we get the status of the pod we retrieve:

$ kubectl get pods --namespace default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 103s

💥 It has been terminated 💥

We can now check the logs of components.

For Falco:

$ kubectl logs daemonset/falco --namespace falco
..
{"output":"19:27:21.002873265: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=97c9868ea832 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=97c9868ea832 image=alpine) k8s.ns=default k8s.pod=alpine container=97c9868ea832","priority":"Notice","rule":"Terminal shell in container","time":"2021-04-10T19:27:21.002873265Z", "output_fields": {"container.id":"97c9868ea832","container.image.repository":"alpine","evt.time":1618082841002873265,"k8s.ns.name":"default","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}
{"output":"19:27:21.038853452: Notice Unexpected connection to K8s API Server from container (command=handler k8s.ns=openfaas-fn k8s.pod=falco-pod-delete-7dc9f5fbb8-gbfk7 container=12fc4de5ccc3 image=devopps/falco-pod-delete:latest connection=172.17.0.9:43812->10.96.0.1:443) k8s.ns=openfaas-fn k8s.pod=falco-pod-delete-7dc9f5fbb8-gbfk7 container=12fc4de5ccc3","priority":"Notice","rule":"Contact K8S API Server From Container","time":"2021-04-10T19:27:21.038853452Z", "output_fields": {"container.id":"12fc4de5ccc3","container.image.repository":"devopps/falco-pod-delete","container.image.tag":"latest","evt.time":1618082841038853452,"fd.name":"172.17.0.9:43812->10.96.0.1:443","k8s.ns.name":"openfaas-fn","k8s.pod.name":"falco-pod-delete-7dc9f5fbb8-gbfk7","proc.cmdline":"handler"}}

For Falcosidekick:

$ kubectl logs deployment/falcosidekick --namespace falco
2021/04/10 19:27:21 [INFO] : OpenFaas - Post OK (200)
2021/04/10 19:27:21 [INFO] : OpenFaas - Function Response : OK
2021/04/10 19:27:21 [INFO] : OpenFaas - Call Function "falco-pod-delete.openfaas-fn" OK
..

For falco-delete-pod function:

$ faas-cli logs -f --name falco-pod-delete
2021/04/10 19:34:03 Deleting pod alpine from namespace default
2021/04/10 19:34:03 POST / - 200 OK - ContentLength: 2

Conclusion

With this really simple example, we only scratched the surface of possibilities, so don't hesitate to share with us on Slack (https://kubernetes.slack.com #falco) your comments, ideas and successes. You're also always welcome to contribute.

Blog: Kubernetes Response Engine, Part 1: Falcosidekick + Kubeless

Fri, 15 Jan 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

Two years ago, we presented to you a Kubernetes Response Engine based on Falco. The idea was to trigger Kubeless serverless functions for deleting infected pod, start a Sysdig capture or forward the events to GCP PubSub. See the README.

To avoid maintaining this custom stack, we worked hard with the community to integrate all components into Falcosidekick and to improve the UX. With the last release 2.20.0 we have the finale piece, the integration of Kubeless as native output. More details in our retrospective of 2020.

In this blog post, we will explain the basic concepts for integrating your own Response Engine into K8S with the stack Falco + Falcosidekick + Kubeless.

Requirements

We require a kubernetes cluster running at least 1.17 release and helm and kubectl installed.

Install Kubeless

Follow the official quick start page:

export RELEASE=$(curl -s https://api.github.com/repos/kubeless/kubeless/releases/latest | grep tag_name | cut -d '"' -f 4)
kubectl create ns kubeless
kubectl create -f https://github.com/kubeless/kubeless/releases/download/$RELEASE/kubeless-$RELEASE.yaml

After a few seconds, we can check that the controller is up and running:

kubectl get pods -n kubeless
NAME READY STATUS RESTARTS AGE
kubeless-controller-manager-99459cb67-tb99d 3/3 Running 3 2m34s

Install Falco

Firstly, we'll create the namespace that will use both Falco and Falcosidekick:

kubectl create ns falco

Add the helm repo:

helm repo add falcosecurity https://falcosecurity.github.io/charts

In a real project, you should get the whole chart with helm pull falcosecurity/falco --untar and then configure the values.yaml. For this tutorial, we will try to keep things as easy as possible and set configs directly by helm install command:

helm install falco falcosecurity/falco --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falcosidekick:2801 -n falco

You should get this output:

NAME: falco
LAST DEPLOYED: Thu Jan 14 23:43:46 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.
No further action should be required.

And you can see your new Falco pods:

kubectl get pods -n falco
NAME READY STATUS RESTARTS AGE
falco-ctmzg 1/1 Running 0 111s
falco-sfnn8 1/1 Running 0 111s
falco-rrg28 1/1 Running 0 111s

The arguments --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falcosidekick:2801 are there configuring the format of events and the URL where Falco will send them. As Falco and Falcosidekick will be in the same namespace, we can directly use the name of the service (falcosidekick).

Install Falcosidekick

The process is quite the same:

helm install falcosidekick falcosecurity/falcosidekick --set config.kubeless.namespace=kubeless --set config.kubeless.function=delete-pod -n falco

You should get this output:

NAME: falcosidekick
LAST DEPLOYED: Thu Jan 14 23:55:12 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
 export POD_NAME=$(kubectl get pods --namespace falco -l "app.kubernetes.io/name=falcosidekick,app.kubernetes.io/instance=falcosidekick" -o jsonpath="{.items[0].metadata.name}")
 kubectl port-forward $POD_NAME 2801:2801
 echo "Visit http://127.0.0.1:2801 to use your application"

We check the logs:

kubectl logs deployment/falcosidekick -n falco
2021/01/14 22:55:31 [INFO] : Enabled Outputs : Kubeless
2021/01/14 22:55:31 [INFO] : Falco Sidekick is up and listening on port 2801

Kubeless is displayed as enabled output, everything is good 👍.

That's it, we really tried to get a nice UX 😉.

Install our Kubeless function

We'll not explain how to write or how to work Kubeless functions, please read the official docs for more information.

Our basic function will receive events from Falco, thanks to Falcosidekick. Check if the triggered rule is Terminal Shell in container. See rule, extract the namespace and pod name from fields of events, and delete the according pod:

from kubernetes import client,config

config.load_incluster_config()

def delete_pod(event, context):
 rule = event['data']['rule'] or None
 output_fields = event['data']['output_fields'] or None

 if rule and rule == "Terminal shell in container" and output_fields:
 if output_fields['k8s.ns.name'] and output_fields['k8s.pod.name']:
 pod = output_fields['k8s.pod.name']
 namespace = output_fields['k8s.ns.name']
 print (f"Deleting pod \"{pod}\" in namespace \"{namespace}\"")
 client.CoreV1Api().delete_namespaced_pod(name=pod, namespace=namespace, body=client.V1DeleteOptions())

Basically, the process is:

 +----------+ +---------------+ +----------+
 | Falco +-----------------> Falcosidekick +--------------------> Kubeless |
 +----^-----+ sends event +---------------+ triggers +-----+----+
 | |
detects a shell | |
 | |
 +----+-------+ deletes |
 | Pwned Pod <----------------------------------------------------------+
 +------------+

Before deploying our function, we need to create a ServiceAccount for it, as it will need the right to delete a pod in any namespace:

cat <<EOF | kubectl apply -n kubeless -f -
apiVersion: v1
kind: ServiceAccount
metadata:
 name: falco-pod-delete
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role
rules:
 - apiGroups: [""]
 resources: ["pods"]
 verbs: ["get", "list", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role-binding
roleRef:
 kind: ClusterRole
 name: falco-pod-delete-cluster-role
 apiGroup: rbac.authorization.k8s.io
subjects:
 - kind: ServiceAccount
 name: falco-pod-delete
 namespace: kubeless
EOF

namespace: kubelessetetion.k8s.io
serviceaccount/falco-pod-delete created
clusterrole.rbac.authorization.k8s.io/falco-pod-delete-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/falco-pod-delete-cluster-role-binding created

Only remains the installation of our function itself:

cat <<EOF | kubectl apply -n kubeless -f -
apiVersion: kubeless.io/v1beta1
kind: Function
metadata:
 finalizers:
 - kubeless.io/function
 generation: 1
 labels:
 created-by: kubeless
 function: delete-pod
 name: delete-pod
spec:
 checksum: sha256:a68bf570ea30e578e392eab18ca70dbece27bce850a8dbef2586eff55c5c7aa0
 deps: |
 kubernetes>=12.0.1
 function-content-type: text
 function: |-
 from kubernetes import client,config

 config.load_incluster_config()

 def delete_pod(event, context):
 rule = event['data']['rule'] or None
 output_fields = event['data']['output_fields'] or None

 if rule and rule == "Terminal shell in container" and output_fields:
 if output_fields['k8s.ns.name'] and output_fields['k8s.pod.name']:
 pod = output_fields['k8s.pod.name']
 namespace = output_fields['k8s.ns.name']
 print (f"Deleting pod \"{pod}\" in namespace \"{namespace}\"")
 client.CoreV1Api().delete_namespaced_pod(name=pod, namespace=namespace, body=client.V1DeleteOptions())
 handler: delete-pod.delete_pod
 runtime: python3.7
 deployment:
 spec:
 template:
 spec:
 serviceAccountName: falco-pod-delete
EOF

function.kubeless.io/delete-pod created

kubectl get pods -n kubeless

NAME READY STATUS RESTARTS AGE
kubeless-controller-manager-99459cb67-tb99d 3/3 Running 3 3d14h
delete-pod-d6f98f6dd-cw228 1/1 Running 0 2m52s

kubectl get svc -n kubeless

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
delete-pod ClusterIP 10.43.211.201 <none> 8080/TCP 4m38s

Test our function

We start by creating a dumb pod:

kubectl run alpine -n default --image=alpine --restart='Never' -- sh -c "sleep 600"

kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 9s

Let's run a shell command inside and see what happens:

kubectl exec -i --tty alpine -n default -- sh -c "uptime"

23:44:25 up 1 day, 19:11, load average: 0.87, 0.77, 0.77

As expected, we got the result of our command, but, to get the status of the pod now:

kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 103s

💥 It has been terminated 💥

We can now check the logs of components.

For Falco:

kubectl logs daemonset/falco -n falco

{"output":"23:39:44.834631763: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=5892b41bcf46 shell=sh parent=<NA> cmdline=sh terminal=34817 container_id=5892b41bcf46 image=<NA>) k8s.ns=default k8s.pod=alpine container=5892b41bcf46","priority":"Notice","rule":"Terminal shell in container","time":"2021-01-14T23:39:44.834631763Z", "output_fields": {"container.id":"5892b41bcf46","container.image.repository":null,"evt.time":1610667584834631763,"k8s.ns.name":"default","k8s.pod.name":"alpine","proc.cmdline":"sh","proc.name":"sh","proc.pname":null,"proc.tty":34817,"user.loginuid":-1,"user.name":"root"}}

For Falcosidekick:

kubectl logs deployment/falcosidekick -n falco

2021/01/14 23:39:45 [INFO] : Kubeless - Post OK (200)
2021/01/14 23:39:45 [INFO] : Kubeless - Function Response :
2021/01/14 23:39:45 [INFO] : Kubeless - Call Function "delete-pod" OK

(Notice, the function returns nothing, this is why the message log is empty)

For delete-pod function:

kubectl logs deployment/delete-pod -n kubeless

10.42.0.31 - - [14/Jan/2021:23:39:45 +0000] "POST / HTTP/1.1" 200 0 "" "Falcosidekick" 0/965744
Deleting pod "alpine" in namespace "default"

Conclusion

With this simple example, we only scratched the surface of possibilities. Everything is possible now, so don't hesitate to share with us on Slack (https://kubernetes.slack.com #falco) your comments, ideas and successes. You're always welcome to contribute.

Bonus: You're running Falcosidekick outside Kubernetes but still want to use the Kubeless output? No problem, you can declare a kubeconfig file to use. See README.

Bonus 2: For people who wants to use Knative in place of Kubeless, it's coming soon 😉

Enjoy

Blog: Falcosidekick 2020

Tue, 12 Jan 2021 00:00:00 +0000

This fantastic post from @leodido about how has been the previous year 2020 for falco inspired me (link) I wanted to bring everyone up to speed on what we built for falcosidekick in 2020

Aside a lot of improvments and bug fixes, 8 new outputs have been integrated:

Rocketchat
Mattermost
Azure Event Hub
Discord
AWS SNS
GCP PubSub
Cloudwatch Logs
Apache Kafka

What really changed with previous releases was that almost all these outputs have been proposed and developed by other members of the falco community (kindly called the famiglia 😉). It warms my ♥️ a lot and makes me learn a lot about how to manage an open source project.

Thanks to everybody for your ideas, your comments, your help, your PR, your reviews, etc.

The following chart shows well how things are getting bigger and bigger for this small project that finally appeared useful for some people and companies.

A special 🙏 to @cpanato, @KeisukeYamashita and @nibalizer, who are now official maintainers of falcosidekick with me. 🎉 to them!

Last but not least, all my friendship to @danpopSD for his support and motivation. Merci mon ami.

What's next?

Release 2.20.0

Few times before this article is out we released one of the biggest versions since the beginning of falcosidekick. It results of a combination of a lot of efforts from many people.

The full changelog can be found here.

The main changes are three new outputs:

STAN (NATS Streaming)
PagerDuty
Kubeless (stay tuned, a post about this will be out soon 😉)

And ?

We believe the duo of falco + falcosidekick to be an obvious solution for most infrastructures, we are working hard to improve the code base and documentation. That will be all the major set of goals for the next major release 3.0.0 which is coming in the next few months. Until then with the help of n3wscott, we're working on adding the Cloudevents spec in a new HTTP output that will able to forward falco's events to more backends, like Knative.

Enjoy