Falco – Integration

Blog: Manage Falco easier with Giant Swarm App Platform

Tue, 09 Aug 2022 00:00:00 +0000

In this article, you will learn how Giant Swarm simplifies the maintenance of the software stack within Kubernetes clusters by using its App Platform technology. Additionally, we will show how customers can leverage this to easily deploy Falco, either individually or as part of Giant Swarm's Security Pack, to secure their managed Kubernetes service.

Giant Swarm

Having CoreOS, Fleet, and Docker as base technologies, Giant Swarm was founded in 2014. In 2016, it chose Kubernetes to reinvent itself. And just a year later, in 2017, it became part of the founding members of the Kubernetes Certified Service Providers. Customers like Adidas or Vodafone backup a company that, supported by a fully remote team, has been able to foresee the trends of technology and working lifestyle.

As a managed Kubernetes company, its services and infrastructure enable enterprises to run resilient distributed systems at scale while removing the burden of Day 2 operations. Giant Swarm takes pride in delivering a fully open source platform that's carefully curated and opinionated.

Security and simplicity

Giant Swarm takes security as seriously as ease of management. Hence, when using a managed Kubernetes platform, everything that happens on the management cluster is as important as the performance of the workload cluster itself.

That's why, leveraging the concept of operators to control all resources that clusters need as 'Custom Resources', Giant Swarm can deploy and update its management clusters in the quickest possible way. Needless to say, this is exactly what Giant Swarm offers to its customers to manage their applications.

Falco, the Runtime Security Project

Falco is the de facto Kubernetes threat detection engine, and also extends its reach to cloud and Linux hosts. It monitors the behavior of every process in the node and can alert us when something fishy happens.

How does Falco do that? Based on a set of rules that Falco interprets at startup time, it waits for events and syscalls that would trigger one of those rules. When a rule is triggered, Falco raises an alert and, thanks to applications like Falco Sidekick, allows teams to react accordingly.

But with great power comes great responsibility. What happens when we start getting false positives our Falco rules haven't been updated for some months, or our Falco daemon is a few versions behind? The answer is as simple as updating. Well, maybe not that simple if we are responsible for tens of clusters with hundreds of nodes.

Giant Swarm App Platform

Giant Swarm describes App Platform as a set of features that allow you to browse, install, and manage the configurations of managed apps from a single place: The management cluster.

The technology behind it is simple: Apps are packaged as Helm charts, can be configured with values, overridden with a different app configuration, etc. - whatever meets your needs. To deploy, a CRD (Custom Resource Definition) resource is created, interpreted by the App Operator (running on the managed cluster), assigned to the Chart Operator (running on the workload cluster), and in a few seconds, our application will be deployed on as many clusters as desired.

The App Platform offers its repertoire of applications from the App Catalog. Giant Swarm offers two App Catalogs out of the box: The Giant Swarm Catalog and the Giant Swarm Playground. But what we love the most from the App Platform is that we can have our additional catalogs, storing our applications and configurations.

What does it look like on the CLI?

It's now time to look at App Platform running. Let's walk through its deployment on a minikube cluster. Following these instructions, it shouldn't take too long until we are ready to deploy our first managed app, Falco, using a single CRD.

To keep this as standard as possible, we'll even go through some steps to compile some interesting Giant Swarm tools, like the plugin kubectl-gs.

Do you already have a Kubernetes cluster nearby?

If not, we can spin up a minikube instance pretty quickly.

$ minikube start --driver virtualbox
😄 minikube v1.25.1 on Darwin 11.6.6
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating virtualbox VM (CPUs=2, Memory=6000MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.23.1 on Docker 20.10.12 ...
▪ kubelet.housekeeping-interval=5m
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

If you don't have kubectl installed or your system, the easiest way to access it would be through an alias to minikube kubectl, like this:

alias kubectl="minikube kubectl --"

Don't forget the -- at the end. That tells the command prompt not to pass any added parameters to minikube, since we need them to be understood by kubectl.

One disadvantage of this method, in comparison to having a local kubectl binary, is that the kubectl-gs plugin might not work when called as kubectl gs (explained later during this tutorial) so you might need to call it directly.

To ensure our cluster is up and running, execute the following command and verify that all nodes, pods, and containers are up and ready:

$ kubectl get nodes,ns,pods -A
NAME STATUS ROLES AGE VERSION
node/minikube Ready control-plane,master 4m16s v1.23.1
NAME STATUS AGE
namespace/default Active 4m14s
namespace/kube-node-lease Active 4m15s
namespace/kube-public Active 4m15s
namespace/kube-system Active 4m16s
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system pod/coredns-64897985d-qbf4n 1/1 Running 0 4m
kube-system pod/etcd-minikube 1/1 Running 0 4m12s
kube-system pod/kube-apiserver-minikube 1/1 Running 0 4m12s
kube-system pod/kube-controller-manager-minikube 1/1 Running 0 4m12s
kube-system pod/kube-proxy-6ds89 1/1 Running 0 4m
kube-system pod/kube-scheduler-minikube 1/1 Running 0 4m14s
kube-system pod/storage-provisioner 1/1 Running 1 (3m29s ago) 4m10s

Prerequisites: Compiling `apptestctl` and `kubectl-gs`

As mentioned above, we'll compile a couple of tools. The first one will be apptestctl. This tool will help us bootstrap App Platform on a cluster not managed by Giant Swarm.

To do this, we'll use the docker.io/golang:1.17 image.

The following command will prepare an available instance of a Golang compiler for us to compile both of these tools:

$ kubectl run golang --image docker.io/golang:1.17 -- sleep infinity
pod/golang created

Compiling `apptestctl`

These steps are quite simple: clone the apptestctl repository and compile it as indicated.

We'll do this inside the container we created in the previous step so we don't pollute our system.

$ kubectl exec -it golang -- git clone https://github.com/giantswarm/apptestctl src/apptestctl
Cloning into 'apptestctl'...
... output omitted ...
Resolving deltas: 100% (791/791), done.
$ kubectl exec -it golang -- make -C src/apptestctl
make: Entering directory '/go/src/apptestctl'
... output omitted ...
====> apptestctl-v-linux-amd64
... output omitted ...
cp -a apptestctl-v-linux-amd64 apptestctl
====> build
make: Leaving directory '/go/src/apptestctl'

Alternatively, you can build a Darwin client.

$ kubectl exec -it golang -- make build-darwin -C src/apptestctl
make: Entering directory '/go/src/apptestctl'
... output omitted ...
====> apptestctl-v-darwin-amd64
... output omitted ...
cp -a apptestctl-v-darwin-amd64 apptestctl-darwin
====> build-darwin
make: Leaving directory '/go/src/apptestctl'

Either way, you can copy the apptestctl binary to your system and use it from wherever you prefer.

$ kubectl cp golang:/go/src/apptestctl/apptestctl-darwin ./apptestctl
$ kubectl chmod u+x ./apptestctl

Compiling `kubectl-gs`

Use the same steps to compile the kubectl-gs plugin this time, which will allow us to interact with App Platform. Pay attention to the fact that we'll compile it just for Darwin in this instance.

$ kubectl exec -it golang -- git clone https://github.com/giantswarm/kubectl-gs src/kubectl-gs
Cloning into 'kubectl-gs'...
... output omitted ...
Resolving deltas: 100% (4427/4427), done.
$ kubectl exec -it golang -- make build-darwin -C src/kubectl-gs
make: Entering directory '/go/src/kubectl-gs'
... output omitted ...
====> kubectl-gs-v-darwin-amd64
... output omitted ...
cp -a kubectl-gs-v-darwin-amd64 kubectl-gs-darwin
====> build-darwin
make: Leaving directory '/go/src/kubectl-gs'
$ kubectl cp golang:/go/src/kubectl-gs/kubectl-gs-darwin ./kubectl-gs
$ kubectl chmod u+x ./kubectl-gs

Deploying App Platform via `apptestctl`

Once we have both tools, apptestctl and kubectl-gs, it's time to bootstrap App Platform. To do that, we'll use the apptestctl bootstrap command.

The command apptestctl bootstrap needs the KUBECONFIG information to access our minikube cluster, so in this case, we will use the command kubectl config view --flatten --minify -o json to obtain it.

Alternatively, we would need to look for the .kube/config file and pass it with the --kubeconfig-path option.

$ ./apptestctl bootstrap --kubeconfig "$(kubectl config view --flatten --minify -o json)"
bootstrapping app platform components
... output omitted ...
app platform components are ready

Once deployed, we can run a few commands to observe the resources created in our cluster.

$ kubectl get deployments -n giantswarm
NAME READY UP-TO-DATE AVAILABLE AGE
app-operator 1/1 1 1 1m20s
chart-operator 1/1 1 1 1m20s
chartmuseum-chartmuseum 1/1 1 1 1m20s
# kubectl get catalog -A
NAMESPACE NAME CATALOG URL AGE
default chartmuseum http://chartmuseum-chartmuseum:8080/charts/ 1m25s
default helm-stable https://charts.helm.sh/stable/packages/ 1m25s

Wait a moment... Where does this Catalog resource come from? The bootstrap process of App Platform creates some CRDs that will support the operators to manage our applications.

$ kubectl get crd
NAME CREATED AT
appcatalogentries.application.giantswarm.io 2022-06-10T15:30:12Z
appcatalogs.application.giantswarm.io 2022-06-10T15:30:12Z
apps.application.giantswarm.io 2022-06-10T15:30:12Z
catalogs.application.giantswarm.io 2022-06-10T15:30:12Z
charts.application.giantswarm.io 2022-06-10T15:30:12Z

In short, once we register a Catalog, several AppCatalogEntries resources will be created. There will be at least one per application and version.

Registering a `Catalog`

Now, it looks like a great time to see what the kubectl-gs plugin can do for us.

$ kubectl-gs get catalogs
NAME NAMESPACE CATALOG URL AGE
chartmuseum default http://chartmuseum-chartmuseum:8080/charts/ 25m
helm-stable default https://charts.helm.sh/stable/packages/ 25m

All right, that was maybe not so impressive, but it'll become much more useful when we register our first catalog. Why is that? Because kubectl gs will help us generate the definition of a Catalog resource through its template subcommand.

$ kubectl-gs template catalog --name giantswarm --namespace default \
--description "Giant Swarm Catalog" --logo http://logo-url \
--url https://giantswarm.github.io/giantswarm-catalog
---
apiVersion: application.giantswarm.io/v1alpha1
kind: Catalog
metadata:
name: giantswarm
labels:
application.giantswarm.io/catalog-visibility: public
namespace: default
spec:
title: giantswarm
description: Giant Swarm Catalog
logoURL: http://logo-url
storage:
URL: https://giantswarm.github.io/giantswarm-catalog
type: helm

Et voilà, our Catalog CRD pointing to a Giant Swarm collection of applications is ready to be deployed into our cluster.

You might have figured out already what each parameter represents. kubectl gs will complain if any of those parameters are missing. Also, pay attention that we didn't use a real logo URL, but if you were using happa, the Giant Swarm Web-UI, would't you like to see a logo identifying your application?

Finally, the URL is the location of the Helm repository from which App Platform will download the applications.

Once we understand what the kubectl gs template command has generated, it's time to create it inside the cluster and let the App Operator do its magic. Let's go for it.

$ kubectl-gs template catalog --name giantswarm --namespace default \
--description "Giant Swarm Catalog" --logo http://logo-url \
--url https://giantswarm.github.io/giantswarm-catalog | kubectl apply -f -
catalog.application.giantswarm.io/giantswarm created
$ kubectl-gs get catalogs
NAME NAMESPACE CATALOG URL AGE
chartmuseum default http://chartmuseum-chartmuseum:8080/charts/ 35m
helm-stable default https://charts.helm.sh/stable/packages/ 35m
giantswarm default https://giantswarm.github.io/giantswarm-catalog 53s
$ kubectl gs get catalog giantswarm
CATALOG APP NAME VERSION UPSTREAM VERSION AGE DESCRIPTION
... output omitted ...
giantswarm falco-app 0.3.2 0.0.1 5m26s A Helm chart for falco
... output omitted ...

Do you remember the aforementioned AppCatalogEntries that the App Operator had to create once we defined the Catalog? Here are the Falco ones.

$ kubectl get AppCatalogEntries | grep falco-app
giantswarm-falco-app-0.1.2 giantswarm falco-app 0.1.2 0.0.1 240d
giantswarm-falco-app-0.2.0 giantswarm falco-app 0.2.0 0.0.1 176d
giantswarm-falco-app-0.3.0 giantswarm falco-app 0.3.0 0.0.1 103d
giantswarm-falco-app-0.3.1 giantswarm falco-app 0.3.1 0.0.1 94d
giantswarm-falco-app-0.3.2 giantswarm falco-app 0.3.2 0.0.1 79d

Installing an App from the App Catalog

What we've done so far was deploy App Platform, which is required only once. Giant Swarm would have configured that for us already if we were using their services.

Now, it's finally time to create the CRD that will trigger the App Operator to assist in the deployment of Falco. How do we do that? kubectl gs comes to the rescue again!

$ kubectl gs template app --catalog giantswarm --name falco-app --namespace falco-ns
--version 0.3.2 --app-name my-falco --in-cluster
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
name: my-falco
labels:
app-operator.giantswarm.io/version: 0.0.0
namespace: falco-ns
spec:
name: falco-app
version: 0.3.2
namespace: falco-ns
kubeConfig:
inCluster: true
catalog: giantswarm

It is worth mentioning that we are testing on a minikube cluster, where we install applications inside the cluster itself. To achieve that, we use the --in-cluster parameter passed to the previous commands.

Otherwise, if we wanted to install or update the application in one of our managed workload clusters, we would use the --cluster parameter to indicate where the application should be deployed:

$ kubectl gs template app --catalog giantswarm --name falco-app --namespace falco-ns \
--version 0.3.2 --cluster cluster-123 --app-name my-falco
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
name: my-falco
namespace: cluster-123
spec:
name: falco-app
version: 0.3.2
namespace: falco-ns
kubeConfig:
inCluster: false
catalog: giantswarm

In the previous output, you can see how the namespace field inside the metadata section receives the name of the cluster instead of the actual namespace where the application should reside.

The reason is that, although the application will be installed on one of the workload clusters, this CRD will be created in a namespace inside the management cluster. This topic alone would be enough for a whole new post.

Here is a graphical representation of the CRDs supporting App Platform, in the management cluster:

Finally, the last step is creating the CRD for the App in the cluster. Don't forget to ensure that the namespace where the CRD will belong exists, or the kubectl apply command will fail.

$ kubectl create ns falco-ns
namespace/falco-ns created
$ kubectl gs template app --catalog giantswarm --name falco-app --namespace falco-ns \
--version 0.3.2 --in-cluster --app-name my-falco | kubectl apply -f-
app.application.giantswarm.io/my-falco created
$ kubectl gs get app -n falco-ns
NAME VERSION LAST DEPLOYED STATUS NOTES
my-falco 0.3.2 113s deployed

Here are the resulting Kubernetes resources when using regular kubectl commands.

$ kubectl get app,deployment,daemonset -n falco-ns
NAME INSTALLED VERSION CREATED AT LAST DEPLOYED STATUS
app.application.giantswarm.io/my-falco 0.3.2 4m25s 4m24s deployed
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/my-falco-falcosidekick 2/2 2 2 4m24s
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
daemonset.apps/my-falco 1 1 1 1 1 <none> 4m24s
daemonset.apps/my-falco-falco-exporter 1 1 1 1 1 <none> 4m24s

The previous output might differ depending on the type of cluster you would be using, among other variables.

As you can see, once App Platform is up and running, we only need to create the namespace that should contain the Falco application (which should already exist if we are deploying from a managed workload cluster), and the CRD based on the template from the kubectl gs plugin. In a matter of seconds, Falco will be up and running, watching for threats and alerting when suspicious behaviors arise.

Managed Security

Zach Stone, Platform Engineer at Giant Swarm, walked us through some of the biggest challenges that the company's customers face and how his team is using Falco to develop thoughtful solutions.

“The biggest problem that most of our customers face isn't what happens in the cluster, it's what happens with the information once they get it out of the cluster,” asserted Stone. “People also focus too much on the capability that a tool offers and don't take a bigger look at the security processes it supports.”

“If a customer has a vulnerability management program, we can track all of the vulnerabilities in their components, but if fixing those vulnerabilities isn’t a priority, then the program doesn’t work,” remarked Stone. "The larger discussion is usually about where the alerts should go, who bears responsibility for remediation, and how to fit that work into the team's limited capacity. We spend a lot of time trying to ensure security isn't just something that sits alongside the business, but rather is a meaningful part of the daily routine."

Part of that effort is in tuning detection rules and alerting. "Any time we surface an alert, it should be actionable and have a clear owner who is invested in never seeing that alert again."

“I think Falco's superpower is in the flexibility of the policies. I'm also really excited about the changes that are slated to make it easier to update them. Most rules aren't one-size-fits-all -- for a given policy, there is usually some refinement needed to ensure the policy makes sense within our platform, and then customers modify it even further to meet their security requirements. All that customization can make it incredibly difficult to reconcile,” said Stone. “The fact that we can already do it with Falco speaks volumes about the versatility of the solution.”

Security Pack

Giant Swarm's Security Pack is a collection of open source security tools offered by Giant Swarm, which not only contains Falco but also a plethora of other open source projects, including Kyverno for policy enforcement, Trivy for image scanning, and Cosign for image signature verification.

Security does not apply to a single level and, therefore, Security Pack consists of multiple applications, each one independently installable and configurable, available via their App Platform. “Falco will be the cornerstone of our node-level security capabilities,” affirmed Stone, “the biggest opportunity for API plug-ins I see is to get feedback from the node level back into the Security Pack so that we can further contextualize events in the ecosystem.”

Conclusion

Adding simplicity to our cluster management is considered a requirement nowadays, especially in those cases where the lack of resources in an organization can keep it from achieving an acceptable level of security.

Features like Giant Swarm's App Platform and Security Pack will help organizations to finally focus on what actually matters to them: Running their business. In the future, Giant Swarm plans to launch its security pack across all its customers' clusters, enabled by default and built on Falco.

Blog: Kubernetes Response Engine, Part 9: Falcosidekick + Fission

Wed, 01 Sep 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7 : Falcosidekick + Cloud Functions

Kubernetes Response Engine, Part 8: Falcosidekick + Flux v2

The earlier posts in this series, show how to use Kubeless, Argo, Knative, and others to trigger a resource after getting input from Falcosidekick. Recently, Falcosidekick received a new output type support for Fission.

In this blog post, we will cover using Falcosidekick and Fission to detect and delete a compromised pod in a Kubernetes cluster. We will briefly talk about Fission in this blog, however, you can check the complete documentation here.

Prerequisites

We need tools with the following minimum versions to achieve this demo:

Minikube v1.19.0
Helm v3.5.4
kubectl v1.21.0
fission-cli v1.13.1

Provision local Kubernetes Cluster

There are various ways to provision a local Kubernetes cluster such as, KinD, k3s, k0s, Minikube, etc. We are going to use Minikube in this walkthrough.

Let's get provisioned a local Kubernetes cluster:

$ minikube start --cpus 3 --memory 8192 --vm-driver virtualbox
😄 minikube v1.19.0 on Darwin 10.15.7
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating virtualbox VM (CPUs=3, Memory=8192MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.4 ...
 ▪ Generating certificates and keys ...
 ▪ Booting up control plane ...
 ▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
 ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Install Fission

Fission is a fast, open source serverless framework for Kubernetes with a focus on developer productivity and high performance. Fission operates on just the code: Docker and Kubernetes are abstracted away under normal operation, though you can use both to extend Fission if you want to.

Follow the official documentation for deploying Fission to Kubernetes.

Here we will be using Helm to install Fission:

$ export FISSION_NAMESPACE="fission"
$ kubectl create namespace $FISSION_NAMESPACE
$ kubectl create -k "github.com/fission/fission/crds/v1?ref=1.13.1"
$ helm repo add fission-charts https://fission.github.io/fission-charts/
$ helm repo update
$ helm install --version 1.13.1 --namespace $FISSION_NAMESPACE fission fission-charts/fission-all
NAME: fission
LAST DEPLOYED: Wed Jul 21 18:03:44 2021
NAMESPACE: fission
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Install the client CLI.

Mac:
 $ curl -Lo fission https://github.com/fission/fission/releases/download/1.13.1/fission-1.13.1-darwin-amd64 && chmod +x fission && sudo mv fission /usr/local/bin/

Linux:
 $ curl -Lo fission https://github.com/fission/fission/releases/download/1.13.1/fission-1.13.1-linux-amd64 && chmod +x fission && sudo mv fission /usr/local/bin/

Windows:
 For Windows, you can use the linux binary on WSL. Or you can download this windows executable: https://github.com/fission/fission/releases/download/1.13.1/fission-1.13.1-windows-amd64.exe

2. You're ready to use Fission!

 # Create an environment
 $ fission env create --name nodejs --image fission/node-env

 # Get a hello world
 $ curl https://raw.githubusercontent.com/fission/examples/master/nodejs/hello.js > hello.js

 # Register this function with Fission
 $ fission function create --name hello --env nodejs --code hello.js

 # Run this function
 $ fission function test --name hello
 Hello, world!

Check if everything is working before moving onto the next step:

$ kubectl get pods --namespace fission
NAME READY STATUS RESTARTS AGE
buildermgr-5698c89fff-rk9z6 1/1 Running 0 7m20s
controller-5dcb44bcd6-vq9hb 1/1 Running 0 7m20s
executor-6b6d6469d6-2xrlk 1/1 Running 0 7m20s
fission-kube-state-metrics-5fc9bd6684-7ffwp 1/1 Running 0 7m20s
fission-prometheus-alertmanager-65f5574885-tlrz6 2/2 Running 0 7m20s
fission-prometheus-node-exporter-jd9w6 1/1 Running 0 7m20s
fission-prometheus-node-exporter-jpzn8 1/1 Running 0 7m20s
fission-prometheus-node-exporter-rb25l 1/1 Running 0 7m20s
fission-prometheus-pushgateway-54c87b5796-28x2h 1/1 Running 0 7m20s
fission-prometheus-server-9d64c74b4-ld97h 2/2 Running 0 7m20s
influxdb-59649c8c6-5vx54 1/1 Running 0 7m20s
kubewatcher-6996fccc6b-5vbvx 1/1 Running 0 7m20s
logger-6kdw4 2/2 Running 0 7m20s
logger-nmw9t 2/2 Running 0 7m20s
logger-zkrq9 2/2 Running 0 7m20s
mqtrigger-keda-7584989c48-n5w6g 1/1 Running 0 7m20s
mqtrigger-nats-streaming-664c55c979-t9gp5 1/1 Running 0 7m19s
nats-streaming-6c6d7c6fbf-ft468 1/1 Running 0 7m20s
router-5c5c6cbb87-989pc 1/1 Running 0 7m20s
storagesvc-57ccf58976-qcr4d 1/1 Running 0 7m19s
timer-794b89579b-6kxwx 1/1 Running 0 7m20s

Install Falco + Falcosidekick

Firstly, we'll create the namespace that will host both Falco and Falcosidekick:

$ kubectl create namespace falco

We add the helm repo:

$ helm repo add falcosecurity https://falcosecurity.github.io/charts
"falcosecurity" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
...Successfully got an update from the "openfaas" chart repository
Update Complete. ⎈Happy Helming!⎈

In a real project, you should get the whole chart with helm pull falcosecurity/falco --untar and then configure the values.yaml. For this tutorial, will try to keep thing as easy as possible and set configs directly by passing arguments to helm install command line:

$ helm upgrade --install falco falcosecurity/falco --namespace falco \
--set falcosidekick.enabled=true \
--set falcosidekick.config.fission.function="falco-pod-delete"

You should get this output:

Release "falco" does not exist. Installing it now.
NAME: falco
LAST DEPLOYED: Tue Apr 13 10:49:49 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.


No further action should be required.

And you can see your new Falco and Falcosidekickpods:

$ kubectl get pods --namespace falco
NAME READY STATUS RESTARTS AGE
falco-falcosidekick-f77c58899-gd467 1/1 Running 0 51s
falco-falcosidekick-f77c58899-hfsjx 1/1 Running 0 51s
falco-hg2wm 1/1 Running 0 51s

The argument falcosidekick.enabled=true sets the following settings in Falco for you:

--set falco.jsonOutput=true \
--set falco.httpOutput.enabled=true \
--set falco.httpOutput.url=http://falco-falcosidekick:2801

The arguments --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falco-falcosidekick:2801 are there to configure the format of events and the URL where Falco will send them. As Falco and Falcosidekick will be in the same namespace, it can directly use the name of the service (falco-falcosidekick) above Falcosidekick pods.

We check the logs:

$ kubectl logs deployment/falco-falcosidekick -n falco
Found 2 pods, using pod/falco-falcosidekick-f77c58899-gd467
2021/07/21 12:52:02 [INFO] : Enabled Outputs : [Fission]
2021/07/21 12:52:02 [INFO] : Falco Sidekick is up and listening on :2801

Fission is displayed as enabled output, everything is good 👍.

Install our Fission Function

Our really basic function will receive events from Falco, thanks to Falcosidekick, check if the triggered rule is Terminal Shell in container, extract the namespace and pod name from the fields of events and delete the according pod:

Basically, the process is:

 +----------+ +---------------+ +----------+
 | Falco +-----------------> Falcosidekick +--------------------> Fission |
 +----^-----+ sends event +---------------+ triggers +-----+----+
 | |
detects a shell | |
 | |
 +----+-------+ deletes |
 | Pwned Pod <----------------------------------------------------------+
 +------------+

Let's get the function and other artifacts:

$ git clone https://github.com/fission/examples.git && cd examples/sample/falco

The function we are going to deploy basically receives events for an infected pod from the Falcosidekick and deletes it immediately. Before deploying the function we need some permissions to delete Pod. We create a ServiceAccount with rights to delete a Pod in any namespace, and we'll associate it to our function:

apiVersion: v1
kind: ServiceAccount
metadata:
 name: falco-pod-delete
 namespace: fission-function
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role
rules:
- apiGroups: [""]
 resources: ["configmaps", "secrets"]
 verbs: ["get"]
- apiGroups: [""]
 resources: ["pods"]
 verbs: ["get", "list", "delete"]
- apiGroups: [""]
 resources: ["events"]
 verbs: ["*"]
- apiGroups: ["fission.io"]
 resources: ["packages"]
 verbs: ["get", "list"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role-binding
roleRef:
 kind: ClusterRole
 name: falco-pod-delete-cluster-role
 apiGroup: rbac.authorization.k8s.io
subjects:
 - kind: ServiceAccount
 name: falco-pod-delete
 namespace: fission-function

Let's create the service account with:

$ kubectl apply -f sa-falco-pod-delete.yaml

The falco-pod-delete/handler.go contains our function:

package main

import (
 "context"
 "encoding/json"
 "io/ioutil"
 "log"
 "net/http"
 "time"

 metaV1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/kubernetes"
 "k8s.io/client-go/rest"
)

var kubeClient *kubernetes.Clientset

func init() {
 // creates the in-cluster config
 config, err := rest.InClusterConfig()
 if err != nil {
 panic(err.Error())
 }
 // creates the clientset
 kubeClient, err = kubernetes.NewForConfig(config)
 if err != nil {
 panic(err.Error())
 }
}

type Alert struct {
 Output string `json:"output"`
 Priority string `json:"priority"`
 Rule string `json:"rule"`
 Time time.Time `json:"time"`
 OutputFields struct {
 ContainerID string `json:"container.id"`
 ContainerImageRepository interface{} `json:"container.image.repository"`
 ContainerImageTag interface{} `json:"container.image.tag"`
 EvtTime int64 `json:"evt.time"`
 FdName string `json:"fd.name"`
 K8SNsName string `json:"k8s.ns.name"`
 K8SPodName string `json:"k8s.pod.name"`
 ProcCmdline string `json:"proc.cmdline"`
 } `json:"output_fields"`
}

var criticalNamespaces = []string{"kube-system", "kube-public", "kube-node-lease", "falco", "fission", "fission-function"}

func Handler(w http.ResponseWriter, r *http.Request) {
 var alert Alert

 if r.Body != nil {
 defer r.Body.Close()

 body, _ := ioutil.ReadAll(r.Body)
 json.Unmarshal(body, &alert)

 log.Printf("\n[ INFO ] Alert : %v\n", alert)

 podName := alert.OutputFields.K8SPodName
 namespace := alert.OutputFields.K8SNsName

 var critical bool
 for _, ns := range criticalNamespaces {
 if ns == namespace {
 critical = true
 break
 }
 }

 if !critical {
 _, err := kubeClient.CoreV1().Pods(namespace).Get(context.Background(), podName, metaV1.GetOptions{})
 if err != nil {
 log.Printf("\n[ WARN ] Failed to get pod '%s' in '%s' namespace", podName, namespace)
 return
 }

 log.Printf("Deleting pod %s from namespace %s", podName, namespace)
 err = kubeClient.CoreV1().Pods(namespace).Delete(context.Background(), podName, metaV1.DeleteOptions{})
 if err != nil {
 log.Printf("\n[ ERROR ] Failed to delete pod: %v", err)
 w.WriteHeader(http.StatusInternalServerError)
 w.Write([]byte(err.Error()))
 }
 }
 }

 w.WriteHeader(http.StatusOK)
 w.Write([]byte("OK"))
}

A fission function requires an environment/runtime to run. The yaml definitions for the runtime, the function and the router are available under the specs directory.

Now, we are ready to deploy our falco-pod-delete function using the specs:

$ fission spec apply
DeployUID: edc80e3e-7d1e-448c-aba8-c8cd75b3a1eb
Resources:
 * 1 Functions
 * 1 Environments
 * 1 Packages
 * 1 Http Triggers
 * 0 MessageQueue Triggers
 * 0 Time Triggers
 * 0 Kube Watchers
 * 1 ArchiveUploadSpec
Validation Successful
Spec doesn't belong to Git Tree.
1 function created: falco-pod-delete
1 HTTPTrigger created: falco-pod-delete
1 environment created: go
1 package created: falco-pod-delete-d18f6a0b-e5a1-4275-9471-38d684ac4dfe

Check if the package was built successfully for our fission function before moving to the next step:

$ fission pkg list
NAME BUILD_STATUS ENV LASTUPDATEDAT
falco-pod-delete-d18f6a0b-e5a1-4275-9471-38d684ac4dfe succeeded go 21 Jul 21 08:26 IST

Test our function

We start by creating a dumb pod:

$ kubectl run alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"
pod/alpine created

$ kubectl get pods --namespace default
AME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 11s

Let's run a shell command inside and see what happens:

$ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"

19:27:21 up 50 min, load average: 0.11, 0.12, 0.11

As expected we got the result of our command, but, if we get the status of the pod we retrieve:

$ kubectl get pods --namespace default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 103s

💥 It has been terminated 💥

We can now check the logs of components.

For Falco:

$ kubectl logs daemonset/falco --namespace falco
..
{"output":"10:36:32.750441241: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=cbd3133ccac6 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=cbd3133ccac6 image=alpine) k8s.ns=default k8s.pod=alpine container=cbd3133ccac6","priority":"Notice","rule":"Terminal shell in container","time":"2021-07-21T10:36:32.750441241Z", "output_fields": {"container.id":"cbd3133ccac6","container.image.repository":"alpine","evt.time":1626863792750441241,"k8s.ns.name":"default","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}
{"output":"10:37:09.101509967: Notice Unexpected connection to K8s API Server from container (command=fetcher -secret-dir /secrets -cfgmap-dir /configs -jaeger-collector-endpoint /userfunc k8s.ns=fission-function k8s.pod=poolmgr-go-default-516098-5bdbf8c8f5-g8gvc container=281c99ea33c2 image=fission/fetcher:1.13.1 connection=192.168.43.223:39526->10.100.0.1:443) k8s.ns=fission-function k8s.pod=poolmgr-go-default-516098-5bdbf8c8f5-g8gvc container=281c99ea33c2","priority":"Notice","rule":"Contact K8S API Server From Container","time":"2021-07-21T10:37:09.101509967Z", "output_fields": {"container.id":"281c99ea33c2","container.image.repository":"fission/fetcher","container.image.tag":"1.13.1","evt.time":1626863829101509967,"fd.name":"192.168.43.223:39526->10.100.0.1:443","k8s.ns.name":"fission-function","k8s.pod.name":"poolmgr-go-default-516098-5bdbf8c8f5-g8gvc","proc.cmdline":"fetcher -secret-dir /secrets -cfgmap-dir /configs -jaeger-collector-endpoint /userfunc"}}
..

For Falcosidekick:

$ kubectl logs deployment/falco-falcosidekick --namespace falco
..
2021/07/21 10:37:13 [INFO] : Fission - Post OK (200)
2021/07/21 10:36:32 [INFO] : Fission - Function Response : OK
2021/07/21 10:36:32 [INFO] : Fission - Call Function "falco-pod-delete" OK
..

For falco-delete-pod function:

$ fission function logs -f --name falco-pod-delete
..
[2021-07-21 10:47:27.206605532 +0000 UTC] 2021/07/21 10:47:27 Deleting pod alpine from namespace default
..

Conclusion

With this really simple example, we got another way to create a Response Engine with amazing pieces of software from the Open Source world. We only scratched the surface of possibilities, so don't hesitate to share with us your comments, ideas and successes.

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 8: Falcosidekick + Flux v2

Tue, 31 Aug 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7 : Falcosidekick + Cloud Functions

Today, we will set up another KRE (Kubernetes Response Engine) based on Flux (version 2). If you don't know about Flux (version 2), let me explain it in a few words. Flux (version 2) is a tool for keeping Kubernetes clusters in sync with configuration sources (such as Git repositories) and automating updates to the configuration when new code is available to deploy.

To learn more about Flux (version 2), see Flux Documentation.

Flux (version 2) might look like a GitOps tool. It is, in reality, another GitOps tool in that it watches Github repositories for configuration changes and keeps the current state and the desired state always in sync. It does that on top of Kubernetes by using a bunch of CRs (Custom Resources). However, unlike Knative Eventing, Tekton Triggers, and Argo Events, Flux (version 2) does not support an eventing system to forward events from one point to another.

To set up KRE with Flux (version 2), we will create a small project that listens to events and updates the GitHub repository, which Flux (version 2) monitors to alter the desired state. For example, get the event of a pwned pod, then change its replicas to zero within the deployment YAML file.

The reference architecture given below illustrates the content of this blog.

Table of Contents

Prerequisites

minikube v1.21.0
helm v3.6.2+gee407bd
kubectl v1.21.1
ko v0.8.3
flux v0.16.0
gcloud v347.0.0

Tutorial

Provision local Kubernetes Cluster

$ minikube config view
- cpus: 3
- driver: virtualbox
- memory: 8192

minikube start

Install Flux V2 - GitOps Toolkit

I highly recommended that you check out getting started page of Flux (version 2). It gives you detailed installation instructions for Flux (version 2).

For Flux (version 2) to create a GitHub repository for its resources, we must define the token and username information. Then, Flux (version 2) installed in a GitOps way, and Flux (version 2) will push its manifest to the repository.

export GITHUB_USER=<username>
export GITHUB_TOKEN=<token>

Once the necessary environment variables are defined, we can install Flux (version 2). The following command will create and push its manifests to the repository, then install Flux components.

flux bootstrap github \
 --owner=$GITHUB_USER \
 --repository=fleet-infra \
 --branch=main \
 --path=./clusters/my-cluster \
 --personal

After the installation is complete, the next step is creating the alpine GitRepository and Kustomization CRD's (Custom Resource Definitions). For more information, see components.

Apply the CRD files as follows:

$ kubectl apply -f https://raw.githubusercontent.com/developer-guy/kubernetes-response-engine-based-on-flux-v2-gitops-toolkit/master/alpine-gitsource.yaml
gitrepository.source.toolkit.fluxcd.io/alpine created

$ kubectl apply -f https://raw.githubusercontent.com/developer-guy/kubernetes-response-engine-based-on-flux-v2-gitops-toolkit/master/alpine-kustomization.yaml
kustomization.kustomize.toolkit.fluxcd.io/alpine created

Alternatively, we can use the Flux CLI.

To create GitRepository:

flux create source git alpine \
 --url https://github.com/developer-guy/desired-state-repository \
 --branch master \
 --interval 30s

To create Kustomization:

flux create kustomization alpine \
 --source alpine \
 --path "./" \
 --prune true \
 --validation client \
 --interval 5m

Install falco-event-listener

As the name suggests, this program will listen to an event in the form of CloudEvents. This CloudEvents forwarded from Falcosidekick, a simple daemon for enhancing available outputs for Falco. After the successful receipt of the event, falco-event-listener will update the YAML definition to scale its replicas to zero based on the pieces of information given in the event.

In most basic form, the architecture of the demo is:

Falco w/HTTP --> Falcosidekick w/CloudEvent --> falco-event-listener w/HTTP --> GitHub

To learn more about CloudEvents, see cloudevents.io.

First, let us clone the falco-event-listener repository.

git clone https://github.com/developer-guy/falco-event-listener
cd falco-event-listener

Before installing this project, we have to do a couple of things.

We have to create a GitHub PAT (Personal Access Token) to be able to update the desired state configurations which are in the GitHub repository after we detect a malicious behavior related to our pod has detected.

As you can see in the above arguments, we should pass GITHUB_TOKEN as an argument to our CLI application. The best option is storing GITHUB_TOKEN in a Kubernetes Secret and using Secret as environment variables.

kubectl create namespace falco
kubectl create secret generic github-secret \
 --from-literal GITHUB_TOKEN=$GITHUB_TOKEN \
 --namespace falco

You might notice that we are using some URLs (--notify-url) within the flags of the project. Let me explain that why we need it a bit. As you might know, we are trying to set up some remediation engine here, so we have to react to those events thrown by the Falcosidekick as soon as possible. We defined an interval while creating a GitRepository, which means that Flux (version 2) will wait at least that long to sync configurations, so we have to notify Flux (version 2) controllers about changes once we edited the desired state by a process whose name is falco-event-listener. To notify the Flux (version 2) controllers about changes in Git or Helm repositories, we can set up webhooks and trigger a cluster reconciliation every time a source changes. For more detail, please see.

There are different kinds of webhook receivers in Flux (version 2), but we'll use the generic one in this guide.

const (
 GenericReceiver string = "generic"
 GenericHMACReceiver string = "generic-hmac"
 GitHubReceiver string = "github"
 GitLabReceiver string = "gitlab"
 BitbucketReceiver string = "bitbucket"
 HarborReceiver string = "harbor"
 DockerHubReceiver string = "dockerhub"
 QuayReceiver string = "quay"
 GCRReceiver string = "gcr"
 NexusReceiver string = "nexus"
 ACRReceiver string = "acr"
)

Let us to create a Receiver, to do that we have to a couple of things again:

$ TOKEN=$(head -c 12 /dev/urandom | sha256sum | cut -d ' ' -f1)
$ echo $TOKEN
0babd54d2b64d6d6fcd10a663cb6195773e968ba6642ca8c1a8a54df7b52efd0

$ kubectl -n flux-system create secret generic webhook-token \
--from-literal=token=$TOKEN
secret/webhook-token created

$ cat << EOF | kubectl apply -f -
apiVersion: notification.toolkit.fluxcd.io/v1beta1
kind: Receiver
metadata:
 name: generic-receiver
 namespace: flux-system
spec:
 type: generic
 secretRef:
 name: webhook-token
 resources:
 - apiVersion: source.toolkit.fluxcd.io/v1beta1
 kind: GitRepository
 name: alpine
 namespace: flux-system
EOF
receiver.notification.toolkit.fluxcd.io/generic-receiver created

$ cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
 name: receiver
 namespace: flux-system
spec:
 type: ClusterIP
 selector:
 app: notification-controller
 ports:
 - name: http
 port: 80
 protocol: TCP
 targetPort: 9292
EOF

This program is basically a CLI application and it uses the following arguments as:

args:
 - "--owner"
 - "developer-guy" # this is the owner of the desired state repository
 - "--repository"
 - "desired-state-repository" # this is the repository which we store desired state configurations
 - "--file"
 - "alpine.yaml" # this is the file that we are going to update
 - "--github-token"
 - "$(GITHUB_TOKEN)"
 - "--notify-url"
 - "http://receiver.flux-system/$(WEBHOOK_URL)"

Let us run our project on the Kubernetes cluster. To do that, we'll be using the ko tool. ko, created by Google, is a simple, fast container image builder for Go applications. For more information, see the official repository of the project.

We'll use Container Registry as an image repository service provided by the Google Cloud to store, manage, and secure our container images. Alternatively, we could also use DockerHub, quay.io, and so on.

export WEBHOOK_URL=$(kubectl get receivers -n flux-system generic-receiver -ojsonpath='{.status.url}')
export PROJECT_ID=$(gcloud config get-value project)
export KO_DOCKER_REPO=gcr.io/$PROJECT_ID
envsubst < falcoeventlistener.yaml | ko apply -f -

If you are using a private container registry, don't forget to create a registry secret to pull and push images. You can follow the following guide to achieve this:
https://colinwilson.uk/2020/07/09/using-google-container-registry-with-kubernetes/

If everything works as expected, we should see an output as given below:

$ kubectl get pods --namespace falco
NAME READY STATUS RESTARTS AGE
falco-event-listener 1/1 Running 0 59s

The last step we have to do is installing Falco and Falcosidekick with configuring Falcodekick to forward events to our application.

Installing Falco and Falcosidekick

For an up-to-date and detailed guide to installing Falco and Falcosidekick, see falcosidekick).

Let us enable CloudEvents support of Falcosidekick.

helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update
helm upgrade --install falco falcosecurity/falco \
 --namespace falco \
 --set falcosidekick.enabled=true \
 --set falcosidekick.webui.enabled=true \
 --set falcosidekick.config.cloudevents.address=http://falco-event-listener

Verify that everything is working as expected:

$ kubectl get pods --namespace falco
NAME READY STATUS RESTARTS AGE
falco-event-listener 1/1 Running 0 5m19s
falco-falcosidekick-5854669c76-ddvrv 1/1 Running 0 2m16s
falco-falcosidekick-5854669c76-rdlqn 1/1 Running 0 2m16s
falco-falcosidekick-ui-7c5fc8dd54-q4qh9 1/1 Running 0 2m16s
falco-vkl4f 1/1 Running 0 2m16s

Test

To test this, we have to connect a shell within the container.

Let's list the pods that we already have.

$ kubectl get pods
NAME READY STATUS RESTARTS AGE
alpine-deployment-77789455d6-m7flp 1/1 Running 15 3h6m
alpine-deployment-77789455d6-v7fkw 1/1 Running 15 3h6m
podinfo-6df788c7b8-gs5qb 1/1 Running 0 3h28m
podinfo-6df788c7b8-sfxvd 1/1 Running 0 3h28m

Now, run the following command to connect a shell.

$ kubectl exec -ti alpine-deployment-77789455d6-m7flp -- sh -c "uptime"
19:35:58 up 3:04, load average: 3.12, 1.91, 1.22

Once you run the command above, Falco will detect that malicious behavior and send it to the Falcosidekick via HTTP.

You should see an output in the Falco logs as given below:

{"output":"19:35:58.532086161: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine-deployment-77789455d6-m7flp container=788861c3cf83 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=788861c3cf83 image=alpine) k8s.ns=default k8s.pod=alpine-deployment-77789455d6-m7flp container=788861c3cf83","priority":"Notice","rule":"Terminal shell in container","time":"2021-06-13T19:35:58.532086161Z", "output_fields": {"container.id":"788861c3cf83","container.image.repository":"alpine","evt.time":1623612958532086161,"k8s.ns.name":"default","k8s.pod.name":"alpine-deployment-77789455d6-m7flp","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}

and the similar output below in the Falcosidekick logs:

falco-falcosidekick-5854669c76-ddvrv falcosidekick 2021/06/13 19:51:45 [INFO] : CloudEvents - Send OK

and the similar output below in the falco-event-listener logs:

falco-event-listener falco-event-listener 2021/06/13 19:35:59 resp.Status=200 OK
falco-event-listener falco-event-listener 2021/06/13 19:35:59 resp.StatusCode=200
falco-event-listener falco-event-listener 2021/06/13 19:35:59 [Terminal shell in container] scaled down to zero alpine-deployment-77789455d6-m7flp from default because 19:35:58.532086161: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine-deployment-77789455d6-m7flp container=788861c3cf83 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=788861c3cf83 image=alpine) k8s.ns=default k8s.pod=alpine-deployment-77789455d6-m7flp container=788861c3cf83

You should also notice that a new commit is available in the desired-state-repository as given below:

After the commit, Flux (version 2) will detect the change and sync the current state of the cluster with the desired state in the GitHub repository so that Flux (version 2) will terminate the alpine deployment pods.

$ kubectl get pods -l app=alpine
No resources found in default namespace.

Blog: Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

Tue, 29 Jun 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Earlier in this series, we have seen how to use Argo, Tekton, and Knative to trigger a resource after getting input from Falcosidekick. Recently, Falcosidekick received a new output type support for Cloud Functions.

In this part, let us learn how we can use Falcosidekick and Cloud Functions to detect and delete a compromised pod.

We will not go through what Cloud Functions is in-depth, however, you can always find a good overview about it in the official documentation.

Here is the high-level overview architecture that shows what we want to achieve at the end of the day:

You can find all the related code and resources in this repository.

Prerequisites

gcloud 342.0.0
kubectl 1.20.5

Tutorial

Provision Google Kubernetes Engine (GKE) Cluster

As the blog title said already, we need to create a GKE cluster with workload identity enabled:

$ GOOGLE_PROJECT_ID=$(gcloud config get-value project)
$ CLUSTER_NAME=falco-falcosidekick-demo
$ gcloud container clusters create $CLUSTER_NAME --workload-pool ${GOOGLE_PROJECT_ID}.svc.id.goog
$ gcloud container clusters get-credentials $CLUSTER_NAME

Configure IAM Service Accounts

We need to create a new Service Account for target $GOOGLE_PROJECT_ID using IAM Binding policies to get access our Cloud Function:

$ SA_ACCOUNT=falco-falcosidekick-sa
$ gcloud iam service-accounts create $SA_ACCOUNT

$ gcloud projects add-iam-policy-binding ${GOOGLE_PROJECT_ID} \
--member="serviceAccount:${SA_ACCOUNT}@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/cloudfunctions.developer"

$ gcloud projects add-iam-policy-binding ${GOOGLE_PROJECT_ID} \
--member="serviceAccount:${SA_ACCOUNT}@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com" \
--role="roles/cloudfunctions.invoker"

In the beginning, we already enabled WorkloadIdentity feature for our GKE Cluster by setting --workload-pool flag. What we need to do here is to add a iam.workloadIdentityUser role for the given Service Account.

$ gcloud iam service-accounts add-iam-policy-binding \
 --role roles/iam.workloadIdentityUser \
 --member "serviceAccount:${GOOGLE_PROJECT_ID}.svc.id.goog[${FALCO_NAMESPACE}/falco-falcosidekick]" \
 ${SA_ACCOUNT}@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com

We need to annotate the falco-falcosidekick resource. So it can grant access for our Cluster. Set up the Falcosidekick SA to impersonate a GCP SA:

$ kubectl annotate serviceaccount \
 --namespace $FALCO_NAMESPACE \
 falco-falcosidekick \
 iam.gke.io/gcp-service-account=${SA_ACCOUNT}@${GOOGLE_PROJECT_ID}.iam.gserviceaccount.com

Create Necessary Cluster Role

To limit function role access in the particular cluster, ensure that only SA has limited permissions within a particular namespace by using Role Bindings.

$ kubectl create serviceaccount pod-destroyer
$ kubectl create clusterrole pod-destroyer
 --verb=delete
 --resource=pod # give only pod resource access for delete op 
$ kubectl create clusterrolebinding pod-destroyer
 --clusterrole pod-destroyer
 --serviceaccount default:pod-destroyer

To obtain the Token from secret, we need to get pod-deleter ServiceAccount resource first:

$ POD_DESTROYER_TOKEN=$(kubectl get secrets $(kubectl get serviceaccounts pod-deleter -o json \
 | jq -r '.secrets[0].name') -o json \
 | jq -r '.data.token' \
 | base64 -D)

Add the pod-destroyer user to your KUBECONFIG:

# Generate your KUBECONFIG
$ kubectl config view --minify --flatten > kubeconfig_pod-destroyer.yaml

# Set the token at the end of yaml
$ cat << EOF >> kubeconfig_pod-destroyer.yaml
users:
- name: user.name
 user:
 token: $POD_DESTROYER_TOKEN

We can test it with auth can-i to check if roles are set correctly

$ kubectl auth can-i list deployments # no
$ kubectl auth can-i delete pod # yes
$ kubectl access-matrix # github.com/corneliusweig/rakkess

Create Secret Manager

The main reason Secret Manager get involved our architecture is because we had to find a way out to initialize our kubeclient in our function by getting pod-destroyer's KUBECONFIG.

We need to create a new secrets IAM policy for the SA member to enable Managing Secrets:

$ gcloud secrets add-iam-policy-binding pod-destroyer \
 --role roles/secretmanager.secretAccessor \
 --member serviceAccount:$SA_ACCOUNT@$GOOLE_PROJECT_ID.iam.gserviceaccount.com

Create a new secret, called pod-destroyer:

$ gcloud secrets create pod-destroyer --replication-policy="automatic"

Push the our generated kubeconfig_pod-destroyer.yaml file as a new version:

$ gcloud secrets versions add pod-destroyer --data-file=kubeconfig_pod-destroyer.yaml

Finally, we are ready to deploy our Cloud Run function!

Deploy Google Cloud Function

In this demonstration our function will simply delete the pwned Pod, as we already pointed it out in the architecture diagram.

The Go code

package kill_the_pwned_pod

import (
 secretmanager "cloud.google.com/go/secretmanager/apiv1"
 "context"
 "encoding/json"
 "fmt"
 secretmanagerpb "google.golang.org/genproto/googleapis/cloud/secretmanager/v1"
 "io/ioutil"
 metaV1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/kubernetes"
 "k8s.io/client-go/tools/clientcmd"
 "net/http"
 "os"
 "time"
)

// Alert falco data structure
type Alert struct {
 Output string `json:"output"`
 Priority string `json:"priority"`
 Rule string `json:"rule"`
 Time time.Time `json:"time"`
 OutputFields struct {
 ContainerID string `json:"container.id"`
 ContainerImageRepository interface{} `json:"container.image.repository"`
 ContainerImageTag interface{} `json:"container.image.tag"`
 EvtTime int64 `json:"evt.time"`
 FdName string `json:"fd.name"`
 K8SNsName string `json:"k8s.ns.name"`
 K8SPodName string `json:"k8s.pod.name"`
 ProcCmdline string `json:"proc.cmdline"`
 } `json:"output_fields"`
}

var op Operation

type Operation struct {
 clientSet *kubernetes.Clientset
}

// init initializes new Kubernetes ClientSet with given config
func init() {
 // The resource name of the KUBECONFIG_SECRET_NAME in the format
 // `projects/*/secrets/*/versions/*`
 resource := os.Getenv("KUBECONFIG_SECRET_NAME")
 if len(resource) == 0 {
 panic(fmt.Errorf("$KUBECONFIG_SECRET_NAME env variable did not set"))
 }

 secret, err := GetSecret(resource)
 if err != nil {
 panic(fmt.Errorf("get secret: %q", err))
 }

 kubeCfg, err := clientcmd.NewClientConfigFromBytes(secret)
 if err != nil {
 panic(fmt.Errorf("new client config: %q", err))
 }

 restCfg, err := kubeCfg.ClientConfig()
 if err != nil {
 panic(fmt.Errorf("client config: %q", err))
 }

 cs, err := kubernetes.NewForConfig(restCfg)
 if err != nil {
 panic(fmt.Errorf("unable to initialize config: %q", err))
 }

 op = Operation{clientSet: cs}
}

// KillThePwnedPod will executed for each Falco event
func KillThePwnedPod(w http.ResponseWriter, r *http.Request) {
 body, err := ioutil.ReadAll(r.Body)
 if err != nil {
 http.Error(w, "cannot read body", http.StatusBadRequest)
 return
 }

 var event Alert

 err = json.Unmarshal(body, &event)
 if err != nil {
 http.Error(w, "cannot parse body", http.StatusBadRequest)
 return
 }

 err = op.PodDestroy(event.OutputFields.K8SPodName, event.OutputFields.K8SNsName)
 if err != nil {
 http.Error(w, fmt.Sprintf("cannot delete pod: %q", err), http.StatusInternalServerError)
 return
 }

 w.WriteHeader(http.StatusOK)
}

// PodDestroy destroys the given pod name in the given namespace
func (d *Operation) PodDestroy(name, namespace string) error {
 err := d.clientSet.CoreV1().Pods(namespace).Delete(context.TODO(), name, metaV1.DeleteOptions{})
 if err != nil {
 return fmt.Errorf("unable to delete pod %s: %q", name, err)
 }
 return nil
}

// GetSecret returns the secret data.
func GetSecret(name string) ([]byte, error) {
 ctx := context.Background()

 client, err := secretmanager.NewClient(ctx)
 if err != nil {
 return nil, fmt.Errorf("failed to create secretmanager client: %v", err)
 }
 defer client.Close()

 result, err := client.AccessSecretVersion(ctx, &secretmanagerpb.AccessSecretVersionRequest{
 Name: name,
 })

 if err != nil {
 return nil, fmt.Errorf("failed to access secret version: %v", err)
 }

 return result.Payload.Data, nil
}

If you rather see it in GitHub.

$ git clone https://github.com/Dentrax/k8s-response-engine-gke-functions.git
$ cd kubernetes-response-engine-based-on-gke-and-gcloudfunctions
...

We need to pass extra --service-account flag in order to get access to Secret Manager.

Deploy the function:

$ FUNCTION_NAME=KillThePwnedPod
$ gcloud functions deploy $FUNCTION_NAME \
--runtime go113 --trigger-http \
--service-account $SA_ACCOUNT@$GOOLE_PROJECT_ID.iam.gserviceaccount.com
Allow unauthenticated invocations of new function [KillThePwnedPod]? (y/N)? N
...

Now, get the name of the function:

$ CLOUD_FUNCTION_NAME=$(gcloud functions describe --format=json $FUNCTION_NAME | jq -r '.name')

Install Falco + Falcosidekick

It is time to install Falco, Falcosidekick with Cloud Function output type enabled:

$ export FALCO_NAMESPACE=falco
$ kubectl create namespace $FALCO_NAMESPACE
$ helm upgrade --install falco falcosecurity/falco \
--namespace $FALCO_NAMESPACE \
--set ebpf.enabled=true \
--set falcosidekick.enabled=true \
--set falcosidekick.config.gcp.cloudfunctions.name=${CLOUD_FUNCTION_NAME} \
--set falcosidekick.webui.enabled=true

Test

Try to run a busybox image and execute a command:

$ kubectl run busybox --image=busybox --restart='Never' -- sh -c "sleep 600"

Try to exec into:

$ kubectl exec -it busybox -- sh -c "uptime"

We can check the logs of the Falco, and Falcosidekick to see what happened:

For Falcosidekick:

$ kubectl logs deployment/falco-falcosidekick --namespace falco
2021/06/14 21:01:24 [INFO] : GCPCloudFunctions - Call Cloud Function OK
..

Conclusion

We got another way to create a Response Engine with amazing pieces of software from Open Source world. Of course, it's just the beginning, feel free to share your functions and workflows with the community for starting the creation of a true library of remediation methods.

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 6: Falcosidekick + Cloud Run

Fri, 25 Jun 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

Recently, we added two new output-type support to Falcosidekick, and they are Cloud Functions, and Cloud Run. This blog post will discuss how to set up Kubernetes Response Engine on GKE (Google Kubernetes Engine) by using Cloud Run.

Let's start by explaining a little bit about Cloud Run. Cloud Run is a managed compute platform that enables you to run containers that are invocable via requests or events. Cloud Run is serverless: it abstracts away all infrastructure management, so you can focus on what matters most — building great applications.

For more information, see Cloud Run.

Given below is a reference architecture of what's being explained in this blog.

This demo might be useful for Google Cloud users who might already be using GKE with Falco to protect container runtime against malicious behaviors, and wants to take any action for them with Cloud Run.

Table of Contents

Prerequisites

gcloud 342.0.0
ko 0.8.3

Tutorial

Provision GKE (Google Kubernetes Engine) Cluster

First, let us create a GKE cluster.

$ GKE_CLUSTER_NAME=cloud-run-demo
$ gcloud container clusters create $GKE_CLUSTER_NAME

To learn more about the setup GKE Cluster, see quickstart guide.

Deploy Cloud Run Function

Once GKE is set up, we are ready to deploy Cloud Run. But before doing that, let us examine the responsibility of the Cloud Run function. As you can see in the reference architecture, this function will delete the pwned pods. To be able to do that, Cloud Run should be given appropriate permissions.

There are two approaches to obtain these permissions.

The first approach is creating a Kubernetes Service Account, an appropriate Role with granted permissions to delete pod resource, and a RoleBinding to bind Role to Service Account. Then create the kubeconfig file, package it up with the function code while deploying the Cloud Run function, and use this file to create a Kubernetes client.

To learn more about the kubeconfig files, see kubeconfig.

The second approach is producing a valid ~/.kube/config with a library called google.golang.org/api/ within the function code. We are doing this because the representation of the valid ~/.kube/config file is clientcmd/api/Config in Go.

We'll go with the second approach in this demo. Thanks to Scott Blum and his detailed blog post on this topic. I highly recommend that you check that out.

Let's deploy the function. If you want to take a look at the function code, see the repository.

Note that we're going to use the ko tool to build and push our container image which is created by Google. ko is a simple and fast container image builder for Go applications.

To learn more, see the official repository of the project.

We are also going to use Container Registry as an image repository service provided by the Google Cloud to store, manage, and secure your Docker container images. Alternatively, you can also use DockerHub, quay.io, etc.

$ git clone https://github.com/developer-guy/kubernetes-response-engine-based-on-gke-and-cloud-run.git
$ cd kubernetes-response-engine-based-on-gke-and-cloud-run
$ GKE_CLUSTER_NAME=$(kubectl config view --minify -o jsonpath='{.clusters[].name}{"\n"}')
$ PROJECT_ID=$(gcloud config get-value project)
$ FUNCTION_NAME=pod-deleter
$ export KO_DOCKER_REPO=gcr.io/$PROJECT_ID # Please, change this variable if you are not using Container Registry.
$ gcloud config set run/region us-west1
$ gcloud config set run/platform managed
$ gcloud run deploy $FUNCTION_NAME --image=$(ko publish .) \
 --set-env-vars GKE_CLUSTER_NAME=$GKE_CLUSTER_NAME \
 --set-env-vars PROJECT_ID=$PROJECT_ID
...
Allow unauthenticated invocations to [pod-deleter] (y/N)? N

Deploying container to Cloud Run service [pod-deleter] in project [developerguy-311909] region [us-west1]
✓ Deploying... Done.
 ✓ Creating Revision...
 ✓ Routing traffic...
Done.
Service [pod-deleter] revision [pod-deleter-00002-cej] has been deployed and is serving 100 percent of traffic.
Service URL: https://pod-deleter-uoz6q2wria-uw.a.run.app

Install Falco + Falcosidekick

Now, it is time to set up Falco, Falcosidekick with the Cloud Run output type enabled.

$ kubectl create namespace falco
$ helm repo add falcosecurity https://falcosecurity.github.io/charts
$ helm repo update
$ helm install falco falcosecurity/falco --namespace falco \
--set ebpf.enabled=true \
--set falcosidekick.enabled=true \
--set falcosidekick.webui.enabled=true \
--set falcosidekick.config.gcp.cloudrun.endpoint=$(gcloud run services list --format json | jq -r ".[] | select(.metadata.name==\"$FUNCTION_NAME\") | .status.address.url") \
--set falcosidekick.config.gcp.cloudrun.jwt=$(gcloud auth print-identity-token)

Check the logs to see if Cloud Run output enabled for Falcosidekick.

$ kubectl logs -f falco-falcosidekick-7cd7bc6859-2nd9t --namespace falco
falco-falcosidekick-7cd7bc6859-2nd9t falcosidekick 2021/06/07 16:03:14 [INFO] : Enabled Outputs : [GCPCloudRun WebUI]
falco-falcosidekick-7cd7bc6859-2nd9t falcosidekick 2021/06/07 16:03:14 [INFO] : Falco Sidekick is up and listening on :2801
...

If you see the GCPCloudRun in the list of enabled outputs, you can confirm that everything is working as expected 👍.

Test

Let us start by creating a test pod:

$ kubectl run alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"
pod/alpine created

$ kubectl get pods --namespace default
AME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 11s

Let's run a shell command inside and see what happens:

$ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"
19:27:21 up 50 min, load average: 0.11, 0.12, 0.11

As expected the command returned the output. However, the status of the pod we retrieved is Terminating as follows:

$ kubectl get pods --namespace default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 103s

To investigate further, check the logs of the Cloud Run function from the Google Cloud Console:

Let us check the logs of Falco and Falcosidekick to see what happened.

 $ kubectl logs daemonset/falco --namespace falco
..
{"output":"19:27:21.002873265: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=97c9868ea832 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=97c9868ea832 image=alpine) k8s.ns=default k8s.pod=alpine container=97c9868ea832","priority":"Notice","rule":"Terminal shell in container","time":"2021-04-10T19:27:21.002873265Z", "output_fields": {"container.id":"97c9868ea832","container.image.repository":"alpine","evt.time":1618082841002873265,"k8s.ns.name":"default","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}
...

For Falcosidekick:

$ kubectl logs deployment/falco-falcosidekick --namespace falco
2021/06/07 16:03:15 [INFO] : GCPCloudRun - Post OK (200)
2021/06/07 16:03:15 [INFO] : GCPCloudRun - Function Response : OK
..

Conclusion

We got another way to create a Response Engine with amazing pieces of software from the Open Source world. Of course, it's just the beginning, feel free to share your functions and workflows with the community for creating a library of remediation methods.

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 5: Falcosidekick + Argo

Sun, 23 May 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

The Open Source ecosystem is very vibrant, there are many ways to create a Kubernetes Response Engine based on our dynamic duo, Falco + Falcosidekick. Today, we will use two components of the CNCF project Argo:

Argo Events, will receive events from Falcosidekick and push into it event bus.
Argo Workflow, will listen the event bus and then trigger the workflow if certain criteria are encountered.

Like we did for previous examples with Kubeless, OpenFaas and Knative, we'll address the situation where a shell is spawned in a pod and we want to remediate that by deleting it.

This is how we will set this up:

┌─────────────┐ ┌─────────┐ ┌────────────────┐
│ │ detect │ │ push │ │
│ pwned pod ├───────────► falco ├──────────► falcosidekick ├────┐
│ │ │ │ │ │ │
└──────▲──────┘ └─────────┘ └────────────────┘ │ notify
│ │
│ │
delete │ ┌──────────────┐ ┌───────────────┐ ┌──────▼──────┐
│ │ │ │ │ │ │
└───┤ deletion pod ◄──────────┤ argo workflow │ │ argo events │
│ │ create │ │ │ │
└──────────────┘ └────────────▲──┘ └─┬───────────┘
│ │
trigger │ │ push
│ │
┌─┴─────────────▼──┐
│ bus │
└──────────────────┘

Requirements

We require a kubernetes cluster running at least 1.17 release, helm and kubectl installed in your locale environment.

Installation of Argo Events

We simply follow the official documentation.

kubectl create namespace argo-events
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install-validating-webhook.yaml
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml

Installation of Argo Workflow

Again, the official documentation will help us.

kubectl create namespace argo
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo-workflows/stable/manifests/install.yaml
kubectl patch -n argo cm workflow-controller-configmap -p '{"data": {"containerRuntimeExecutor": "pns"}}'

The kubectl patch is there for allowing the workflows to run in minikube, kind, etc. See docs about Workflow Executors to learn more about.

After a while, you should have access to Argo Workflow UI through a dport-forward:

kubectl port-forward svc/argo-server -n argo 2746:2746 &

The link is https://localhost:2746 (you can ignore the certificate error, we're in a lab 😉).

Creation of the Event Source

We'll use an Event Source with Webhook type. It will receive Falco events from Falcosidekick and push them then into the Event Bus.

This component is pretty easy to understand. Falcosidekick will have to POST the events to an endpoint /falco of a service opened on port 12000. Easy.

cat <<EOF | kubectl apply -n argo-events -f -
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
name: webhook-falco
namespace: argo-events
spec:
service:
ports:
- port: 12000
targetPort: 12000
webhook:
# event-source can run multiple HTTP servers. Simply define a unique port to start a new HTTP server
falco-event:
# port to run HTTP server on
port: "12000"
# endpoint to listen to
endpoint: /falco
# HTTP request method to allow. In this case, only POST requests are accepted
method: POST
EOF

As expected, we now have a new service which will listen events from Falcosidekick on port 12000 and endpoint /falco:

kubectl get svc -n argo-events
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
webhook-falco-eventsource-svc ClusterIP 10.43.117.26 <none> 12000/TCP 11m

Creation of the Sensor

In Argo Events architecture, Sensors are responsible for listening to the Event Bus and triggering something should the criteria we set match. In our case, our Sensor:

listen only events for pushed by webhook-falco Event Source
consider only events where the body (in JSON) contains the value Terminal shell in container for field with key rule, we want to match for only this Falco rule in one word.
trigger a workflow based on a template with our event as input

First, create the Service Account which allows our Sensor will.

cat <<EOF | kubectl apply -n argo-events -f -
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: argo-events
name: sensor-terminal-shell-container-sa
---
# Similarly you can use a ClusterRole and ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: sensor-terminal-shell-container-role
namespace: argo-events
rules:
- apiGroups:
- argoproj.io
verbs:
- "*"
resources:
- workflows
- workflowtemplates
- cronworkflows
- clusterworkflowtemplates
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: sensor-terminal-shell-container-role-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: sensor-terminal-shell-container-role
subjects:
- kind: ServiceAccount
name: sensor-terminal-shell-container-sa
namespace: argo-events
EOF

And now we deploy our Sensor.

cat <<EOF | kubectl apply -n argo-events -f -
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
name: terminal-shell-container
namespace: argo-events
spec:
template:
serviceAccountName: sensor-terminal-shell-container-sa
dependencies:
- name: falco-event
eventSourceName: webhook-falco
eventName: falco-event
filters:
data:
- path: body.rule
type: string
value:
- "Terminal shell in container"
triggers:
- template:
name: delete-pod-trigger
argoWorkflow:
group: argoproj.io
version: v1alpha1
resource: workflows
operation: submit
parameters:
- src:
dependencyName: falco-event
dest: spec.arguments.parameters.0.value
source:
resource:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: delete-pod-
namespace: argo
spec:
workflowTemplateRef:
name: delete-pod-template
arguments:
parameters:
- name: falco-event
value: {}
EOF

Creation of the Workflow Template

There is one piece missing in our Argo stack, we mentioned a template above, we logically need to create it too, with the service account it needs.

cat <<EOF | kubectl apply -n argo -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: delete-pod-sa
namespace: argo
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: delete-pod-sa-cluster-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "list", "delete", "patch", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: delete-pod-sa-cluster-role-binding
roleRef:
kind: ClusterRole
name: delete-pod-sa-cluster-role
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: delete-pod-sa
namespace: argo
EOF

cat <<EOF | kubectl apply -n argo -f -
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
name: delete-pod-template
namespace: argo
spec:
entrypoint: delete-pod
serviceAccountName: delete-pod-sa
arguments:
parameters:
- name: falco-event
value: "{}"
templates:
- name: delete-pod
inputs:
parameters:
- name: falco-event
container:
image: devopps/kubernetes-response-engine-based-on-event-driven-workflow@sha256:22ee203a33fe88f0f99968daebdcea0ca52c8a3d6f7af4c823ed78ac15b7c5db
env:
- name: BODY
value: "{{inputs.parameters.falco-event}}"
EOF

Argo Workflow runs all workflow steps inside their own pods, we'll use for this tutorial a Golang image developped by @developer-guy (who wrote the Part 2 of this series 😄), the sources are there.

At this stage, everything is ready to receive events from Falco and protect our cluster. If you go in Argo Workflow UI you will find the architecture we described at beginning.

Installation of Falco and Falcosidekick

Last but not least, it's time to install our beloved Falco and Falcosidekick and connect them to our shiny new Response Engine.

As with other posts of this series we'll use Helm as conveniant installation method.

kubectl create ns falco
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco \
--namespace falco \
--set falcosidekick.enabled=true \
--set falcosidekick.config.webhook.address=http://webhook-falco-eventsource-svc.argo-events.svc.cluster.local:12000/falco

Remember the service we "mentioned" earlier? This is it in its FQDN format as an endpoint.

Test our Response Engine

Let's delete pwned pod !

We'll simulate a webshell by executing a shell command into a running pod.

kubectl run alpine -n default --image=alpine --restart='Never' -- sh -c "sleep 6000"
kubectl get pods -n default

NAME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 8s

Run a shell command inside.

kubectl exec -i --tty alpine -n default -- sh -c "uptime"
22:03:23 up 44 min, load average: 0.07, 0.13, 0.19

If you're quick enough, you may see the termination of the pod.

kubectl get pods -n default

NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 8s

And in Argo Workflow UI.

👍

Go a little further with Argo

We can even go further by deploying all components with Argo CD, another project from Argo team. You can find out all you need in this repo.

Here a quick demo of the results with the exact same workflow we just created in this tutorial.

Conclusion

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 4: Falcosidekick + Tekton

Fri, 14 May 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

Falcosidekick + Tekton

Earler in this series we have seen how to use Kubeless, OpenFaas and Knative to trigger a pod after getting input from falcosidekick to delete a compromised pod.

In this part I will showcase how we can use Tekton and not have to add any extra complexity to your cluster by adding a serverless runtime.

I won't go through how Tekton works in depth but, you can find a good overview in the official docs. But here is the crash course:

Tekton is built to be reusable.
The smallest part of tekton is a step, a step can be something like this:
- Run unit tests
- Run linting
In a task you can have multiple steps.
A pipeline consist of one or multiple tasks.
To trigger a pipeline to actually run you need a pipelinerun or a trigger-template.

Tekton also supports eventlisteners that is used to listen for webhooks. Normally these webhooks listen for incoming changes to a git repo, for example a PR. But we will use it to listen for Falco events.

You can find all the yaml and code in my repo.

Prerequisites

As always within Kubernetes we need a few tools, I have used the following versions of Helm, Minikube and kubectl in my setup.

Minikube v1.19.0
Helm v3.4.2
kubectl v1.20.5

Provision local Kubernetes Cluster

I'm sure you can use a kind cluster as well to follow along, but falco complained a bit when I tried and I was too lazy to check out what extra flags I need so I went with minikube.

minikube start --cpus 3 --memory 8192 --vm-driver virtualbox

Install Tekton

Install Tekton pipelines and triggers. When doing this in production I recommend the Tekton operator but for now let us use some pure yaml.

kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
kubectl apply --filename https://storage.googleapis.com/tekton-releases/triggers/latest/release.yaml

Within a few seconds you should be able to see a few pods in the tekton-pipelines namespace.

kubectl get pods -n tekton-pipelines
NAME READY STATUS RESTARTS AGE
tekton-pipelines-controller-6b94f5f96-cmf8m 1/1 Running 0 1h
tekton-pipelines-webhook-5bfbbd6475-fmjp4 1/1 Running 0 1h
tekton-triggers-controller-7cbd49fbb8-p4lrz 1/1 Running 0 1h
tekton-triggers-webhook-748fb7778c-w6zxv 1/1 Running 0 1h

If you want a deeper understanding how Tekton triggers work check out the getting-started guide.

Install Falco + Falcosidekick

Create the falco namespace and add the helm repo:

kubectl create namespace falco
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm repo update

For simplicity and long term usability let us create a custom values file and start falco.

cat <<'EOF' >> values.yaml
falcosidekick:
 config:
 webhook:
 address: http://el-falco-listener.falcoresponse.svc.cluster.local:8080
 enabled: true

customRules:
 # Applications which are expected to communicate with the Kubernetes API
 rules_user_known_k8s_api_callers.yaml: |-
 - macro: user_known_contact_k8s_api_server_activities
 condition: >
 (container.image.repository = "gcr.io/tekton-releases/github.com/tektoncd/triggers/cmd/eventlistenersink") or
 (container.image.repository = "quay.io/nissessenap/poddeleter")
EOF

# Install falco
helm upgrade --install falco falcosecurity/falco --namespace falco -f values.yaml

Note the customRules and the webhook address.

We haven't setup this webhook address nor is there currently any reason for us to have customRules for eventlistenersink or poddeleter, but it will come. Both the Tekton event listener and my poddeleter does a few kubernetes API calls and we don't want falco generate alarms for our own infrastructure.

You should be able to see falco and falcosidekick pods in the falco namespace:

kubectl get pods --namespace falco

NAME READY STATUS RESTARTS AGE
falco-44p4v 1/1 Running 0 64m
falco-falcosidekick-779b87f446-8zf9m 1/1 Running 0 2h
falco-falcosidekick-779b87f446-fdk55 1/1 Running 0 2h

Protect me Falco

My current setup is rather harsh and will delete any pods that breaks any falco rule. In the future I plan to make both the go code and the tekton setup better and more flexible, hopefully this is something that we can do in the community.

During this demo I will use the Terminal Shell in container since it's very easy to reproduce.

So how does all this work?

We start a random pod and perform a simple exec.
Falco will notice that a pod have broken the rule
Sends an event to Falcosidekick
Sends a webhook to tekton event-listener
Tekton triggers a new pipeline
A task is started with a small go program that deletes the pod

So let us look at some yaml.

The go code

I have adapted the code that Batuhan Apaydın wrote in Falcosidekick + OpenFaas = a Kubernetes Response Engine, Part 2 to listen for json in an environment variable instead of a http request.

Below you can see the code, in short it does the following:

Check for environment variable BODY.
Unmarshal the data according to the Alert struct.
Setups a kubernetes client, by calling setupKubeClient function.
Calls the deletePod with a kubernetes client, the falcoEvent we gotten and a hash map of critical Namespaces.
Check in the event that we got from falcosidekick and see if the pod that triggered the event is in our critical namespaces hash map.
If it is return to the main and shutdown the application.
Else deletes the pod in the namespace specified in the falcosidekick event.

package main

import (
 "context"
 "encoding/json"
 "log"
 "os"
 "time"

 metaV1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/kubernetes"
 "k8s.io/client-go/rest"
)

// Alert falco data structure
type Alert struct {
 Output string `json:"output"`
 Priority string `json:"priority"`
 Rule string `json:"rule"`
 Time time.Time `json:"time"`
 OutputFields struct {
 ContainerID string `json:"container.id"`
 ContainerImageRepository interface{} `json:"container.image.repository"`
 ContainerImageTag interface{} `json:"container.image.tag"`
 EvtTime int64 `json:"evt.time"`
 FdName string `json:"fd.name"`
 K8SNsName string `json:"k8s.ns.name"`
 K8SPodName string `json:"k8s.pod.name"`
 ProcCmdline string `json:"proc.cmdline"`
 } `json:"output_fields"`
}

func main() {
 criticalNamespaces := map[string]bool{
 "kube-system": true,
 "kube-public": true,
 "kube-node-lease": true,
 "falco": true,
 }

 var falcoEvent Alert

 bodyReq := os.Getenv("BODY")
 if bodyReq == "" {
 log.Fatalf("Need to get environment variable BODY")
 }
 bodyReqByte := []byte(bodyReq)
 err := json.Unmarshal(bodyReqByte, &falcoEvent)
 if err != nil {
 log.Fatalf("The data doesent match the struct %v", err)
 }

 kubeClient, err := setupKubeClient()
 if err != nil {
 log.Fatalf("Unable to create in-cluster config: %v", err)
 }

 err = deletePod(kubeClient, falcoEvent, criticalNamespaces)
 if err != nil {
 log.Fatalf("Unable to delete pod due to err %v", err)
 }
}

// setupKubeClient
func setupKubeClient() (*kubernetes.Clientset, error) {
 config, err := rest.InClusterConfig()
 if err != nil {
 return nil, err
 }

 // creates the clientset
 kubeClient, err := kubernetes.NewForConfig(config)
 if err != nil {
 return nil, err
 }
 return kubeClient, nil
}

// deletePod, if not part of the criticalNamespaces the pod will be deleted
func deletePod(kubeClient *kubernetes.Clientset, falcoEvent Alert, criticalNamespaces map[string]bool) error {
 podName := falcoEvent.OutputFields.K8SPodName
 namespace := falcoEvent.OutputFields.K8SNsName
 log.Printf("PodName: %v & Namespace: %v", podName, namespace)

 log.Printf("Rule: %v", falcoEvent.Rule)
 if criticalNamespaces[namespace] {
 log.Printf("The pod %v won't be deleted due to it's part of the critical ns list: %v ", podName, namespace)
 return nil
 }

 log.Printf("Deleting pod %s from namespace %s", podName, namespace)
 err := kubeClient.CoreV1().Pods(namespace).Delete(context.Background(), podName, metaV1.DeleteOptions{})
 if err != nil {
 return err
 }
 return nil
}

If you rather see it in github.

Now that you know what I will make run in your cluster let us take a look at the Tekton yaml.

Tekton pipeline

Create the falcoresponse namespace to do our tests in.

kubectl create ns falcoresponse

Task

So let us start with the smallest part, the task.

cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
 name: pod-delete
 namespace: falcoresponse
spec:
 params:
 - name: falco-event
 description: The entire msg from falco
 steps:
 - name: pod-delete
 image: quay.io/nissessenap/poddeleter@sha256:ae94ec2c9f005573e31e4944d1055a0dd92ee7594e7e7e36a4540a1811977270
 env:
 - name: BODY
 value: \$(params.falco-event)
EOF

The task needs a input variable falco-event.
The step called pod-delete uses the poddeleter image.
Step pod-delete sets the environment BODY from the input parameter called falco-event.

Pipeline

Here you can see the reusability of tekton. This pipeline can easily add more tasks and other pipelines can use the exact same task as this one.

Just like the task this pipeline expects a parameter called falco-event which it sends in to the pod-delete task.

cat <<EOF | kubectl apply -f -
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
 name: pod-delete-pipeline
 namespace: falcoresponse
spec:
 params:
 - name: falco-event
 description: The entire msg from falco
 tasks:
 - name: run-pod-delete
 taskRef:
 name: pod-delete
 params:
 - name: falco-event
 value: \$(params.falco-event)
EOF

RBAC

We will be using two separate serviceAccounts, one for the event-listener and one for the poddeleter it self.

So let us create these serviceAccounts and give them some access.

Below you can find the event listener RBAC config.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
 name: tekton-triggers-example-sa
 namespace: falcoresponse
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
 name: tekton-triggers-example-minimal
 namespace: falcoresponse
rules:
 # EventListeners need to be able to fetch all namespaced resources
 - apiGroups: ["triggers.tekton.dev"]
 resources:
 ["eventlisteners", "triggerbindings", "triggertemplates", "triggers"]
 verbs: ["get", "list", "watch"]
 - apiGroups: [""]
 # configmaps is needed for updating logging config
 resources: ["configmaps"]
 verbs: ["get", "list", "watch"]
 # Permissions to create resources in associated TriggerTemplates
 - apiGroups: ["tekton.dev"]
 resources: ["pipelineruns", "pipelineresources", "taskruns"]
 verbs: ["create"]
 - apiGroups: [""]
 resources: ["serviceaccounts"]
 verbs: ["impersonate"]
 - apiGroups: ["policy"]
 resources: ["podsecuritypolicies"]
 resourceNames: ["tekton-triggers"]
 verbs: ["use"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
 name: tekton-triggers-example-binding
 namespace: falcoresponse
subjects:
 - kind: ServiceAccount
 name: tekton-triggers-example-sa
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: Role
 name: tekton-triggers-example-minimal
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: tekton-triggers-example-clusterrole
rules:
 # EventListeners need to be able to fetch any clustertriggerbindings
 - apiGroups: ["triggers.tekton.dev"]
 resources: ["clustertriggerbindings", "clusterinterceptors"]
 verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
 name: tekton-triggers-example-clusterbinding
subjects:
 - kind: ServiceAccount
 name: tekton-triggers-example-sa
 namespace: falcoresponse
roleRef:
 apiGroup: rbac.authorization.k8s.io
 kind: ClusterRole
 name: tekton-triggers-example-clusterrole
EOF

And here is the poddeleter serviceAccount:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
 name: falco-pod-delete
 namespace: falcoresponse
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role
rules:
 - apiGroups: [""]
 resources: ["pods"]
 verbs: ["get", "list", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role-binding
roleRef:
 kind: ClusterRole
 name: falco-pod-delete-cluster-role
 apiGroup: rbac.authorization.k8s.io
subjects:
 - kind: ServiceAccount
 name: falco-pod-delete
 namespace: falcoresponse
EOF

Event listener

Finally time to configure the tekton webhook receiver. Just like rest of Tekton the event listener builds on multiple parts.

cat <<EOF | kubectl apply -f -
apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
 name: falco-listener
 namespace: falcoresponse
spec:
 serviceAccountName: tekton-triggers-example-sa
 triggers:
 - name: cel-trig
 bindings:
 - ref: falco-pod-delete-binding
 template:
 ref: falco-pod-delete-trigger-template
EOF

It is possible to expose a event listener using a ingress, this is a rather normal use case if you want github to trigger a pipeline for example.

I cannot stress this enough DO NOT MAKE THE EVENT LISTENER PUBLIC TO THE INTERNET. We haven't added any protection and this task have the power to kill pods in your cluster. Don't give a potential hacker this power!

The event listener is rather complex and can do allot. For example one way to improve this tekton pipeline could be to check for a specific Priority from Falco. This could be done with a cel interceptor and filter on body.Priority.

But for now let us just trigger on everything.

The triggerBinding let us you define what data should be gathered from the incoming webhook. In this case I take the entire request body.

cat <<EOF | kubectl apply -f -
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerBinding
metadata:
 name: falco-pod-delete-binding
 namespace: falcoresponse
spec:
 params:
 - name: falco-event
 value: \$(body)
EOF

We use the TriggerTemplate to call on the pipeline that we defined earlier using the parameter that the TriggerBinding gives us.

cat <<EOF | kubectl apply -f -
apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
 name: falco-pod-delete-trigger-template
 namespace: falcoresponse
 annotations:
 triggers.tekton.dev/old-escape-quotes: "true"
spec:
 params:
 - name: falco-event
 description: The entire msg from falco
 resourcetemplates:
 - apiVersion: tekton.dev/v1beta1
 kind: PipelineRun
 metadata:
 generateName: falco-pod-delete-pipeline-run-
 spec:
 serviceAccountName: falco-pod-delete
 pipelineRef:
 name: pod-delete-pipeline
 params:
 - name: falco-event
 value: \$(tt.params.falco-event)
EOF

Notice the annotations, without it the pipeline will never get triggered.

We define the serviceAccount to use in our pipeline/task, point to the pipeline that we should use. And what parameter to send down to the pipeline, notice the tt in front of parma. This is special syntax for TriggerBindings.

The triggerTemplate was the final piece needed and you should see a pod spinning up in the falcoresponse namespace.

kubectl get pods -n falcoresponse
NAME READY STATUS RESTARTS AGE
el-falco-listener-557786f598-zdmw2 1/1 Running 0 2h

Trigger job

Finally it's time to test our setup.

I would recommend that you start a second terminal for this part.

Terminal 1 follow the falco logs:

kubectl logs -f $(kubectl get pods -l app=falco -o jsonpath="{.items[0].metadata.name}" -n falco) -n falco

Terminal 2 let us trigger the Terminal Shell in container falco rule

# Start a alpine pod
kubectl run alpine --namespace falcoresponse --image=alpine --restart='Never' -- sh -c "sleep 600"
# Trigger the rule breaking behavior
kubectl exec -i --tty alpine --namespace falcoresponse -- sh -c "uptime"
# Watch for pods in falcoresponse namespace
kubectl get pods -n falcoresponse -w

In Terminal 1 you should see something like this:

* Setting up /usr/src links from host
* Running falco-driver-loader for: falco version=0.28.0, driver version=5c0b863ddade7a45568c0ac97d037422c9efb750
* Running falco-driver-loader with: driver=module, compile=yes, download=yes
* Unloading falco module, if present
* Trying to load a system falco module, if present
* Success: falco module found and loaded with modprobe
Sun May 2 18:00:10 2021: Falco version 0.28.0 (driver version 5c0b863ddade7a45568c0ac97d037422c9efb750)
Sun May 2 18:00:10 2021: Falco initialized with configuration file /etc/falco/falco.yaml
Sun May 2 18:00:10 2021: Loading rules from file /etc/falco/falco_rules.yaml:
Sun May 2 18:00:10 2021: Loading rules from file /etc/falco/falco_rules.local.yaml:
Sun May 2 18:00:10 2021: Loading rules from file /etc/falco/rules.d/rules_user_known_k8s_api_callers.yaml:
Sun May 2 18:00:10 2021: Starting internal webserver, listening on port 8765
{"output":"20:24:10.361728219: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=falcoresponse k8s.pod=alpine container=6ac7d190134e shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=6ac7d190134e image=alpine) k8s.ns=falcoresponse k8s.pod=alpine container=6ac7d190134e k8s.ns=falcoresponse k8s.pod=alpine container=6ac7d190134e","priority":"Notice","rule":"Terminal shell in container","time":"2021-05-02T20:24:10.361728219Z", "output_fields": {"container.id":"6ac7d190134e","container.image.repository":"alpine","evt.time":1619987050361728219,"k8s.ns.name":"falcoresponse","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}

In Terminal 2 you should see a pod starting and hopefully Complete without any errors and the alpine pod getting killed.

NAME READY STATUS RESTARTS AGE
alpine 0/1 Terminating 0 1m7s
el-falco-listener-557786f598-znzk9 1/1 Running 0 10m
falco-pod-delete-pipeline-run-w2vf8-run-pod-delete-jlxl7--mk44k 0/1 Completed 0 59s

Hurray our "hacked" pod have been killed

If you look in the logs of the task

kubectl logs -f $(kubectl get pods -l tekton.dev/task=pod-delete -o jsonpath="{.items[0].metadata.name}" -n falcoresponse) -n falcoresponse
2021/05/02 18:11:00 PodName: alpine & Namespace: falcoresponse
2021/05/02 18:11:00 Rule: Terminal shell in container
2021/05/02 18:11:00 Deleting pod alpine from namespace falcoresponse

Conclusion

This was a rather simple example on how we can use the power of tekton together with Falco to protect us from bad actors that is trying to take over pods in our cluster.

As noted during this post there are a lot of potential improvements before this is production ready:

The criticalNamespaces in our go code is currently hard-coded and needs to be input variable of some kind.
We need to be able to delete pods depending on priority level, rule or something similar.
To be able to debug pods we might need to shell in to them, we need a way to ignore pods temporary without the pod getting restarted. Probably a annotation to look for in the pod before deleting it.
And probably many other needs that you can come up with.

If you have any ideas/issues come and share them in the falco slack https://kubernetes.slack.com #falco.

Tekton

If you would like to find out more about Tekton:

Get started in tekton.dev.
Check out the Tekton Project in GitHub.
Meet the maintainers on the TektonCD Slack.
Follow @tektoncd on Twitter.

Falco

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 3: Falcosidekick + Knative

Thu, 13 May 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

As the Cloud Native ecosystem grows and the idea that an integrator can browse the offerings and slap them together like an a la carte menu resonates. We call this Thinking Cloud Native.

Falco already produced events, but in the form of a webhook with bespoke payloads, which is fine, unless you would like to integrate into an ecosystem for event routing. To enable this for Falco we had to think about how these events are moved from producer to consumer via something else. Enter: CloudEvents.

What is CloudEvents? It is a specification for translating an event and the metadata onto a specific protocol and back. What? It lets you think about the event in a generic way without it being tied to particular choices the integration is making today, and with minor effort CloudEvents lets that integration change the protocol choice without changing the meaning of the event.

This lossless property of CloudEvents means the integrator is free to choose middleware that also speaks CloudEvents and has its own choices of persistence and protocol, but the consumer of the event need not be aware of these translations that have happened between the producer and consumer.

There are several choices that support CloudEvents today: Serverless.com Event Gateway, Argo, Google Cloud Pub/Sub, Azure Event Grid, and Knative Eventing. A more full list is at the cloudevents/spec repo.

For this blog post, we are going to focus in on Falco+Knative and see what we can do with that a la carte selection.

Falco+Knative

What is Knative? It is two things: Knative Serving and Knative Eventing. Serving provides a container based scale to zero, scale real big functionality; as well as rainbow deploys, auto-TLS, domain mappings, and various knobs to control concurrency and scale traits. Eventing provides a thin abstraction on top of traditional message brokers (think Kafka or AMQP) that lets you compose your application without considering the message persistence choices in the moment (CloudEvents).

From Knative Eventing, we will use two components: Broker and Trigger. A Knative Eventing Broker represents a event delivery and persistence layer, sort of an eventing mesh. A Knative Eventing Trigger works with the Broker to ask that a consumer be involved with a CloudEvent that matches some specified attributes. So the Broker is the stream of events, the Trigger is how you select events out of the stream and get them delivered.

With Falco producing CloudEvents, we can point our alerts from Falco at the Knative Eventing Broker. Then create a Trigger that selects the Falco event we want to react to. But we also need something to consume the event and react!

From Knative Serving, we can leverage a Knative Serving Service (KService). A KService looks like a lot like a Kubernetes deployment, but it is realized on the cluster as an autoscaling and routable component without the need for manually creating additional Kubernetes Services. KService can run any container as long as it is stateless, and the lifecycle is defined only in the context of an active HTTP request.

To tie this up in a picture,

Falco --[via Sidekick]--> Broker --[via Trigger]--> KService

We are free to make the subscriber of the Trigger be anything we want it to be as long as it is routable from the Broker, and it accepts HTTP POSTs. The request will be a CloudEvent in Binary mode, and Falco makes JSON events, so the payload will be the standard JSON Falco is known for. In-fact, we can replace the KService in with a Kubeless function and it will work.

Demo

To demonstrate this, we have prepared a simple example: We will detect root shell creations and delete that pod.

Prerequisites

K3s Cluster

For this blog post, we a will show the demo using k3s using multipass. Here is a cluster creation commands:

multipass launch --name k3s-leader --cpus 2 --mem 2048M --disk 10G

multipass exec k3s-leader -- /bin/bash -c "curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE=644 sh -"

export K3S_IP_SERVER="https://$(multipass info k3s-leader | grep "IPv4" | awk -F' ' '{print $2}'):6443"

multipass exec k3s-leader -- /bin/bash -c "cat /etc/rancher/k3s/k3s.yaml" | sed "s%https://127.0.0.1:6443%${K3S_IP_SERVER}%g" | sed "s/default/k3s/g" > ./k3s.yaml
export KUBECONFIG=./k3s.yaml

You should get this final output:

(╯°□°)╯︵ multipass launch --name k3s-leader --cpus 2 --mem 2048M --disk 10G
Launched: k3s-leader
(╯°□°)╯︵ multipass exec k3s-leader -- /bin/bash -c "curl -sfL https://get.k3s.io | K3S_KUBECONFIG_MODE=644 sh -"
[INFO] Finding release for channel stable
[INFO] Using v1.20.6+k3s1 as release
[INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.20.6+k3s1/sha256sum-amd64.txt
[INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.20.6+k3s1/k3s
[INFO] Verifying binary download
[INFO] Installing k3s to /usr/local/bin/k3s
[INFO] Creating /usr/local/bin/kubectl symlink to k3s
[INFO] Creating /usr/local/bin/crictl symlink to k3s
[INFO] Creating /usr/local/bin/ctr symlink to k3s
[INFO] Creating killall script /usr/local/bin/k3s-killall.sh
[INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh
[INFO] env: Creating environment file /etc/systemd/system/k3s.service.env
[INFO] systemd: Creating service file /etc/systemd/system/k3s.service
[INFO] systemd: Enabling k3s unit
Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service.
[INFO] systemd: Starting k3s

Now we have a bare bones k3s cluster!

Install Knative

To install the rest of Knative into k3s:

# Installs Knative Serving
kubectl apply -f https://github.com/knative/serving/releases/download/v0.22.0/serving-crds.yaml
kubectl wait --for=condition=Established --all crd
kubectl apply -f https://github.com/knative/serving/releases/download/v0.22.0/serving-core.yaml
kubectl apply -f https://github.com/knative/net-kourier/releases/download/v0.22.0/kourier.yaml
kubectl patch configmap/config-network \
 --namespace knative-serving \
 --type merge \
 --patch '{"data":{"ingress.class":"kourier.ingress.networking.knative.dev"}}'

# Installs Knative Eventing
kubectl apply -f https://github.com/knative/eventing/releases/download/v0.22.0/eventing-crds.yaml
kubectl wait --for=condition=Established --all crd
kubectl apply -f https://github.com/knative/eventing/releases/download/v0.22.0/eventing-core.yaml
kubectl apply -f https://github.com/knative/eventing/releases/download/v0.22.0/in-memory-channel.yaml
kubectl apply -f https://github.com/knative/eventing/releases/download/v0.22.0/mt-channel-broker.yaml

# Creates a default Broker
kubectl create -f - <<EOF
apiVersion: eventing.knative.dev/v1
kind: Broker
metadata:
 name: default
 namespace: default
EOF

See also knative.dev install instructions for installing these into your own cluster.

Falco/Falcosidekick/sidekick UI

We'll use helm to install Falco ,Falcosidekick and Falcosidekick UI.

First, add the falcosecurity helm repo:

helm repo add falcosecurity https://falcosecurity.github.io/charts

helm install falco falcosecurity/falco \
 --create-namespace --namespace falco \
 --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true \
 --set falcosidekick.config.cloudevents.address=http://broker-ingress.knative-eventing.svc.cluster.local/default/default

You should get this output:

NAME: falco
LAST DEPLOYED: Thu Jan 14 23:43:46 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.
No further action should be required.

And you can see your new Falco,Falco Sidekick,Falco Sidekick UI pods:

kubectl get pods -n falco
NAME READY STATUS RESTARTS AGE
falco-jh75c 1/1 Running 0 1d
falco-falcosidekick-554b8859d5-v9xkg 1/1 Running 0 1d
falco-falcosidekick-554b8859d5-x2zkk 1/1 Running 0 1d
falco-falcosidekick-ui-5d747688f9-g96x5 1/1 Running 11 1d

The arguments --set falcosidekick.enabled=true --set falcosidekick.webui.enabled=true enables Falcosidekick and the UI as the below shows:

You can now test it with a typical port-forwarding:

kubectl port-forward svc/falco-falcosidekick-ui -n falco 2802:2802

Drop demo

Install the demo with:

kubectl apply -f https://github.com/n3wscott/falco-drop/releases/download/v0.1.0/release.yaml

This will install a Knative Service that will consume the Falco events sent by falcosidekick (to the broker), some RBAC to enable that service to delete pods, and a Knative Trigger to register this consumer for events from the Knative Eventing Broker.

Consumer KService

The simplified go code in use is like the following:

func main() {
 ...setup context...
 kc := kubeclient.Get(ctx)

 // Make a CloudEvents Client.
 c, _ := cloudevents.NewDefaultClient(p)

 // StartReceiver is blocking, it will deliver events to the inline function.
 c.StartReceiver(ctx, func(event cloudevents.Event) {
 // Filter based on source and type.
 if event.Source() == "falco.org" && event.Type() == "falco.rule.output.v1" {

 // Extract the Falco event Payload
 payload := &FalcoPayload{}
 if err := event.DataAs(payload); err != nil {
 return
 }

 // Only react to "Terminal shell in container" triggered rules.
 if payload.Rule == "Terminal shell in container" {
 if err := kc.CoreV1().Pods(payload.Fields.Namespace).Delete(ctx, payload.Fields.Pod, metav1.DeleteOptions{}); err != nil {
 log.Println("failed to delete pod from event:", err)
 return
 }
 log.Printf("[%s] deleted %s from %s because %s\n", payload.Rule, payload.Fields.Pod, payload.Fields.Namespace, payload.Output)
 }
 }
 })
}

The full implementation can be found in the falco-drop repo.

Pro-tip: if you are developing in Go for Kubernetes, take a look at ko. ko enables containerizing go applications without needing a Dockerfile.

Even though the Trigger only delivers events that match the Trigger filter, it is a good idea to validate the event that the function is receiving, which is why we are validating again in the above code (trust, but verify).

Eventing Triggers

The Trigger configures the Broker for a subscriber to be invoked when the Broker ingresses an event that matches the spec.filter settings.

apiVersion: eventing.knative.dev/v1
kind: Trigger
metadata:
 name: drop
 namespace: default
spec:
 broker: default
 filter:
 attributes:
 source: falco.org
 type: falco.rule.output.v1
 subscriber:
 ref:
 apiVersion: serving.knative.dev/v1
 kind: Service
 name: drop

Note: the kind: Service, name: drop resource is the Knative Service we created above.

Here we are requesting that the broker only deliver events that have the attributes (CloudEvent attributes) of source=falco.org and type=falco.rule.output.v1. These events are delivered to our subscriber KService.

Want to learn how that spec.subscriber.ref works?! It is duck typing and you can learn more, but tl;dr: it is basically doing this (except fancy),

kubectl get ksvc drop -o jsonpath='{.status.address.url}'

Test

First we will create a pod that we can execute code on later:

kubectl run alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"

You should see two pods runing, drop-00001-* and a alpine:

(╯°□°)╯︵ kubectl get pods
NAME READY STATUS RESTARTS AGE
drop-00001-deployment-6b4c5d8bb-m8q4z 2/2 Running 0 4m9s
alpine 1/1 Running 0 39s

Next, we will execute a command in that alpine pod:

kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"

The alpine pod will be terminated by the drop function once the events are processed:

(╯°□°)╯︵ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"
pod/alpine created
(╯°□°)╯︵ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"
 19:29:29 up 17 min, load average: 0.90, 0.85, 0.59
(╯°□°)╯︵ kubectl get pods
NAME READY STATUS RESTARTS AGE
drop-00001-deployment-6b4c5d8bb-m8q4z 2/2 Running 0 10m
alpine 1/1 Terminating 0 5s

Or simply start a hanging shell:

kubectl run alpine-alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"
kubectl exec -i --tty alpine-hang --namespace default -- sh

And the shell will be closed:

(╯°□°)╯︵ kubectl exec -i --tty alpine-hang --namespace default -- sh
/ # command terminated with exit code 137

The event that the drop function is reacting to is a CloudEvent that looks something like this:

Context Attributes,
specversion: 1.0
type: falco.rule.output.v1
source: falco.org
id: f7628198-3822-4c98-ac3f-71770e272a16
time: 2021-01-11T23:46:19.82302759Z
datacontenttype: application/json
Extensions,
foo: bar
priority: Notice
rule: Terminal shell in container
Data,
{
"output": "23:46:19.823027590: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=f29b261f8831 shell=bash parent=runc cmdline=bash -il terminal=34816 container_id=f29b261f8831 image=mysql) k8s.ns=default k8s.pod=mysql-db-7d59548d75-wh44s container=f29b261f8831",
"priority": "Notice",
"rule": "Terminal shell in container",
"time": "2021-01-11T23:46:19.82302759Z",
"output_fields": {
"container.id": "f29b261f8831",
"container.image.repository": "mysql",
"evt.time": 1610408779823027700,
"k8s.ns.name": "default",
"k8s.pod.name": "alpine",
"proc.cmdline": "bash -il",
"proc.name": "bash",
"proc.pname": "runc",
"proc.tty": 34816,
"user.loginuid": -1,
"user.name": "root"
}
}

The KService consumes this event and simply deletes the pod. You can also see this activity in the falcosidekick UI.

Conclusion

Thinking Cloud Native is a mindset of picking the right tool for the job and assembling these tools into something greater than their parts. Falco is a great tool for detection and alerts, it gets really interesting once we can react to those events in ways we never imagined, because integrators are creative and innovative.

What will you build?

Knative

If you would like to find out more about Knative:

Get started in knative.dev.
Check out the Knative Project in GitHub.
Meet the maintainers on the Knative Slack.
Follow @KnativeProject on Twitter.

Falco

If you would like to find out more about Falco:

Get started in Falco.org.
Check out the Falco project in GitHub.
Get involved in the Falco community.
Meet the maintainers on the Falco Slack.
Follow @falco_org on Twitter.

Blog: Kubernetes Response Engine, Part 2: Falcosidekick + OpenFaas

Sun, 11 Apr 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 1 : Falcosidekick + Kubeless

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

We recently talked about a concept called "Kubernetes Response Engine", and we achieved this by using Falco

Falcosidekick + Kubeless. But as you might guess, Falcosidekick project is evolving day after day, which means new outputs are added. With the release 2.22.0, we are proud to support OpenFaaS as a new output for Falcosidekick. This allows us to achieve the same concept, "Kubernetes Response Engine", but this time by using "OpenFaaS" instead of "Kubeless".

In this blog post, we will explain the basic concepts for integrating your own Response Engine into K8S with the stack Falco + Falcosidekick + OpenFaaS.

Prerequisites

We need tools with the following minimum versions to achieve this demo:

Minikube v1.19.0
Helm v3.5.3
kubectl v1.21.0
arkade v0.7.13
faas-cli v0.13.9

Provision local Kubernetes Cluster

There are various ways to provision a local Kubernetes cluster such as, KinD, k3s, k0s, Minikube etc. We are going to use Minikube in this walkthrough.

Let's get provisioned a local Kubernetes cluster:

$ minikube start --cpus 3 --memory 8192 --vm-driver virtualbox
😄 minikube v1.19.0 on Darwin 10.15.7
✨ Using the virtualbox driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🔥 Creating virtualbox VM (CPUs=3, Memory=8192MB, Disk=20000MB) ...
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.4 ...
 ▪ Generating certificates and keys ...
 ▪ Booting up control plane ...
 ▪ Configuring RBAC rules ...
🔎 Verifying Kubernetes components...
 ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟 Enabled addons: storage-provisioner, default-storageclass
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

Install OpenFaaS

OpenFaaS can be deployed into a variety of container orchestrators like Kubernetes, OpenShift, Docker Swarm or into a single host with faasd.

Follow the official documentation for deploying OpenFaaS to Kubernetes.

The fastest option is the tool called arkade to deploy OpenFaaS:

$ arkade install openfaas
Using Kubeconfig: /Users/batuhan.apaydin/.kube/config
Client: x86_64, Darwin
2021/04/10 21:39:29 User dir established as: /Users/batuhan.apaydin/.arkade/
"openfaas" already exists with the same configuration, skipping

Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
...Successfully got an update from the "openfaas" chart repository
Update Complete. ⎈Happy Helming!⎈

VALUES values.yaml
Command: /Users/batuhan.apaydin/.arkade/bin/helm [upgrade --install openfaas openfaas/openfaas --namespace openfaas --values /var/folders/pf/6h9t0mnd4d342ncgpjq_3zl80000gp/T/charts/openfaas/values.yaml --set queueWorker.replicas=1 --set queueWorker.maxInflight=1 --set clusterRole=false --set operator.create=false --set faasnetes.imagePullPolicy=Always --set basicAuthPlugin.replicas=1 --set gateway.replicas=1 --set gateway.directFunctions=false --set openfaasImagePullPolicy=IfNotPresent --set ingressOperator.create=false --set basic_auth=true --set serviceType=NodePort]
Release "openfaas" does not exist. Installing it now.
NAME: openfaas
LAST DEPLOYED: Sat Apr 10 21:39:37 2021
NAMESPACE: openfaas
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
To verify that openfaas has started, run:

 kubectl -n openfaas get deployments -l "release=openfaas, app=openfaas"
=======================================================================
= OpenFaaS has been installed. =
=======================================================================

# Get the faas-cli
curl -SLsf https://cli.openfaas.com | sudo sh

# Forward the gateway to your machine
kubectl rollout status -n openfaas deploy/gateway
kubectl port-forward -n openfaas svc/gateway 8080:8080 &

# If basic auth is enabled, you can now log into your gateway:
PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
echo -n $PASSWORD | faas-cli login --username admin --password-stdin

faas-cli store deploy figlet
faas-cli list

# For Raspberry Pi
faas-cli store list \
 --platform armhf

faas-cli store deploy figlet \
 --platform armhf

# Find out more at:
# https://github.com/openfaas/faas

Thanks for using arkade!

Check if everything is working before moving onto the next step:

$ kubectl get pods --namespace openfaas
NAME READY STATUS RESTARTS AGE
alertmanager-74f9b48464-7gvrj 1/1 Running 0 2m13s
basic-auth-plugin-54bbd886f5-fclgn 1/1 Running 0 2m13s
gateway-6f8f5d5c87-tbxns 2/2 Running 0 2m13s
nats-695bf7587-hcbc2 1/1 Running 0 2m13s
prometheus-577c65f58c-4nvm7 1/1 Running 0 2m13s
queue-worker-b45b85966-g7kpt 1/1 Running 0 2m13s

Now, it is time to deploy our function. The function we are going to deploy basically receives events for an infected pod from the Falcosidekick and deletes it immediately. Before deploying the function we need some permissions to delete Pod. We create a ServiceAccount with right to delete a Pod in any namespace, and we'll associate it to our function:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
 name: falco-pod-delete
 namespace: openfaas-fn
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role
rules:
 - apiGroups: [""]
 resources: ["pods"]
 verbs: ["get", "list", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role-binding
roleRef:
 kind: ClusterRole
 name: falco-pod-delete-cluster-role
 apiGroup: rbac.authorization.k8s.io
subjects:
 - kind: ServiceAccount
 name: falco-pod-delete
 namespace: openfaas-fn
EOF

Now, we are ready to deploy our falco-pod-delete function, log in into OpenFaaS Gateway first:

$ kubectl port-forward -n openfaas svc/gateway 8080:8080 &
$ PASSWORD=$(kubectl get secret -n openfaas basic-auth -o jsonpath="{.data.basic-auth-password}" | base64 --decode; echo)
$echo -n $PASSWORD | faas-cli login --username admin --password-stdin
Calling the OpenFaaS server to validate the credentials...
credentials saved for admin http://127.0.0.1:8080

Install Falco + Falcosidekick

Firstly, we'll create the namespace that will host both Falco and Falcosidekick:

$ kubectl create namespace falco

We add the helm repo:

$ helm repo add falcosecurity https://falcosecurity.github.io/charts
"falcosecurity" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "falcosecurity" chart repository
...Successfully got an update from the "openfaas" chart repository
Update Complete. ⎈Happy Helming!⎈

$ helm upgrade --install falco falcosecurity/falco --namespace falco \
--set falcosidekick.enabled=true \
--set falcosidekick.config.openfaas.functionname="falco-pod-delete"

You should get this output:

Release "falco" does not exist. Installing it now.
NAME: falco
LAST DEPLOYED: Tue Apr 13 10:49:49 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.


No further action should be required.

And you can see your new Falco and Falcosidekickpods:

$ kubectl get pods --namespace falco
NAME READY STATUS RESTARTS AGE
falco-falcosidekick-7779579477-mwsb4 1/1 Running 0 67s
falco-falcosidekick-7779579477-n5v89 1/1 Running 0 67s
falco-p97rw 1/1 Running 0 67s

The argument falcosidekick.enabled=true sets the following settings in Falco for you:

--set falco.jsonOutput=true \
--set falco.httpOutput.enabled=true \
--set falco.httpOutput.url=http://falco-falcosidekick:2801

We check the logs:

$ kubectl logs deployment/falco-falcosidekick -n falco
Found 2 pods, using pod/falcosidekick-5c696d7fd8-9bnnj
2021/04/10 19:21:55 [INFO] : Enabled Outputs : [OpenFaaS]
2021/04/10 19:21:55 [INFO] : Falco Sidekick is up and listening on :2801

OpenFaaS is displayed as enabled output, everything is good 👍.

Install our OpenFaaS function

Our really basic function will receive events from Falco thanks to Falcosidekick, check if the triggered rule is * Terminal Shell in container* , extract the namespace and pod name from the fields of events and delete the according pod:

Basically, the process is:

 +----------+ +---------------+ +----------+
 | Falco +-----------------> Falcosidekick +--------------------> OpenFaaS |
 +----^-----+ sends event +---------------+ triggers +-----+----+
 | |
detects a shell | |
 | |
 +----+-------+ deletes |
 | Pwned Pod <----------------------------------------------------------+
 +------------+

Let's create the function and deploy it:

$ faas-cli template store pull golang-middleware
Fetch templates from repository: https://github.com/openfaas/golang-http-template at
2021/04/10 21:56:34 Attempting to expand templates from https://github.com/openfaas/golang-http-template
2021/04/10 21:56:35 Fetched 2 template(s) : [golang-http golang-middleware] from https://github.com/openfaas/golang-http-template
$ tree -L 2 .
.
└── template
├── golang-http
└── golang-middleware
# Don't forget to set your docker id in the prefix section, mine is devopps.
$ faas-cli new falco-pod-delete --lang golang-middleware --prefix devopps
faas-cli new falco-pod-delete --lang golang-middleware --prefix devopps
Folder: falco-pod-delete created.
___ _____ ____
/ _ \ _ __ ___ _ __ | ___|_ _ __ _/ ___|
| | | | '_ \ / _ \ '_ \| |_ / _` |/ _` \___ \
| |_| | |_) | __/ | | | _| (_| | (_| |___) |
\___/| .__/ \___|_| |_|_| \__,_|\__,_|____/
|_|
Function created in folder: falco-pod-delete
Stack file written: falco-pod-delete.yml
Notes:
You have created a new function which uses Golang 1.13.
To include third-party dependencies, use Go modules and use
"--build-arg GO111MODULE=on" with faas-cli build or configure this
via your stack.yml file.
See more: https://docs.openfaas.com/cli/templates/
For detailed examples:
https://github.com/openfaas-incubator/golang-http-template
$ tree -L 2 .
.
├── falco-pod-delete
│ └── handler.go
├── falco-pod-delete.yml
└── template
├── golang-http
└── golang-middleware

First, replace the falco-pod-delete.yml with the following content:

version: 1.0
provider:
 name: openfaas
 gateway: http://127.0.0.1:8080
functions:
 falco-pod-delete:
 lang: golang-middleware
 handler: ./falco-pod-delete
 image: devopps/falco-pod-delete:latest # be careful this line, it should be your docker id.
 annotations:
 com.openfaas.serviceaccount: falco-pod-delete
 build_args:
 GO111MODULE: on

Once you have edited it, let's continue with the code, create a go.mod.

$ cd falco-pod-delete
$ go mod init falco-pod-delete
go: creating new go.mod: module falco-pod-delete
go: to add module requirements and sums:
go mod tidy

Then, replace the handler.go with the following content:

package function

import (
 "context"
 "encoding/json"
 "io/ioutil"
 "log"
 "net/http"
 "time"

 metaV1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 "k8s.io/client-go/kubernetes"
 "k8s.io/client-go/rest"
)

var kubeClient *kubernetes.Clientset

func init() {

 // creates the in-cluster config
 config, err := rest.InClusterConfig()
 if err != nil {
 panic(err.Error())
 }
 // creates the clientset
 kubeClient, err = kubernetes.NewForConfig(config)
 if err != nil {
 panic(err.Error())
 }
}

type Alert struct {
 Output string `json:"output"`
 Priority string `json:"priority"`
 Rule string `json:"rule"`
 Time time.Time `json:"time"`
 OutputFields struct {
 ContainerID string `json:"container.id"`
 ContainerImageRepository interface{} `json:"container.image.repository"`
 ContainerImageTag interface{} `json:"container.image.tag"`
 EvtTime int64 `json:"evt.time"`
 FdName string `json:"fd.name"`
 K8SNsName string `json:"k8s.ns.name"`
 K8SPodName string `json:"k8s.pod.name"`
 ProcCmdline string `json:"proc.cmdline"`
 } `json:"output_fields"`
}

var CriticalNamespaces = []string{"kube-system", "kube-public", "kube-node-lease", "falco", "openfaas", "openfaas-fn"}

func Handle(w http.ResponseWriter, r *http.Request) {
 var alert Alert

 if r.Body != nil {
 defer r.Body.Close()

 body, _ := ioutil.ReadAll(r.Body)

 json.Unmarshal(body, &alert)

 podName := alert.OutputFields.K8SPodName
 namespace := alert.OutputFields.K8SNsName

 var critical bool
 for _ , ns := range CriticalNamespaces {
 if ns == namespace {
 critical = true
 break
 }
 }

 if !critical {
 log.Printf("Deleting pod %s from namespace %s", podName, namespace)
 kubeClient.CoreV1().Pods(namespace).Delete(context.Background(), podName, metaV1.DeleteOptions{})
 }
 }

 w.WriteHeader(http.StatusOK)
 w.Write([]byte("OK"))
}

After that, update your Go Modules by doing go mod tidy:

$ go mod tidy
go: finding module for package k8s.io/client-go/rest
go: finding module for package k8s.io/apimachinery/pkg/apis/meta/v1
go: finding module for package k8s.io/client-go/kubernetes
go: downloading k8s.io/client-go v0.21.0
go: downloading k8s.io/apimachinery v0.21.0
go: found k8s.io/apimachinery/pkg/apis/meta/v1 in k8s.io/apimachinery v0.21.0
go: found k8s.io/client-go/kubernetes in k8s.io/client-go v0.21.0
go: found k8s.io/client-go/rest in k8s.io/client-go v0.21.0
go: downloading k8s.io/api v0.21.0
go: downloading golang.org/x/net v0.0.0-20210224082022-3d97a244fca7
go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.1.0
go: downloading golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d

Now, you should be able to build, push and deploy your function with faas-cli:

$ cd ..
$ faas-cli up -f falco-pod-delete.yml
[0] > Building falco-pod-delete.
Clearing temporary build folder: ./build/falco-pod-delete/
Preparing: ./falco-pod-delete/ build/falco-pod-delete/function
Building: devopps/falco-pod-delete:latest with golang-middleware template. Please wait..
#1 [internal] load build definition from Dockerfile
#1 sha256:8cd765381aabb90df3bcfbc06f4d175af37d66b85125d463585abc1fc878b94b
#1 transferring dockerfile: 1.81kB done
#1 DONE 0.0s
...

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Image: devopps/falco-pod-delete:latest built.
[0] < Building falco-pod-delete done in 1.31s.
[0] Worker done.

Total build time: 1.31s

[0] > Pushing falco-pod-delete [devopps/falco-pod-delete:latest].
The push refers to repository [docker.io/devopps/falco-pod-delete]
8096edd09fbc: Layer already exists
464d68aca3d9: Layer already exists
e4766ea46ad0: Layer already exists
5f70bf18a086: Layer already exists
a823d50a5b72: Layer already exists
060f21486264: Layer already exists
8ea3b23f387b: Layer already exists
latest: digest: sha256:f94abba203232b97cb2873ef5d60eec31b52d492f3d3ee106d6a9877bf131d95 size: 1782
[0] < Pushing falco-pod-delete [devopps/falco-pod-delete:latest] done.
[0] Worker done.

Deploying: falco-pod-delete.

Deployed. 202 Accepted.
URL: http://127.0.0.1:8080/function/falco-pod-delete

Check if everything is working before moving to the next step:

$ kubectl get pods --namespace openfaas-fn
NAME READY STATUS RESTARTS AGE
falco-pod-delete-7dc9f5fbb8-gbfk7 1/1 Running 0 27s

Test our function

We start by creating a dumb pod:

$ kubectl run alpine --namespace default --image=alpine --restart='Never' -- sh -c "sleep 600"
pod/alpine created

$ kubectl get pods --namespace default
AME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 11s

Let's run a shell command inside and see what happens:

$ kubectl exec -i --tty alpine --namespace default -- sh -c "uptime"

19:27:21 up 50 min, load average: 0.11, 0.12, 0.11

As expected we got the result of our command, but, if we get the status of the pod we retrieve:

$ kubectl get pods --namespace default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 103s

💥 It has been terminated 💥

We can now check the logs of components.

For Falco:

$ kubectl logs daemonset/falco --namespace falco
..
{"output":"19:27:21.002873265: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=97c9868ea832 shell=sh parent=runc cmdline=sh -c uptime terminal=34816 container_id=97c9868ea832 image=alpine) k8s.ns=default k8s.pod=alpine container=97c9868ea832","priority":"Notice","rule":"Terminal shell in container","time":"2021-04-10T19:27:21.002873265Z", "output_fields": {"container.id":"97c9868ea832","container.image.repository":"alpine","evt.time":1618082841002873265,"k8s.ns.name":"default","k8s.pod.name":"alpine","proc.cmdline":"sh -c uptime","proc.name":"sh","proc.pname":"runc","proc.tty":34816,"user.loginuid":-1,"user.name":"root"}}
{"output":"19:27:21.038853452: Notice Unexpected connection to K8s API Server from container (command=handler k8s.ns=openfaas-fn k8s.pod=falco-pod-delete-7dc9f5fbb8-gbfk7 container=12fc4de5ccc3 image=devopps/falco-pod-delete:latest connection=172.17.0.9:43812->10.96.0.1:443) k8s.ns=openfaas-fn k8s.pod=falco-pod-delete-7dc9f5fbb8-gbfk7 container=12fc4de5ccc3","priority":"Notice","rule":"Contact K8S API Server From Container","time":"2021-04-10T19:27:21.038853452Z", "output_fields": {"container.id":"12fc4de5ccc3","container.image.repository":"devopps/falco-pod-delete","container.image.tag":"latest","evt.time":1618082841038853452,"fd.name":"172.17.0.9:43812->10.96.0.1:443","k8s.ns.name":"openfaas-fn","k8s.pod.name":"falco-pod-delete-7dc9f5fbb8-gbfk7","proc.cmdline":"handler"}}

For Falcosidekick:

$ kubectl logs deployment/falcosidekick --namespace falco
2021/04/10 19:27:21 [INFO] : OpenFaas - Post OK (200)
2021/04/10 19:27:21 [INFO] : OpenFaas - Function Response : OK
2021/04/10 19:27:21 [INFO] : OpenFaas - Call Function "falco-pod-delete.openfaas-fn" OK
..

For falco-delete-pod function:

$ faas-cli logs -f --name falco-pod-delete
2021/04/10 19:34:03 Deleting pod alpine from namespace default
2021/04/10 19:34:03 POST / - 200 OK - ContentLength: 2

Conclusion

With this really simple example, we only scratched the surface of possibilities, so don't hesitate to share with us on Slack (https://kubernetes.slack.com #falco) your comments, ideas and successes. You're also always welcome to contribute.

Blog: Kubernetes Response Engine, Part 1: Falcosidekick + Kubeless

Fri, 15 Jan 2021 00:00:00 +0000

This blog post is part of a series of articles about how to create a Kubernetes response engine with Falco, Falcosidekick and a FaaS.

See other posts:

Kubernetes Response Engine, Part 2 : Falcosidekick + OpenFaas

Kubernetes Response Engine, Part 3 : Falcosidekick + Knative

Kubernetes Response Engine, Part 4 : Falcosidekick + Tekton

Kubernetes Response Engine, Part 5 : Falcosidekick + Argo

Kubernetes Response Engine, Part 6 : Falcosidekick + Cloud Run

Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

Two years ago, we presented to you a Kubernetes Response Engine based on Falco. The idea was to trigger Kubeless serverless functions for deleting infected pod, start a Sysdig capture or forward the events to GCP PubSub. See the README.

To avoid maintaining this custom stack, we worked hard with the community to integrate all components into Falcosidekick and to improve the UX. With the last release 2.20.0 we have the finale piece, the integration of Kubeless as native output. More details in our retrospective of 2020.

In this blog post, we will explain the basic concepts for integrating your own Response Engine into K8S with the stack Falco + Falcosidekick + Kubeless.

Requirements

We require a kubernetes cluster running at least 1.17 release and helm and kubectl installed.

Install Kubeless

Follow the official quick start page:

export RELEASE=$(curl -s https://api.github.com/repos/kubeless/kubeless/releases/latest | grep tag_name | cut -d '"' -f 4)
kubectl create ns kubeless
kubectl create -f https://github.com/kubeless/kubeless/releases/download/$RELEASE/kubeless-$RELEASE.yaml

After a few seconds, we can check that the controller is up and running:

kubectl get pods -n kubeless
NAME READY STATUS RESTARTS AGE
kubeless-controller-manager-99459cb67-tb99d 3/3 Running 3 2m34s

Install Falco

Firstly, we'll create the namespace that will use both Falco and Falcosidekick:

kubectl create ns falco

Add the helm repo:

helm repo add falcosecurity https://falcosecurity.github.io/charts

In a real project, you should get the whole chart with helm pull falcosecurity/falco --untar and then configure the values.yaml. For this tutorial, we will try to keep things as easy as possible and set configs directly by helm install command:

helm install falco falcosecurity/falco --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falcosidekick:2801 -n falco

You should get this output:

NAME: falco
LAST DEPLOYED: Thu Jan 14 23:43:46 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
Falco agents are spinning up on each node in your cluster. After a few
seconds, they are going to start monitoring your containers looking for
security issues.
No further action should be required.

And you can see your new Falco pods:

kubectl get pods -n falco
NAME READY STATUS RESTARTS AGE
falco-ctmzg 1/1 Running 0 111s
falco-sfnn8 1/1 Running 0 111s
falco-rrg28 1/1 Running 0 111s

The arguments --set falco.jsonOutput=true --set falco.httpOutput.enabled=true --set falco.httpOutput.url=http://falcosidekick:2801 are there configuring the format of events and the URL where Falco will send them. As Falco and Falcosidekick will be in the same namespace, we can directly use the name of the service (falcosidekick).

Install Falcosidekick

The process is quite the same:

helm install falcosidekick falcosecurity/falcosidekick --set config.kubeless.namespace=kubeless --set config.kubeless.function=delete-pod -n falco

You should get this output:

NAME: falcosidekick
LAST DEPLOYED: Thu Jan 14 23:55:12 2021
NAMESPACE: falco
STATUS: deployed
REVISION: 1
NOTES:
1. Get the application URL by running these commands:
 export POD_NAME=$(kubectl get pods --namespace falco -l "app.kubernetes.io/name=falcosidekick,app.kubernetes.io/instance=falcosidekick" -o jsonpath="{.items[0].metadata.name}")
 kubectl port-forward $POD_NAME 2801:2801
 echo "Visit http://127.0.0.1:2801 to use your application"

We check the logs:

kubectl logs deployment/falcosidekick -n falco
2021/01/14 22:55:31 [INFO] : Enabled Outputs : Kubeless
2021/01/14 22:55:31 [INFO] : Falco Sidekick is up and listening on port 2801

Kubeless is displayed as enabled output, everything is good 👍.

That's it, we really tried to get a nice UX 😉.

Install our Kubeless function

We'll not explain how to write or how to work Kubeless functions, please read the official docs for more information.

Our basic function will receive events from Falco, thanks to Falcosidekick. Check if the triggered rule is Terminal Shell in container. See rule, extract the namespace and pod name from fields of events, and delete the according pod:

from kubernetes import client,config

config.load_incluster_config()

def delete_pod(event, context):
 rule = event['data']['rule'] or None
 output_fields = event['data']['output_fields'] or None

 if rule and rule == "Terminal shell in container" and output_fields:
 if output_fields['k8s.ns.name'] and output_fields['k8s.pod.name']:
 pod = output_fields['k8s.pod.name']
 namespace = output_fields['k8s.ns.name']
 print (f"Deleting pod \"{pod}\" in namespace \"{namespace}\"")
 client.CoreV1Api().delete_namespaced_pod(name=pod, namespace=namespace, body=client.V1DeleteOptions())

Basically, the process is:

 +----------+ +---------------+ +----------+
 | Falco +-----------------> Falcosidekick +--------------------> Kubeless |
 +----^-----+ sends event +---------------+ triggers +-----+----+
 | |
detects a shell | |
 | |
 +----+-------+ deletes |
 | Pwned Pod <----------------------------------------------------------+
 +------------+

Before deploying our function, we need to create a ServiceAccount for it, as it will need the right to delete a pod in any namespace:

cat <<EOF | kubectl apply -n kubeless -f -
apiVersion: v1
kind: ServiceAccount
metadata:
 name: falco-pod-delete
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role
rules:
 - apiGroups: [""]
 resources: ["pods"]
 verbs: ["get", "list", "delete"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
 name: falco-pod-delete-cluster-role-binding
roleRef:
 kind: ClusterRole
 name: falco-pod-delete-cluster-role
 apiGroup: rbac.authorization.k8s.io
subjects:
 - kind: ServiceAccount
 name: falco-pod-delete
 namespace: kubeless
EOF

namespace: kubelessetetion.k8s.io
serviceaccount/falco-pod-delete created
clusterrole.rbac.authorization.k8s.io/falco-pod-delete-cluster-role created
clusterrolebinding.rbac.authorization.k8s.io/falco-pod-delete-cluster-role-binding created

Only remains the installation of our function itself:

cat <<EOF | kubectl apply -n kubeless -f -
apiVersion: kubeless.io/v1beta1
kind: Function
metadata:
 finalizers:
 - kubeless.io/function
 generation: 1
 labels:
 created-by: kubeless
 function: delete-pod
 name: delete-pod
spec:
 checksum: sha256:a68bf570ea30e578e392eab18ca70dbece27bce850a8dbef2586eff55c5c7aa0
 deps: |
 kubernetes>=12.0.1
 function-content-type: text
 function: |-
 from kubernetes import client,config

 config.load_incluster_config()

 def delete_pod(event, context):
 rule = event['data']['rule'] or None
 output_fields = event['data']['output_fields'] or None

 if rule and rule == "Terminal shell in container" and output_fields:
 if output_fields['k8s.ns.name'] and output_fields['k8s.pod.name']:
 pod = output_fields['k8s.pod.name']
 namespace = output_fields['k8s.ns.name']
 print (f"Deleting pod \"{pod}\" in namespace \"{namespace}\"")
 client.CoreV1Api().delete_namespaced_pod(name=pod, namespace=namespace, body=client.V1DeleteOptions())
 handler: delete-pod.delete_pod
 runtime: python3.7
 deployment:
 spec:
 template:
 spec:
 serviceAccountName: falco-pod-delete
EOF

function.kubeless.io/delete-pod created

kubectl get pods -n kubeless

NAME READY STATUS RESTARTS AGE
kubeless-controller-manager-99459cb67-tb99d 3/3 Running 3 3d14h
delete-pod-d6f98f6dd-cw228 1/1 Running 0 2m52s

kubectl get svc -n kubeless

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
delete-pod ClusterIP 10.43.211.201 <none> 8080/TCP 4m38s

Test our function

We start by creating a dumb pod:

kubectl run alpine -n default --image=alpine --restart='Never' -- sh -c "sleep 600"

kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Running 0 9s

Let's run a shell command inside and see what happens:

kubectl exec -i --tty alpine -n default -- sh -c "uptime"

23:44:25 up 1 day, 19:11, load average: 0.87, 0.77, 0.77

As expected, we got the result of our command, but, to get the status of the pod now:

kubectl get pods -n default
NAME READY STATUS RESTARTS AGE
alpine 1/1 Terminating 0 103s

💥 It has been terminated 💥

We can now check the logs of components.

For Falco:

kubectl logs daemonset/falco -n falco

{"output":"23:39:44.834631763: Notice A shell was spawned in a container with an attached terminal (user=root user_loginuid=-1 k8s.ns=default k8s.pod=alpine container=5892b41bcf46 shell=sh parent=<NA> cmdline=sh terminal=34817 container_id=5892b41bcf46 image=<NA>) k8s.ns=default k8s.pod=alpine container=5892b41bcf46","priority":"Notice","rule":"Terminal shell in container","time":"2021-01-14T23:39:44.834631763Z", "output_fields": {"container.id":"5892b41bcf46","container.image.repository":null,"evt.time":1610667584834631763,"k8s.ns.name":"default","k8s.pod.name":"alpine","proc.cmdline":"sh","proc.name":"sh","proc.pname":null,"proc.tty":34817,"user.loginuid":-1,"user.name":"root"}}

For Falcosidekick:

kubectl logs deployment/falcosidekick -n falco

2021/01/14 23:39:45 [INFO] : Kubeless - Post OK (200)
2021/01/14 23:39:45 [INFO] : Kubeless - Function Response :
2021/01/14 23:39:45 [INFO] : Kubeless - Call Function "delete-pod" OK

(Notice, the function returns nothing, this is why the message log is empty)

For delete-pod function:

kubectl logs deployment/delete-pod -n kubeless

10.42.0.31 - - [14/Jan/2021:23:39:45 +0000] "POST / HTTP/1.1" 200 0 "" "Falcosidekick" 0/965744
Deleting pod "alpine" in namespace "default"

Conclusion

With this simple example, we only scratched the surface of possibilities. Everything is possible now, so don't hesitate to share with us on Slack (https://kubernetes.slack.com #falco) your comments, ideas and successes. You're always welcome to contribute.

Bonus: You're running Falcosidekick outside Kubernetes but still want to use the Kubeless output? No problem, you can declare a kubeconfig file to use. See README.

Bonus 2: For people who wants to use Knative in place of Kubeless, it's coming soon 😉

Enjoy

Falco – Integration

Blog: Manage Falco easier with Giant Swarm App Platform

Giant Swarm

Security and simplicity

Falco, the Runtime Security Project

Giant Swarm App Platform

What does it look like on the CLI?

Do you already have a Kubernetes cluster nearby?

Prerequisites: Compiling apptestctl and kubectl-gs

Compiling apptestctl

Compiling kubectl-gs

Deploying App Platform via apptestctl

Registering a Catalog

Installing an App from the App Catalog

Managed Security

Security Pack

Conclusion

Blog: Kubernetes Response Engine, Part 9: Falcosidekick + Fission

Prerequisites

Provision local Kubernetes Cluster

Install Fission

Install Falco + Falcosidekick

Install our Fission Function

Test our function

Conclusion

Blog: Kubernetes Response Engine, Part 8: Falcosidekick + Flux v2

Prerequisites

Tutorial

Provision local Kubernetes Cluster

Install Flux V2 - GitOps Toolkit

Install falco-event-listener

Installing Falco and Falcosidekick

Test

Blog: Kubernetes Response Engine, Part 7: Falcosidekick + Cloud Functions

Prerequisites

Tutorial

Provision Google Kubernetes Engine (GKE) Cluster

Configure IAM Service Accounts

Create Necessary Cluster Role

Create Secret Manager

Deploy Google Cloud Function

Install Falco + Falcosidekick

Test

Conclusion

Blog: Kubernetes Response Engine, Part 6: Falcosidekick + Cloud Run

Prerequisites

Tutorial

Provision GKE (Google Kubernetes Engine) Cluster

Deploy Cloud Run Function

Install Falco + Falcosidekick

Test

Conclusion

Blog: Kubernetes Response Engine, Part 5: Falcosidekick + Argo

Requirements

Installation of Argo Events

Installation of Argo Workflow

Creation of the Event Source

Creation of the Sensor

Creation of the Workflow Template

Installation of Falco and Falcosidekick

Test our Response Engine

Go a little further with Argo

Conclusion

Blog: Kubernetes Response Engine, Part 4: Falcosidekick + Tekton

Falcosidekick + Tekton

Prerequisites

Provision local Kubernetes Cluster

Install Tekton

Install Falco + Falcosidekick

Protect me Falco

The go code

Tekton pipeline

Task

Pipeline

RBAC

Event listener

Trigger job

Conclusion

Tekton

Falco

Prerequisites: Compiling `apptestctl` and `kubectl-gs`

Compiling `apptestctl`

Compiling `kubectl-gs`

Deploying App Platform via `apptestctl`

Registering a `Catalog`