Falco – Security-Concept

Blog: Preventing attacker persistence with Falco on AWS

Mon, 11 Mar 2024 00:00:00 +0000

I recently read an interesting blog on how hackers could use a Lambda function alongside Lex to establish persistence in an AWS account. For those unfamiliar with the term, persistence is when attackers leverage some technique to retain access to systems without being detected. A recent news article cited a study that reported that some Chinese hackers have lurked in systems for up to FIVE YEARS! Luckily for all of us, Falco can be used to detect the exact scenario detailed in the blog and immediately raise an alert.

The blog detailed how an attacker who has gained access to an AWS account could modify an existing Lex-based Lambda function to provide a set of AWS credentials. In short, the attacker modified the function of the Lex-based chatbot to respond whenever a secret phrase was entered with the Lambda’s AWS key ID and secret key.

The author notes there are several ways to establish persistence on AWS, and maybe this wasn’t the most practical, but I still found it a fun exercise. It got me thinking: How could Falco help here? My immediate thought was to use Falco’s AWS Cloudtrail plugin.

The plugin, as the name implies, ingests Cloudtrail events. The events can be evaluated against a set of rules to alert engineers of any suspicious activity. There are currently just over 20 different rules that can be assessed. They include scenarios like creating new users, having someone log into the root account without MFA, changing permissions on an S3 bucket, and, most relevant to our discussion here, modifying a Lambda function.

- rule: Update Lambda Function Code
 desc: Detect updates to a Lambda function code.
 condition:
 ct.name="UpdateFunctionCode20150331v2" and not ct.error exists
 output:
 The code of a Lambda function has been updated.
 (requesting user=%ct.user,
 requesting IP=%ct.srcip,
 AWS region=%ct.region,
 lambda function=%ct.request.functionname)
 priority: WARNING
 tags:
 - cloud
 - MITRE_TA0003_persistence
 - aws_lambda
 source: aws_cloudtrail

Alternatively, you could create custom rules based on a whole list of AWS CloudTrail events. In the below Falco rule, we referenced the Cloudtrail event name UpdateFunctionCode20150331v2. Over 3000 event names can be used with Falco for deep incident response and forensics in the cloud.

So, in the scenario above, whenever the attacker modifies the function, an entry is written to the Cloudtrail logs (note that Cloudtrail is enabled by default, so no extra work is needed to get it running). Those logs would be immediately forwarded to Falco via the Cloudtrail plugin. Falco would evaluate the event against the ruleset and fire off an alert that the rule had been violated.

 The code of a Lambda function has been updated.
(requesting user=mikegcoleman, requesting IP=10.0.01, AWS region=us-west-1,lambda function=AirlinesBusinessLogic)

Upon receiving the alert, an AWS engineer could review whether or not the change had been authorized and act accordingly.

Now, it’s important to note that there are other ways that Lambda functions can be stored. For instance, the function's code can be zipped up and then stored in an S3 bucket. The rules we used above wouldn't cover this scenario. You could very likely craft a rule to ensure that a bucket’s contents have not been modified. We will save that for another blog.

The Falco plugins repository includes the AWS Cloudtrail plugin and plugins for Okta, GitHub, Google Cloud, and more. In fact, I recently published a walkthrough on how to use the Google Cloud plugin. Be sure to check it if you’d like to learn more about how plugins are installed and configured.

Blog: Automate Kubernetes Network Security with Falco Talon

Fri, 09 Feb 2024 00:00:00 +0000

Falco Talon Repository: https://github.com/Falco-Talon/falco-talon
Falco Talon Documentation: https://docs.falco-talon.org/

Falco Talon is currently under active development and remains in the alpha stage; therefore, breaking changes may occur at any time, and the documentation may not always be up to date.

Setting up robust network security in Kubernetes is a challenge that demands both precision and adaptability. NetworkPolicy offers the potential for highly specific network configurations, enabling or blocking traffic based on a comprehensive set of criteria. However, the dynamic nature of network topologies and the complexities of managing policy implementations present ongoing challenges. The need for constant policy updates, especially in response to changing threat landscapes, introduces risks such as the potential for misconfiguration and the unintended dropping of packets.

The Challenge of IP-Based Network Policies

Building network policies around IP addresses is notoriously challenging. For instance, threat feeds, which list known malicious IP addresses, are constantly changing. An IP address associated with a malicious entity one week might be reassigned and deemed safe the next. This fluidity necessitates an agile approach to network policy management, integrating solutions like NetworkSets to dynamically update policies based on the latest intelligence. However, the sheer volume of threat intelligence feeds – from Tor IP lists to cryptomining blocklists – complicates this integration, making it a daunting task to maintain accurate network controls.

Here, Falco Talon emerges as a transformative solution. By leveraging Falco's detection capabilities, such as identifying Outbound Connections to C2 Servers, Falco Talon can instantly update Kubernetes network policies to block all egress traffic except allowed CIDR ranges. This is facilitated through the kubernetes:networkpolicy Talon action, demonstrating a seamless integration of dynamic threat detection with network policy enforcement.

- action: Disable outbound connections
actionner: kubernetes:networkpolicy
parameters:
allow:
- "192.168.1.0/24"
- "172.17.0.0/16"
- "10.0.0.0/32"
- rule: Suspicious outbound connection
match:
rules:
- Outbound Connections to C2 Servers
actions:
- action: Disable outbound connections

While this would certainly block the egress network connections, it’s not really convenient for some organizations as it’s a scorched-earth approach to blocking traffic but still relies on an IP-based allowlist on each Talon rule - rather than blocking the actual suspicious IP address specified in the Falco rule. Instead, we will propose the use of labels as a response action in Talon to better isolate network traffic at runtime.

Shifting from IPs to Labels for Network Security

While IP-based blocking is effective in response to specific threats detected by Falco, it's not the most scalable solution for ongoing network policy management in production environments. An alternative approach focuses on using labels to create quarantine-style network policies. This method involves configuring a network policy that applies a default-deny stance on all ingress and egress traffic for pods matching certain labels, effectively isolating potentially compromised workloads. This can easily be achieved with the below one-liner:

kubectl label pod <pod-name> -n <namespace-name> quarantine=true

This is certainly a cleaner approach than the previous enforced network policy implementation, but the challenge here is the manual process of labeling those suspected workloads, which can be cumbersome and slow in response to emerging threats. How many minutes will it take our security team to enforce this label action, and what happens if this happens over the weekend?

Falco Talon addresses this gap with its kubernetes:labelize response action. Upon detecting a threat, such as the Detect outbound connections to common miner pool ports, Talon can automatically apply a quarantine:true label to the affected pod, triggering the enforcement of the quarantine network policy in real-time. This capability not only enhances the speed and efficiency of response actions but also underscores the power of integrating dynamic threat detection with network policy enforcement.

- action: Quarantine Pod in Network Policy
actionner: kubernetes:labelize
parameters:
labels:
quarantine: true
- rule: Suspicious outbound connection
match:
rules:
- Detect outbound connections to common miner pool ports
actions:
- action: Quarantine Pod in Network Policy

While Talon can apply the label, you still need a component that will enforce the quarantine. Most CNI's can do this, but for the purpose of this blog I'll add an example with Calico:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
name: quarantine
spec:
selector: quarantine == "true"
ingress:
- action: Deny
source: {}
destination: {}
egress:
- action: Deny
source: {}
destination: {}
types:
- Ingress
- Egress

Conclusion

The integration of Falco Talon into Kubernetes network security strategies represents a significant advancement in the field. By automating the enforcement of network policies based on real-time threat detection, Falco Talon simplifies the complexities associated with managing network security in a constantly evolving landscape. Whether responding to immediate threats through IP-based policies or proactively isolating workloads with label-based quarantine policies, Falco Talon provides a flexible, powerful tool for enhancing the security and resilience of Kubernetes environments. As organizations navigate the challenges of cloud-native security, solutions like Falco Talon offer a beacon of adaptability and effectiveness, ensuring that network security keeps pace with the dynamic nature of containerized deployments.

Blog: Install and Test Atomic Red in Kubernetes

Tue, 12 Dec 2023 00:00:00 +0000

In cloud-native environments, where applications scale up and down much faster than traditional monolithic application architectures, the ability to proactively identify and respond to threats in real time is paramount. As more organizations embrace cloud-native architectures for application delivery, more robust security measures need to be introduced. In this blog post, we delve into the dynamic realm of Kubernetes threat detection by exploring how open source Falco can seamlessly detect Atomic Red Team tests in real time within Kubernetes environments.

Atomic Red Team is a powerful framework designed to simulate real-world attacks, providing organizations with a controlled environment to validate the effectiveness of their security measures. We take this a step further by deploying Atomic Red to Kubernetes with a single command, creating a realistic testing ground for evaluating the responsiveness of Falco.

Our journey begins with the effortless deployment of Atomic Red to Kubernetes, showcasing the simplicity and efficiency of orchestrating security testing within containerized environments. Once deployed, we invoke specific Atomic Red Team tests, simulating a range of threat scenarios. The true test lies in Falco's ability to detect these threats in line with the MITRE ATT&CK framework, a globally-recognized matrix mapping adversary techniques to defensive tactics.

This exploration is not just about identifying threats; it's a collaborative effort to enhance Falco's coverage. Should we identify any gaps in detection, we dive deeper, revising the executed techniques and crafting custom rules. This iterative process aims to extend our MITRE ATT&CK coverage, aligning Falco with the industry's best practices for threat detection and mitigation.

Deploying Atomic Red Team

To avoid any potential service disruption in production environments, we recommend installing Atomic Red in a test lab environment, or at least a staging environment of Kubernetes. We have a step-by-step video for installing Atomic Red on Youtube

Before we start the deployment, remember to create the atomic-red network namespace.

kubectl create ns atomic-red

A single pod will be deployed with privileged set to true. Atomic Red requires admin-level securityContext to perform certain actions that require elevated permissions.

kubectl apply -f - <<EOF
apiVersion: apps/v1
kind: Deployment
metadata:
 name: atomicred
 namespace: atomic-red
 labels:
 app: atomicred
spec:
 replicas: 1
 selector:
 matchLabels:
 app: atomicred
 template:
 metadata:
 labels:
 app: atomicred
 spec:
 containers:
 - name: atomicred
 image: issif/atomic-red:latest
 imagePullPolicy: "IfNotPresent"
 command: ["sleep", "3560d"]
 securityContext:
 privileged: true
 nodeSelector:
 kubernetes.io/os: linux
EOF

Note: This creates a pod called 'atomicred' in the 'atomic-red' network namespace. You can check on the state of the installation with the below command:

kubectl get pods -n atomic-red

If you ever want to remove the Atomic Red project from your Kubernetes cluster, simply run:

kubectl delete deployment atomicred -n atomic-red

Familiarize Yourself with Atomic Red Tests

Once deployed, you will need to shell into the Atomic Red pod to perform the following test scenarios. This might seem a little confusing, but Atomic Red was developed with PowerShell in mind, so the below instructions ask the user to shell into a container, and once they are in the running pod, they must run Powershell to import and invoke the various Atomic Test Scenarios.

Once you are familiar with this logic, you’ll find Atomic Red is a truly simple security simulation tool.

kubectl exec -it -n atomic-red deploy/atomicred -- bash

As mentioned, you need to run Powershell once you are in the Atomic Red pod:

pwsh

Now, you can finally load the Atomic Red Team module:

Import-Module "~/AtomicRedTeam/invoke-atomicredteam/Invoke-AtomicRedTeam.psd1" -Force

Check the details of the TTPs:

Invoke-AtomicTest T1070.004 -ShowDetails

Check the prerequisites to ensure the test conditions are right:

Invoke-AtomicTest T1070.004 -GetPreReqs

You can see in the below screenshot, there are no prerequisites required to perform these tests. As a result, we can invoke the bulk file deletion test scenario.

Remove the feature flags to execute the test simulation.

Invoke-AtomicTest T1070.004

This test will attempt to delete individual files or individual directories. When we have Falco installed, this Atomic test should trigger the 'Warning bulk data removed from disk' rule by default. Next, we discuss Falco’s installation.

Congrats! Now that we know how Atomic Red works, let’s install Falco and run it side-by-side against Atomic Red to prove it detects these tests in real time. We will need to open two terminal windows to see the real-time response detections.

Installing & Testing Falco

For this lab guide, we can install Falco via Helm on a fixed version prior to the segregation of rules into different rules feeds, such as 'incubating', 'sandbox' and 'stable'. The reason we are doing this is to ensure all Falco rules are accessible in our lab scenario. To use the latest version of Falco, simply remove the '--version' feature flag from the Helm install script.

helm install falco falcosecurity/falco --namespace falco \
--create-namespace \
--set tty=true \
--version 3.3.0

Just like the Atomic Red deployment, we need to monitor the progress of the Falco installation. The pods will change state a few times during the installation, but should eventually all be in a 'RUNNING' status after about a minute or so.

Please use the below command to check on the status change of Falco pods:

kubectl get pods -n falco -w

Once Falco is installed, we can track the events as they are generated using the following command in the second terminal window.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco

Jump back into the first terminal window and re-run the bulk file deletion Atomic Test - 'T1070.004':

Invoke-AtomicTest T1070.004

You’re going to identify certain noise in the detection rules. For example, all Atomic Tests are run under the 'Root' user, therefore, we will always get a detection for scripts running under root. To ignore this noise, let’s instead just check for the specific Falco rule we are looking to detect:

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Bulk data has been removed from disk'

Hurrah! We see the exact detection matching the context of the Atomic Red test scenario. Let’s move on to the next Atomic Test to invoke. There are a bunch of test scenarios for Linux that you can test out today.

Check out the list on the official Atomic Red Team Github project.

T1556.003 Modify Authentication Process

In this scenario, Atomic Red generates three Pluggable Authentication Modules (PAM): two malicious PAM rules for Linux and FreeBSD, as well as a malicious PAM module for Linux. These programs can be used to open and read sensitive file content, and we can agree that they are non-trusted programs. Again, we have an out-of-the-box rule for these activities:

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Sensitive file opened for reading by non-trusted program'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1556.003

T1036.005 Masquerading: Match Legitimate Name or Location

This test scenario executes a process from a directory masquerading as the current parent directory.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Executing binary not part of base'

Now, it's time to simulate our threat.

Invoke-AtomicTest T1036.005

You can see that in the left terminal window, there is an echo message in the terminal saying '”Hello from the Atomic Red Team.”' Any string output in the command line can also be detected in Falco’s outputs.

T1070.002 Indicator Removal on Host

Adversaries may clear system logs to hide evidence of an intrusion. macOS and Linux both keep track of system or user-initiated actions via system logs. The majority of native system logging is stored under the '/var/log/' directory.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Log files were tampered'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1070.002

T1070.003 Clear Command History

In addition to clearing system logs, an adversary may clear the command history of a compromised account to conceal the actions undertaken during an intrusion. Various command interpreters keep track of the commands users type in their terminals so that users can retrace what they've done.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Shell history had been deleted or renamed'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1070.003

You can see from the below screenshot that four different operations were performed. Therefore, four unique Falco detections were triggered on those individual attempts to clear the command line history.

T1014 Loadable Kernel Module Based Rootkit

Adversaries may use Rootkits to hide the presence of programs, files, network connections, services, drivers, and other system components. Rootkits are programs that hide the existence of malware by intercepting/hooking and modifying operating system API calls that supply system information.

Rootkits may reside at the user or kernel level in the operating system or lower, to include a hypervisor, Master Boot Record, or System Firmware. As such, it’s critical that Falco detects Rootkits in real time.

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Linux Kernel Module injection from container detected'

Now, it's time to simulate our threat:

Invoke-AtomicTest T1014

Falco is detecting the Linux kernel module injection attempt, whether it was a successful execution or not.

T1037.004 [CUSTOM RULE] Boot Initialization - RC Scripts

Adversaries may establish persistence by modifying RC scripts that are executed during a Unix-like system’s startup. These files allow system administrators to map and start custom services at startup for different run levels. RC scripts require root privileges to modify.

Command to simulate the 'T1037.004' test:

Invoke-AtomicTest T1037.004

You’ll notice that this is the first test where we don’t get a useful Falco detection related to the threat. As a result, we need to create a 'custom-rules.yaml' file with the custom Falco rule for detecting boot initialization scripts.

cat > custom-rules.yaml <<EOF
customRules:
custom_rules.yaml: |-
- rule: Base64-encoded Python Script Execution
desc: >
This rule detects base64-encoded Python scripts on command line arguments.
condition: >
spawned_process and (
((proc.cmdline contains "python -c" or proc.cmdline contains "python3 -c" or proc.cmdline contains "python2 -c") and (proc.cmdline contains "echo" or proc.cmdline icontains "base64")) or ((proc.cmdline contains "import" and proc.cmdline contains "base64" and proc.cmdline contains "decode")))
output: >
Potentially malicious Python script encoded on command line
(proc.cmdline=%proc.cmdline user.name=%user.name proc.name=%proc.name
proc.pname=%proc.pname evt.type=%evt.type gparent=%proc.aname[2]
ggparent=%proc.aname[3] gggparent=%proc.aname[4] evt.res=%evt.res
container.id=%container.id container.name=%container.name file=%fd.name)
priority: warning
tags:
- T1037.004
- MITRE_Defense_Evasion
source: syscall
EOF

Adversaries can establish persistence by adding a malicious binary path or shell commands to 'rc.local', 'rc.common', and other RC scripts specific to the Unix-like distribution. Upon reboot, the system executes the script's contents as root, resulting in persistence.

Let’s try upgrading Falco to reflect the 'custom-rules.yaml' file:

helm upgrade falco falcosecurity/falco \
-n falco \
--set tty=true \
--version 3.3.0 \
--reuse-values \
-f custom-rules.yaml

Granted, there’s no obvious formatting issues when creating the 'custom-rules.yaml' manifest, you should be able to successfully upgrade Falco and the pods should be in a 'RUNNING' state. If there was an issue with the custom rules file, the Falco pod state will likely change to 'CrashLoopBackOff'.

Let’s see if we detect the Atomic test after upgrading Falco with the newly-created custom rule. Remember to have Falco running in a second terminal window with the following command:

kubectl logs -f --tail=0 -n falco -c falco -l app.kubernetes.io/name=falco | grep 'Potentially malicious Python script'

We can simulate the technique ID 'T1037.004' one last time:

Invoke-AtomicTest T1037.004

Hurrah! We detected the boot initialization scripts with the above command. To recap, adversaries looking to abuse those RC scripts are especially effective for lightweight Unix-like distributions using the root user as default, such as IoT or embedded systems. If you are wondering why the Falco rule was specifically looking at Base64-encoded Python scripts, well, we need to look back at the details associated with the Atomic test simulation.

Invoke-AtomicTest T1037.004 -ShowDetails

We can see from the command that it is using the 'python3' command to run Python scripts. However, the script itself is executed as a base64-encoded string to evade some traditional detection tools.

Conclusion

In conclusion, this article serves as a comprehensive guide aimed at enhancing knowledge in cloud-native threat simulation and detection. Its primary audience includes security practitioners, DevOps teams, and anyone involved in securing Kubernetes environments. The focal point is the utilization of Atomic Red for deploying test simulations aligned with the MITRE ATT&CK framework and leveraging Falco to detect these threats in real time.

The article not only illustrates how Atomic Red can be instrumental in improving the rules shipped with Falco, but also emphasizes its role in identifying potential gaps in coverage. By doing so, it offers a valuable resource for customers seeking alternative methods to validate the effectiveness of their Falco rules and, consequently, fortify the security posture of their Kubernetes deployments.

The notion of creating a realistic testing ground for evaluating the responsiveness of Falco is particularly beneficial for DevOps teams. This facet addresses the need for hands-on experience in threat simulation, adversarial approaches, and understanding the intricacies of the MITRE ATT&CK framework. In providing this practical testing environment, the article empowers DevOps teams to proactively enhance their security measures, ensuring robust coverage and responsiveness in the dynamic landscape of Kubernetes threat detection.