Falco – Case Studies

About: Incepto Medical Case Study

Thu, 07 Mar 2024 00:00:00 +0000

Protect shared clusters for medical imaging

Incepto Medical provides on-demand medical imaging analysis to healthcare facilities. This analysis is based on AI technology manufactured or distributed by Incepto for mammography, X-ray, emergency, CT, MR and PET scanners. Incepto’s partners can also use shared clusters to host their own medical devices and AI models.

A secure, multi-tenant medical imaging service

Incepto Medical specializes in providing medical images analysis using artificial intelligence. Their models enable hospitals, private institutions and doctors to rapidly detect and diagnose cancer and other pathologies. Incepto shared platform can also be used to host and run their partner’s image analysis models.

Their service processes sensitive medical data in a multi-tenant environment. For these reasons, privacy and security are of utmost importance. Falco has been a good fit for their needs.

GPU-enabled Kubernetes deployments

Incepto deploys Kubernetes clusters in AWS in self-managed EC2 instances, having then full control over the infrastructure. The associated AWS services are managed via Terraform and the clusters are deployed using KOPS. The clusters consist of GPU-enabled Ubuntu instances. Each environment (dev/staging/prod) has its own cluster, and each cluster serves multiple customers. Tenant segmentation is carried out by namespace and by using Cillium CNI to manage the network.

Incepto’s API receives pseudonymized medical images from health institutions to comply with GDPR requirements. Falco is deployed as a DaemonSet in each cluster, monitoring both syscalls and Kubernetes audit logs. Falcosidekick runs alongside Falco to forward alerts to Slack. The alerts are segmented by client/partner namespace.

Empowering custom workloads securely

Incepto’s partners can submit their own container images to customize workloads and models. To provide this flexibility, they must ensure that customer workloads behave safely, and do not interfere with workloads from other tenants. For this reason, Falco runs on every node to alert of any policy violations at the OS level or in the Kubernetes environment by inspecting system calls and Kubernetes audit logs. Any drift that is detected in production is instantly reported.

Falco’s flexible rule engine allows Incepto team to continuously improve their detections by developing new custom Falco rules. They built a process to tune and promote Falco rules: Nothing goes into production without a staging period. The development and staging environments enable the testing of the new rules and ensure only relevant alerts will fire in production. Incepto went beyond Kubernetes, and they also created a custom set of Falco rules to detect suspicious activity in their S3 buckets, such as data exfiltration or corruption.

Choosing a security solution for Kubernetes

Incepto's DevSecOps team had previous experience with Falco, so it was a natural choice to adopt it. Adopting Falco was not without its challenges. Incepto team hit issues related to compatibility between Falco’s drivers and the Linux kernel in their VMs, to detection noise related due to different versions of the Nvidia drivers, and forwarding Kubernetes audit logs to Falco. However, once Falco was operational, they were assured that any security event would be detected.

In the end, Falco’s holistic approach to securing workloads gives Incepto the assurance that their customers’ data and proprietary models are safe. In the coming months, Incepto will be studying the feasibility of updating to the latest Falco version. Currently, they are using version 0.31 and the pre-plugin mechanism to ingest Kubernetes Audit logs.

About: Trendyol Case Study

Wed, 26 Jul 2023 00:00:00 +0000

Threat hunting at scale: auditing hundreds of clusters with Falco

Trendyol is a leading e-commerce platform in Turkey, with a fast-growing customer base of over 30 million people and a dedicated team of 4,000+ employees. With an extensive product selection spanning fashion, electronics, home & furniture, food, mother-child, and cosmetics, Trendyol has over 200 million items on its platform and delivers to 27 European countries. The company's impressive growth and broad range of offerings have solidified its position as one of the region's largest and most successful e-commerce platforms.

To ensure a seamless shopping experience for customers, they operate numerous production-grade Kubernetes clusters spread across nine distinct regions. Given the vast size of their infrastructure, it can be difficult to track each component, resource, user, and team promptly.

Background

On an average workday, Trendyol's production environment produces more than 700,000 Kubernetes audit logs per minute. Handling audits efficiently at this scale while minimizing disruption to the cluster's regular operations can pose a challenge.

Moreover, Trendyol required a reliable and scalable indexing and backend storage system that seamlessly integrates with the chosen solution to manage this substantial amount of data.

Trendyol aimed to create a system capable of identifying three specific anti-patterns: unauthorized privilege escalation, attempts to access Kubernetes secrets without proper authorization, and interactive access to running containers in their production environment. To enhance the security of their systems, Trendyol devised a monitoring system as their primary defense mechanism. This system implemented threat-hunting techniques to proactively identify potential security vulnerabilities and issues before they could be exploited.

Journey to Falco

To tackle tracking activities in its production environment, Trendyol created a monitoring solution by leveraging two open source projects: Falco and Fluent Bit. The team successfully developed an audit observability system and implemented alerting mechanisms by utilizing this architecture. These components work together to efficiently identify recurring patterns, enabling improved threat detection and enhanced visibility within the system.

Learn about the Technology

Fluent Bit is an open source tool that is lightweight and high-speed, serving as a data forwarder. It can collect, process, and forward logs and metrics from diverse sources to different destinations in real time. Unlike other popular open source tools, Fluent Bit is specifically designed to be more efficient and consume fewer resources. It can be used as a standalone tool or as a lightweight substitute for Fluentd in larger logging infrastructures.

Falco is an open source project focused on cloud-native runtime security. Its primary purpose is to monitor and identify unexpected behavior within cloud, host, and container-based environments, particularly in Kubernetes. By leveraging various event sources, such as Kubernetes audit logs and kernel system calls, Falco can promptly detect and raise alerts for potential security threats. It offers in-depth insights into the nature of these threats, empowering security teams to respond swiftly and efficiently to mitigate risks.

Events related to the Kernel tell us most of what happens above. Leveraging syscalls and kernel events is essential for monitoring the system and detecting potential security threats, as they play a crucial role in providing essential information about the activities and behavior of processes within the system.

To illustrate this, imagine a movie where a group of bad guys kidnaps a communication satellite to gain an advantage over the good guys. In this scenario, we assume the role of the good guys, and the kernel represents that communication satellite, which grants control and an advantage to whoever possesses it. This parallels how the good guys would use the information from the satellite to gain an advantage and foil the bad guys' plans.

The Architecture

When designing the architecture, Trendyol emphasized achieving optimal performance and scalability. They carefully aligned the architecture with its intended purpose and identified potential bottlenecks that could arise from integrating various components. Additionally, they prioritized factors such as fault tolerance and aimed to maintain vendor independence whenever feasible.

Because the architecture incorporates Fluent Bit and Falco, both active projects within the Cloud Native Computing Foundation (CNCF), vendor independence was not a significant concern. However, it remains important to consider the potential for future replacements and not overlook the possibility of maintaining vendor independence. This architecture is designed to function effectively in tightly coupled and component-independent configurations, offering flexibility and adaptability to suit different needs and potential future changes.

This architecture aims to effectively gather, process effectively, and store system and application logs with a focus on reliability. Fluent Bit is sufficient for the initial tasks of log collection and storage, particularly due to its ability to extract information from all the containers. However, the monitoring system goes beyond basic log processing by incorporating Falco. Falco introduces an additional layer of log processing capabilities by actively detecting Indicators of Compromise (IoC) within the log content. This integration enhances the system's ability to identify security threats and take appropriate actions.

In Trendyol's monitoring system, information is obtained from the Linux kernel of each node and the audit logs generated by the nodes that make up the control plane of each Kubernetes cluster. This could introduce additional complexity in the architecture since only a specific number of nodes within a cluster run instances of the Kubernetes API server. However, the right configuration makes Fluent Bit treat those particular nodes as any other, removing that potentially added complexity. Therefore, capturing and processing the audit logs from the control plane nodes would require a few additional tweaks for the Fluent Bit to retrieve the logs correctly.

Within Cloud-Native environments, a widely recommended approach among practitioners is to treat compute nodes and applications as cattle rather than pets. In this approach, the focus is on scalability and resilience rather than the individual instances themselves. The specific number of instances becomes less relevant, as the system is designed to scale up dynamically or down based on demand.

The Trendyol team recognized the importance of this principle and adopted it seriously. They prioritized building a Cloud-Native architecture that could easily scale and handle varying workloads without being constrained by the fixed number of instances. By embracing this approach, Trendyol ensured its system's flexibility and ability to adapt to changing requirements and fluctuations in demand, leading to improved performance and resilience.

As mentioned, the Kubernetes Audit Logs and Linux Kernel System Calls serve as the primary sources of logs in the monitoring system. While Fluent Bit serves as the main log collector and forwarder, it is incapable of understanding and collecting Linux Kernel System Calls. To address this limitation, Falco is responsible for retrieving information related to syscalls using its dedicated libraries. Falco diligently performs this task, ensuring that the necessary syscall data is captured and made available for further processing and analysis within the monitoring system. By leveraging the combined capabilities of Fluent Bit and Falco, Trendyol achieves comprehensive log collection, including both Kubernetes Audit Logs and Linux Kernel System Calls, enhancing their ability to detect and respond to potential security threats.

Understood, let's pause and focus on the Audit Logs. Falco does have the capability to receive Kubernetes Audit Logs directly. However, as of the time of writing, Kubernetes is limited to sending these logs to a single destination using the webhook backend. If the Audit Logs were sent directly to Falco, it would mean losing the option to enrich the log metadata and store the original ones in a dedicated logging storage for later analysis and compliance purposes.

To overcome this limitation, a potential solution is to implement a log-forwarding mechanism. By setting up a log forwarding system, the Kubernetes Audit Logs can be sent simultaneously to both Falco and the dedicated logging storage simultaneously. This way, Falco can effectively analyze the logs in real-time for immediate threat detection, while the original logs are also preserved in the dedicated storage for future analysis, auditing, and compliance requirements.

By employing this log forwarding approach, Trendyol can maintain the benefits of both real-time monitoring and long-term log storage, ensuring a comprehensive, resilient and robust monitoring system that aligns with operational and compliance needs.

In the solution implemented by Trendyol, the log backend mechanism provided by Kubernetes is utilized for collecting the Audit Logs. This mechanism involves writing the audit events to a file, which is then accessed by Fluent Bit. Fluent Bit retrieves the audit events from the file and sends them to multiple destinations, including the Falco Service and previously mentioned dedicated storage.

During this process, Fluent Bit takes the opportunity to enrich the log stream by adding relevant data such as the cluster origin, region, and the team associated with the cluster. This additional information provides contextual details that can be valuable for analysis and monitoring. By leveraging the Kubernetes Audit log backend mechanism and employing Fluent Bit's log enrichment and distribution capabilities, Trendyol achieves a comprehensive monitoring system that incorporates real-time threat detection with Falco and ensures long-term log storage for operational and compliance needs.

Indeed, with Falco receiving information from the Kernel via native System Calls and the Kubernetes API Audit Logs through Fluent Bit using the K8s Audit Plug-in, the next step is processing this collected data.

Falco excels in real-time processing and analysis of the received logs. It leverages its rule-based detection engine to evaluate the log entries against a set of predefined rules or policies. These rules define specific behaviors or Indicators of Compromise (IoCs) that Falco actively looks for within the log content. When a log entry matches a rule, Falco generates an alert or triggers an action, providing the security team with immediate visibility and an opportunity to respond to potential threats swiftly.

To clarify, the same rule engine will process both sets of events, including the parsed System Calls and the K8s Audit Logs. However, they will each be evaluated against different rules tailored to their specific context.

For the System Calls, Falco will apply rules designed to detect anomalous behavior related to file access, process creation, or other relevant activities. These rules are crafted to identify potential security threats or unauthorized activities within the system. When a System Call event matches one of these rules, Falco will generate an alert that will be sent back to Fluent Bit.

On the other hand, the K8s Audit Logs will undergo analysis using a separate set of rules specifically created to identify Indicators of Compromise (IoC) related to the usage of the Kubernetes API. These rules will focus on detecting actions such as unauthorized access attempts, attempts to access deployment secrets, or the exposure of applications to the external world unnecessarily. Whenever an event from the K8s Audit Logs matches one of these rules, Falco will generate an alert that will be treated as those generated when processing the System calls.

To enable simultaneous forwarding of alerts generated by Falco to multiple destinations, additional configuration and auxiliary tools such as Falco Sidekick would be required. However, Trendyol opted for a different approach to achieve this goal.

Following a similar method used to collect the K8s Audit Logs, Trendyol decided to leverage file-based reading to handle the alerts generated by Falco. Each container within the environment is associated with a file descriptor (FD) in a known location on the node. Fluent Bit, configured accordingly, captures the contents of these container-associated log files and sends them to the dedicated logging storage.

Fluent Bit also adds value by labeling the logs with tags for improved identification during later stages of analysis and processing. This ensures that the alerts can be easily distinguished and categorized based on their origin and other relevant metadata. This labeling assigns a different priority to the Falco logs during further processing.

Indeed, utilizing Fluent Bit for forwarding Falco alerts saves resources on the Falco instances themselves. It eliminates the need to configure each Falco instance individually to send alerts to potentially dynamic destinations. With Fluent Bit's capability to identify the cluster it is running in, it can seamlessly handle forwarding the alerts.

By leveraging Fluent Bit's features and implementing a standardized configuration pattern, Trendyol has optimized resource utilization, facilitated log identification, and established an efficient and scalable monitoring system that can be easily replicated across their infrastructure.

Results

The architecture implemented by Trendyol emphasizes optimal performance, scalability, fault tolerance, and vendor independence. The system collects and processes Kubernetes Audit Logs and Linux Kernel System Calls, using Falco and Fluent Bit to enrich and distribute the logs. Falco applies rule-based detection to evaluate the logs, generating alerts when specific behaviors or Indicators of Compromise (IoC) are detected. By forwarding alerts through Fluent Bit, Trendyol efficiently processes and stores them, ensuring comprehensive monitoring and long-term log storage for real-time threat detection and future analysis.

Overall, Trendyol's use of Falco and Fluent Bit has optimized resource utilization, streamlined configuration, and established a scalable monitoring system. The combination of these open source projects has allowed Trendyol to enhance security, improve visibility, and efficiently track activities within its complex infrastructure. Moreover, Trendyol has achieved a repeatable configuration pattern that can be applied to new clusters, regardless of the region they are created in. This consistency in configuration allows for streamlined deployment and management of the monitoring system across different clusters, simplifying the operational processes and ensuring a consistent security monitoring approach.

About: R6 Security Case Study

Tue, 25 Jul 2023 00:00:00 +0000

R6 Security Leverages Falco to Enhance Their Moving Threat Detection Platform, Phoenix

R6 Security was founded in 2020 to address the unique challenges of securing modern computing environments. Realizing that traditional security offerings based on static signatures were insufficient in today’s cloud native world, R6 Security created Phoenix to offer a more proactive approach to address the ever-changing security challenges around Kubernetes and containers.

R6 Security’s flagship product, Phoenix, leverages Falco’s threat detection capabilities. Phoenix is a security solution for Kubernetes that takes protection to a higher level by introducing the Moving Target Defense (MTD) paradigm. MTD ensures the monitored system is constantly changing and evolving, helping to render hacker’s efforts ineffective. MTD does this by killing and relabeling pods on fixed or random time intervals, automatic reconfiguration and other complex obfuscation actions.

Building on Falco

While building Phoenix, R6 Security received customer feedback that real-time threat detection across various scenarios was a mandatory feature. The R6 Security team was familiar with Falco through connections to the wider Falco community and previous experience with the tool. Additionally, the team had previous experience with commercial security offerings, and they evaluated other open source projects as well.

However, they ultimately settled on Falco for three key reasons:

Its powerful threat detection capabilities
The strength of the Falco community
A proven track record of success with other users

Falco serves as the underlying detection mechanism for Phoenix. When Falco detects suspicious activity, Phoenix’s automated remediation processes kicks off. Phoenix’s custom Kubernetes operator handles the remediation process.

As an example, let’s say someone executes a shell into a running container. Falco running as part of the Phoenix platform, would detect that activity in real time and then forward that event to a Phoenix sidecar running inside the Falco pod. From there the Phoenix Kubernetes operator would receive the event from the sidecar, and perform some sort of remediation. That remediation could include tagging the container as compromised and deactivating it (but not deleting it), so it could be examined later as part of any forensics activities.

This workflow happens even if the tool receives a false positive. Companies spend a lot of time investigating false positives, then deciding if they need to take action. So, rather than trying to investigate these false positive indicators, the system simply performs the automated remediation, keeping users up and running while reducing the time operators need to spend investigating.

The combination of Falco's threat detection capabilities and Phoenix's mitigation features helps effectively address security issues that might evade other security offerings.

The addition of real-time alerting via Falco allows Phoenix to adjust the cluster’s configuration as soon as suspicious activity is detected instead of waiting for the next random update interval.

During the development process R6 Security discovered they needed a way to forward Falco events to their Kubernetes operator. To achieve this they added a sidecar to the Falco pod that receives events from Falco, and then forwards them to their operator. These changes were submitted as a PR and eventually merged into Falco.

Falco has proven to be very accurate in detecting threats, catching around 80-85% of attacks based on red teaming and real-life scenarios. Reaching this level of detection required a multi-pronged approach. There was, of course, fine tuning of the Falco rules, but also the addition of static application security testing. And, it’s important to note, that there is an amount of overhead associated with using Falco and performing the automated remediation. Estimates put this at approximately 1-8% additional CPU or memory. Of course, this varies by workload, and you should plan on doing your own testing.

Conclusion

Today, R6 Security’s Phoenix product is leveraging Falco to help customers deal with attacks that might otherwise have gone undetected. Although there were some challenges integrating the two technologies, the R6 Security team ultimately found Falco's real-time detection capabilities and the strength of the Falco community to be invaluable.