Test for rule/macro having unknown source

The rules file has a rule/macro with an unknown source as well as a rule
that matches a trace file, and verifies that the rule/macro with the
unknown source doesn't interfere with rule loading/event matching.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>

ADOPTERS.md

@@ -26,5 +26,6 @@ This is a list of production adopters of Falco (in alphabetical order):
 
 * [Sumo Logic](https://www.sumologic.com/) - Sumo Logic provides a SaaS based log aggregation service that provides dashboards and applications to easily identify and analyze problems in your application and infrastructure. Sumo Logic provides native integrations for many CNCF projects, such as Falco, that allows end users to easily collect Falco events and analyze Falco events on DecSecOps focused dashboards.
 
-*  [Sysdig](https://www.sysdig.com/) Sysdig originally created Falco in 2016 to detect unexpected or suspicious activity using a rules engine on top of the data that comes from the sysdig kernel system call probe. Sysdig provides tooling to help with vulnerability management, compliance, detection, incident response and forensics in Cloud-native environments. Sysdig Secure has extended Falco to include: a rule library, the ability to update macros, lists & rules via the user interface and API, automated tuning of rules, and rule creation based on profiling known system behavior. On top of the basic Falco rules, Sysdig Secure implements the concept of a "Security policy" that can comprise several rules which are evaluated for a user-defined infrastructure scope like Kubernetes namespaces, OpenShift clusters, deployment workload, cloud regions etc.
+* [Swissblock Technologies](https://swissblock.net/) At Swissblock we connect the dots by combining cutting-edge algorithmic trading strategies with in-depth market analysis. We route all Falco events to our control systems, both monitoring and logging. Being able to deeply analyse alerts, we can understand what is running on our Kubernetes clusters and check against security policies, specifically defined for each workload. A set of alarms notifies us in case of critical events, letting us react fast. In the near future we plan to build a little application to route Kubernetes internal events directly to Falco, fully leveraging Falco PodSecurityPolicies analyses.
 
+* [Sysdig](https://www.sysdig.com/) Sysdig originally created Falco in 2016 to detect unexpected or suspicious activity using a rules engine on top of the data that comes from the sysdig kernel system call probe. Sysdig provides tooling to help with vulnerability management, compliance, detection, incident response and forensics in Cloud-native environments. Sysdig Secure has extended Falco to include: a rule library, the ability to update macros, lists & rules via the user interface and API, automated tuning of rules, and rule creation based on profiling known system behavior. On top of the basic Falco rules, Sysdig Secure implements the concept of a "Security policy" that can comprise several rules which are evaluated for a user-defined infrastructure scope like Kubernetes namespaces, OpenShift clusters, deployment workload, cloud regions etc.

docker/README.md

@@ -1,17 +1,16 @@
 # Falco Dockerfiles
 
-This directory contains various ways to package Falco as a container and related tools. 
+This directory contains various ways to package Falco as a container and related tools.
 
 ## Currently Supported Images
 
 | Name | Directory | Description |
 |---|---|---|
-| [falcosecurity/falco:latest](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:master](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/falco | Falco (DEB built from git tag or from the master) with all the building toolchain. | 
-| [falcosecurity/falco-driver-loader:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:master](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader) | docker/driver-loader | `falco-driver-loader` as entrypoint with the building toolchain. | 
-| [falcosecurity/falco-no-driver:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-no-driver), [falcosecurity/falco-no-driver:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco-no-driver),[falcosecurity/falco-no-driver:master](https://hub.docker.com/repository/docker/falcosecurity/falco-no-driver) | docker/no-driver | Falco (TGZ built from git tag or from the master) without the building toolchain. | 
-| [falcosecurity/falco-builder:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-builder) | docker/builder | The complete build tool chain for compiling Falco from source. See [the documentation](https://falco.org/docs/source/) for more details on building from source. Used to build Falco (CI). | 
-| [falcosecurity/falco-tester:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-tester) | docker/tester | Container image for running the Falco test suite. Used to run Falco integration tests (CI). | 
+| [falcosecurity/falco:latest](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco), [falcosecurity/falco:master](https://hub.docker.com/repository/docker/falcosecurity/falco) | docker/falco | Falco (DEB built from git tag or from the master) with all the building toolchain. |
+| [falcosecurity/falco-driver-loader:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader), [falcosecurity/falco-driver-loader:master](https://hub.docker.com/repository/docker/falcosecurity/falco-driver-loader) | docker/driver-loader | `falco-driver-loader` as entrypoint with the building toolchain. |
+| [falcosecurity/falco-no-driver:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-no-driver), [falcosecurity/falco-no-driver:_tag_](https://hub.docker.com/repository/docker/falcosecurity/falco-no-driver),[falcosecurity/falco-no-driver:master](https://hub.docker.com/repository/docker/falcosecurity/falco-no-driver) | docker/no-driver | Falco (TGZ built from git tag or from the master) without the building toolchain. |
+| [falcosecurity/falco-builder:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-builder) | docker/builder | The complete build tool chain for compiling Falco from source. See [the documentation](https://falco.org/docs/getting-started/source/) for more details on building from source. Used to build Falco (CI). |
+| [falcosecurity/falco-tester:latest](https://hub.docker.com/repository/docker/falcosecurity/falco-tester) | docker/tester | Container image for running the Falco test suite. Used to run Falco integration tests (CI). |
 | _to not be published_ | docker/local | Built on-the-fly and used by falco-tester. |
 
 > Note: `falco-builder`, `falco-tester` (and the `docker/local` image that it's built on the fly) are not integrated into the release process because they are development and CI tools that need to be manually pushed only when updated.
-

proposals/2021019-libraries-donation.md

@@ -0,0 +1,167 @@
+# OSS Libraries Donation Plan
+
+## Summary
+
+Sysdig Inc. intends to donate **libsinsp**, **libscap**, the **kernel module driver** and the **eBPF driver sources** by moving them to the Falco project.
+
+This means that some parts of the [draios/sysdig](https://github.com/draios/sysdig) repository will be moved to a new GitHub repository called [falcosecurity/libs](https://github.com/falcosecurity/libs).
+
+This plan aims to describe and clarify the terms and goals to get the donation done.
+
+## Motivation
+
+There are two main OSS projects using the libraries and drivers that we are aware of:
+
+- [sysdig](https://github.com/draios/sysdig)  the command line tool
+- [Falco](https:/github.com/falcosecurity/falco), the CNCF project.
+
+Since the Falco project is a heavy user of the libraries, a lot more than the sysdig cli tool, Sysdig (the company) decided to donate the libraries and the driver to the Falco community.
+
+Sysdig (the command line tool) will continue to use the libraries now provided by the Falco community underneath.
+
+This change is win-win for both parties because of the following reasons:
+
+- The Falco community owns the source code of the three most important parts of the software it distributes.
+  - Right now it is "only" an engine on top of the libraries. This **donation** helps in making the scope of the Falco project broader. Having the majority of the source code under an **open governance** in the same organization gives the Falco project more contribution opportunities, helps it in **evolving independently** and makes the whole Falco community a strong owner of the processes and decision making regarding those crucial parts.
+
+- Given the previous point, Sysdig (the command line tool) will benefit from the now **extended contributors base**
+
+- Sysdig (the company) can now focus on the user experience and user space features
+
+- **Contributions** to the libraries and drivers will be **easier** to spread across the Falco community
+
+- By being donated, with their own **release process**, **release artifacts**, and **documentation**, the libraries can now live on their own and possibly be used directly in other projects by becoming fundamental pieces for their success.
+
+## Goals
+
+There are many sub-projects and each of them interacts in a different way in this donation.
+
+Let's see the goals per sub-project.
+
+### libsinsp
+
+1. Extract libsinsp from `draios/sysdig/userspace/libsinsp` (keeping the commit history) into [falcosecurity/libs](https://github.com/falcosecurity/libs)
+
+2. The migration comes first, then we can do additional PRs for the points below so that we do only one thing at a time and keep the history linear
+
+3. Keep the same code, refactorings will need to be done in subsequent PRs and approved separately
+
+4. Adapt the CMake and build files
+
+5. Install [poiana](https://github.com/poiana) and its workflows on it
+
+6. Define the `OWNERS`
+
+   - Owners are chosen from the current major contributors (considering the past two years) to this project, given their availability, commitment is key
+
+7. When possible, migrate issues and PRs to the new repository
+
+8. Distribute the `libsinsp.so` library and headers as an artifact (rpm, deb, tar.gz) following the falcosecurity current process
+
+9. Distribute the `libsinsp.a` library and headers as an artifact (rpm, deb, tar.gz) following the falcosecurity current process
+
+10. Creation of the CI scripts using the Falco CI and Falco Infra
+
+11. The CI scripts will need to publish the artifacts in the current falcosecurity artifacts repository
+
+12. Artifacts will be pushed for every tag (release) and for every master merge (development release)
+
+13. Falco follows a [multi-stage model for adopting new projects](https://github.com/falcosecurity/evolution#falco-project-evolution), in this case we will do an exception since the library is foundational for Falco and it has a very good track record already
+
+14. This project will go already "Official support" once the donation is completed
+
+15. Contributing, Code of Conduct, Governance, Security, and Support will be the same as the rest of the organization, find them [here](https://github.com/falcosecurity/.github)
+
+16. Every other additional change will need to have its own process with a proposal
+
+17. Implement the release process as described above
+
+18. Propose a change to Falco repository to use the artifacts produced by the libsinsp release process for the build
+
+19. Document the API
+
+### libscap
+
+1. Extract libscap from `draios/sysdig/userspace/libscap` (keeping the commit history) into [falcosecurity/libs](https://github.com/falcosecurity/libs)
+
+2. The migration comes first, then we can do additional PRs for the points below so that we do only one thing at a time and keep the history linear
+
+3. Keep the same code, refactorings will need to be done in subsequent PRs and approved separately
+
+4. Adapt the CMake and build files
+
+5. Install [poiana](https://github.com/poiana) and its workflows on it
+
+6. Define the `OWNERS`
+
+   - Owners are chosen from the current major contributors (considering the past two years) to this project, given their availability, commitment is key
+
+7. When possible, migrate issues and PRs to the new repository
+
+8. Distribute the `libscap.so` library and headers as an artifact (rpm, deb, tar.gz) following the falcosecurity current process
+
+9. Distribute the `libscap.a` library and headers as an artifact (rpm, deb, tar.gz) following the falcosecurity current process
+
+10. Creation of the CI scripts using the Falco CI and Falco Infra
+
+11. The CI scripts will need to publish the artifacts in the current falcosecurity artifacts repository
+
+12. Artifacts will be pushed for every tag (release) and for every master merge (development release)
+
+13. Falco follows a [multi-stage model for adopting new projects](https://github.com/falcosecurity/evolution#falco-project-evolution), in this case we will do an exception since the library is foundational for Falco and it has a very good track record already
+
+14. This project will go already "Official support" once the donation is completed
+
+15. Contributing, Code of Conduct, Governance, Security, and Support will be the same as the rest of the organization, find them [here](https://github.com/falcosecurity/.github)
+
+16. Every other additional change will need to have its own process with a proposal
+
+17. Implement the release process as described above
+
+18. Propose a change to Falco repository to use the artifacts produced by the libscap  release process for the build
+
+19. Document the API
+
+### Drivers: Kernel module and eBPF probe
+
+1. Extract them from `draios/sysdig/driver` (keeping the commit history) into [falcosecurity/libs](https://github.com/falcosecurity/libs)
+
+2. The migration comes first, then we can do additional PRs for the point below so that we do only one thing at a time and keep the history linear
+
+3. Keep the same code, refactorings will need to be done in subsequent PRs and approved separately
+
+4. Adapt the Makefiles and build files
+
+5. Install [poiana](https://github.com/poiana) and its workflows on it
+
+6. Define the `OWNERS`
+
+   - Owners are chosen from the current major contributors (considering the past two years) to this project, given their availability, commitment is key
+
+7. When possible, migrate issues and PRs to the new repository
+
+8. Falco follows a [multi-stage model for adopting new projects](https://github.com/falcosecurity/evolution#falco-project-evolution), in this case we will do an exception since the library is foundational for Falco and it has a very good track record already. We are just changing maintenance ownership
+
+9. Contributing, Code of Conduct, Governance, Security, and Support will be the same as the rest of the organization, find them [here](https://github.com/falcosecurity/.github)
+
+10. Every other additional change will need to have its own process with a proposal
+
+11. The Falco community already ships driver artifacts using [driverkit](https://github.com/falcosecurity/driverkit) and the [test-infra repository](https://github.com/falcosecurity/test-infra)
+
+    - Adapt the place from which [driverkit](https://github.com/falcosecurity/driverkit) grabs the drivers source
+
+12. This project will go already "Official support" once the migration is completed.
+
+### Falco
+
+1. Adapt the CMake files to point to the new homes for libscap, libsinsp and the drivers
+
+2. When distributing the deb and rpm, libscap and libsinsp will need to be install dependencies and not anymore compiled into Falco
+
+### Driverkit
+
+1. Change the source location for the drivers to point to the new driver repository
+
+### pdig
+
+1. The project will need to be adapted to use libscap and libsinsp and the fillers from their new location

rules/k8s_audit_rules.yaml

@@ -50,7 +50,8 @@
     vertical_pod_autoscaler_users,
     cluster-autoscaler,
     "system:addon-manager",
-    "cloud-controller-manager"
+    "cloud-controller-manager",
+    "eks:node-manager"
     ]
 
 - rule: Disallowed K8s User

test/falco_tests.yaml

@@ -1277,3 +1277,10 @@ trace_files: !mux
     trace_file: trace_files/cat_write.scap
     stdout_contains: "2016-08-04T16:17:57.882054739\\+0000: Warning An open was seen"
     stderr_contains: "^\\d\\d\\d\\d-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d\\+0000"
+
+  unknown_source:
+    detect: True
+    detect_level: WARNING
+    rules_file:
+      - rules/unknown_source.yaml
+    trace_file: trace_files/cat_write.scap

test/rules/unknown_source.yaml

@@ -0,0 +1,31 @@
+#
+# Copyright (C) 2021 The Falco Authors.
+#
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+- macro: Macro with unknown source
+  condition: some other unknown filter
+  source: unknown-source
+
+- rule: Rule with unknown source
+  condition: some unknown filter
+  output: some unknown output
+  priority: INFO
+  source: unknown-source
+
+- rule: open_from_cat
+  desc: A process named cat does an open
+  condition: evt.type=open and proc.name=cat
+  output: "An open was seen (command=%proc.cmdline)"
+  priority: WARNING

userspace/engine/lua/rule_loader.lua

@@ -436,6 +436,11 @@ function load_rules_doc(rules_mgr, doc, load_state)
 	    v['source'] = "syscall"
 	 end
 
+	 -- Ignore macros with unknown sources
+	 if (v['source'] ~= "syscall" and v['source'] ~= "k8s_audit") then
+	    goto next_object
+	 end
+
 	 if state.macros_by_name[v['macro']] == nil then
 	    state.ordered_macro_names[#state.ordered_macro_names+1] = v['macro']
 	 end
@@ -522,6 +527,11 @@ function load_rules_doc(rules_mgr, doc, load_state)
 	    v['source'] = "syscall"
 	 end
 
+	 -- Ignore rules with unknown sources
+	 if (v['source'] ~= "syscall" and v['source'] ~= "k8s_audit") then
+	    goto next_object
+	 end
+
 	 -- Add an empty exceptions property to the rule if not
 	 -- defined, but add a warning about defining one
 	 if v['exceptions'] == nil then
@@ -668,6 +678,8 @@ function load_rules_doc(rules_mgr, doc, load_state)
 	 arr = build_error_with_context(context, "Unknown top level object: "..table.tostring(v))
 	 warnings[#warnings + 1] = arr[1]
       end
+
+      ::next_object::
    end
 
    return true, {}, warnings