This version works

Minimally working version that can link a go program against a so with
the embedded falco engine. Running the program opens the engine for
syscalls and prints any output strings on alert.

It assumes the device already exists and the kernel module is
loaded. Also assumes the lua code is below /user/share--we'll want to
bake that into the shared library.

Lots of memory leaks still, the interface from go to c is still
monolithic, and I had to change the config of openssl crypto and
luajit to compile with -fPIC in order to link into the shared library,
but this version shows its feasible.

.circleci/config.yml

@@ -403,7 +403,7 @@ jobs:
           name: Setup
           command: |
             apt update -y
-            apt-get install apt-utils bzip2 gpg python python3-pip -y
+            apt-get install apt-utils bzip2 gpg python python-pip -y
             pip install awscli
             echo $GPG_KEY | base64 -d | gpg --import
       - run:
@@ -517,7 +517,7 @@ jobs:
           name: Setup
           command: |
             apt update -y
-            apt-get install apt-utils bzip2 gpg python python3-pip -y
+            apt-get install apt-utils bzip2 gpg python python-pip -y
             pip install awscli
             echo $GPG_KEY | base64 -d | gpg --import
       - run:

ADOPTERS.md

@@ -40,8 +40,6 @@ This is a list of production adopters of Falco (in alphabetical order):
 * [Preferral](https://www.preferral.com) - Preferral is a HIPAA-compliant platform for Referral Management and Online Referral Forms. Preferral streamlines the referral process for patients, specialists and their referral partners. By automating the referral process, referring practices spend less time on the phone, manual efforts are eliminated, and patients get the right care from the right specialist. Preferral leverages Falco to provide a Host Intrusion Detection System to meet their HIPPA compliance requirements.
   * https://hipaa.preferral.com/01-preferral_hipaa_compliance/
 
-* [Qonto](https://qonto.com) - Qonto is a modern banking for SMEs and freelancers. Qonto provides a fully featured business account with a simplified accounting flow. Falco is used by our SecOps team to detect suspicous behaviors in our clusters.
-
 * [Replicated](https://www.replicated.com/) - Replicated is the modern way to ship on-prem software. Replicated gives software vendors a container-based platform for easily deploying cloud native applications inside customers'​ environments to provide greater security and control. Replicated uses Falco as runtime security to detect threats in the Kubernetes clusters which host our critical SaaS services.
 
 * [Secureworks](https://www.secureworks.com/) - Secureworks is a leading worldwide cybersecurity company with a cloud-native security product that combines the power of human intellect with security analytics to unify detection and response across cloud, network, and endpoint environments for improved security operations and outcomes. Our Taegis XDR platform and detection system processes petabytes of security relevant data to expose active threats amongst the billions of daily events from our customers. We are proud to protect our platform’s Kubernetes deployments, as well as help our customers protect their own Linux and container environments, using Falco. 

CHANGELOG.md

@@ -1,47 +1,5 @@
 # Change Log
 
-## v0.30.0
-
-Released on 2021-10-01
-
-### Major Changes
-
-* new: add `--k8s-node` command-line options, which allows filtering by a node when requesting metadata of pods to the K8s API server [[#1671](https://github.com/falcosecurity/falco/pull/1671)] - [@leogr](https://github.com/leogr)
-* new(outputs): expose rule tags and event source in gRPC and json outputs [[#1714](https://github.com/falcosecurity/falco/pull/1714)] - [@jasondellaluce](https://github.com/jasondellaluce)
-* new(userspace/falco): add customizable metadata fetching params [[#1667](https://github.com/falcosecurity/falco/pull/1667)] - [@zuc](https://github.com/zuc)
-
-
-### Minor Changes
-
-* update: bump driver version to 3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4 [[#1744](https://github.com/falcosecurity/falco/pull/1744)] - [@zuc](https://github.com/zuc)
-* docs: clarify that previous Falco drivers will remain available at https://download.falco.org and no automated cleanup is run anymore [[#1738](https://github.com/falcosecurity/falco/pull/1738)] - [@leodido](https://github.com/leodido)
-* update(outputs): add configuration option for tags in json outputs [[#1733](https://github.com/falcosecurity/falco/pull/1733)] - [@jasondellaluce](https://github.com/jasondellaluce)
-
-
-### Bug Fixes
-
-* fix(scripts): correct standard output redirection in systemd config (DEB and RPM packages) [[#1697](https://github.com/falcosecurity/falco/pull/1697)] - [@chirabino](https://github.com/chirabino)
-* fix(scripts): correct lookup order when trying multiple `gcc` versions in the `falco-driver-loader` script [[#1716](https://github.com/falcosecurity/falco/pull/1716)] - [@Spartan-65](https://github.com/Spartan-65)
-
-
-### Rule Changes
-
-* rule(list miner_domains): add new miner domains [[#1729](https://github.com/falcosecurity/falco/pull/1729)] - [@AlbertoPellitteri](https://github.com/AlbertoPellitteri)
-* rule(list https_miner_domains): add new miner domains [[#1729](https://github.com/falcosecurity/falco/pull/1729)] - [@AlbertoPellitteri](https://github.com/AlbertoPellitteri)
-
-
-### Non user-facing changes
-
-* add Qonto as adopter [[#1717](https://github.com/falcosecurity/falco/pull/1717)] - [@Issif](https://github.com/Issif)
-* docs(proposals): proposal for a libs plugin system [[#1637](https://github.com/falcosecurity/falco/pull/1637)] - [@ldegio](https://github.com/ldegio)
-* build: remove unused `ncurses` dependency [[#1658](https://github.com/falcosecurity/falco/pull/1658)] - [@leogr](https://github.com/leogr)
-* build(.circleci): use new Debian 11 package names for python-pip [[#1712](https://github.com/falcosecurity/falco/pull/1712)] - [@zuc](https://github.com/zuc)
-* build(docker): adding libssl-dev, upstream image reference pinned to `debian:buster` [[#1719](https://github.com/falcosecurity/falco/pull/1719)] - [@michalschott](https://github.com/michalschott)
-* fix(test): avoid output_strictly_contains failures [[#1724](https://github.com/falcosecurity/falco/pull/1724)] - [@jasondellaluce](https://github.com/jasondellaluce)
-* Remove duplicate allowed ecr registry rule [[#1725](https://github.com/falcosecurity/falco/pull/1725)] - [@TomKeyte](https://github.com/TomKeyte)
-* docs(RELEASE.md): switch to 3 releases per year [[#1711](https://github.com/falcosecurity/falco/pull/1711)] - [@leogr](https://github.com/leogr)
-
-
 ## v0.29.1
 
 Released on 2021-06-29

RELEASE.md

@@ -4,9 +4,7 @@ Our release process is mostly automated, but we still need some manual steps to
 
 Changes and new features are grouped in [milestones](https://github.com/falcosecurity/falco/milestones), the milestone with the next version represents what is going to be released.
 
-Falco releases are due to happen 3 times per year. Our current schedule sees a new release by the end of January, May, and September each year. Hotfix releases can happen whenever it's needed.
-
-Moreover, we need to assign owners for each release (usually we pair a new person with an experienced one). Assignees and the due date are proposed during the [weekly community call](https://github.com/falcosecurity/community).
+A release happens every two months ([as per community discussion](https://github.com/falcosecurity/community/blob/master/meeting-notes/2020-09-30.md#agenda)), and we need to assign owners for each (usually we pair a new person with an experienced one). Assignees and the due date are proposed during the [weekly community call](https://github.com/falcosecurity/community). Note that hotfix releases can happen as soon as it is needed.
 
 Finally, on the proposed due date the assignees for the upcoming release proceed with the processes described below.
 

cmake/modules/falcosecurity-libs.cmake

@@ -20,8 +20,8 @@ file(MAKE_DIRECTORY ${FALCOSECURITY_LIBS_CMAKE_WORKING_DIR})
 # default below In case you want to test against another falcosecurity/libs version just pass the variable - ie., `cmake
 # -DFALCOSECURITY_LIBS_VERSION=dev ..`
 if(NOT FALCOSECURITY_LIBS_VERSION)
-  set(FALCOSECURITY_LIBS_VERSION "3aa7a83bf7b9e6229a3824e3fd1f4452d1e95cb4")
-  set(FALCOSECURITY_LIBS_CHECKSUM "SHA256=1edb535b3778fcfb46bbeeda891f176a1bd591bebd7b89c27f04837e55a52beb")
+  set(FALCOSECURITY_LIBS_VERSION "new/plugin-system-api-additions")
+  set(FALCOSECURITY_LIBS_CHECKSUM "SHA256=ba0ea2e22121b8543cb1ebe616090097c4dc3f093db8f0bb5cf2ce5a7e0425a0")
 endif()
 
 # cd /path/to/build && cmake /path/to/source

docker/falco/Dockerfile

@@ -1,4 +1,4 @@
-FROM debian:buster
+FROM debian:stable
 
 LABEL maintainer="cncf-falco-dev@lists.cncf.io"
 
@@ -30,7 +30,6 @@ RUN apt-get update \
 	libc6-dev \
 	libelf-dev \
 	libmpx2 \
-	libssl-dev \
 	llvm-7 \
 	netcat \
 	xz-utils \

falco.yaml

@@ -46,12 +46,6 @@ json_output: false
 # (user=root ....") in the json output.
 json_include_output_property: true
 
-# When using json output, whether or not to include the "tags" property
-# itself in the json output. If set to true, outputs caused by rules
-# with no tags will have a "tags" field set to an empty array. If set to
-# false, the "tags" field will not be included in the json output at all.
-json_include_tags_property: true
-
 # Send information logs to stderr and/or syslog Note these are *not* security
 # notification logs! These are just Falco lifecycle (and possibly error) logs.
 log_stderr: true
@@ -255,9 +249,3 @@ grpc:
 # Make sure to have a consumer for them or leave this disabled.
 grpc_output:
   enabled: false
-
-# Container orchestrator metadata fetching params
-metadata_download:
-  max_mb: 100
-  chunk_wait_us: 1000
-  watch_freq_sec: 1

proposals/20200901-artifacts-cleanup.md

@@ -2,8 +2,6 @@
 
 This document reflects when and how we clean up the Falco artifacts from their storage location.
 
-**Superseeded by**: [drivers-storage-s3 proposal](https://github.com/falcosecurity/falco/blob/master/proposals/20201025-drivers-storage-s3.md).
-
 ## Motivation
 
 The [bintray](https://bintray.com/falcosecurity) open-source plan offers 10GB free space for storing artifacts.
@@ -96,19 +94,9 @@ Since the process of building drivers is time and resource consuming, this docum
 
 The candidate is an AWS S3 bucket responsible for holding the deleted driver version files.
 
-#### Notice
-
-The current mechanism the Falco community uses to store the Falco drivers is explained by the [drivers-storage-s3](https://github.com/falcosecurity/falco/blob/master/proposals/20201025-drivers-storage-s3.md) proposal.
-
 ### Implementation
 
 The [test-infra](https://github.com/falcosecurity/test-infra) CI, specifically its part dedicated to run the **Drivers Build Grid** that runs every time it detects changes into the `driverkit` directory of the [test-infra](https://github.com/falcosecurity/test-infra) repository,
 will have a new job - called `drivers/cleanup` - responsible for removing all the Falco driver versions except the last two.
 
-This job will be triggered after the `drivers/publish` completed successfully on the master branch.
-
-#### Notice
-
-At the moment of writing (2021 09 28) the `drivers/cleanup` job is no more in place.
-
-Pragmatically, this means that the older Falco drivers will remain available in their [S3 bucket](https://download.falco.org/?prefix=driver/).
+This job will be triggered after the `drivers/publish` completed successfully on the master branch.

proposals/20210501-plugin-system.md

@@ -1,613 +0,0 @@
-# Plugin System
-
-## Summary
-
-This is a proposal to create an infrastructure to extend the functionality of the Falco libraries via plugins.
-
-Plugins will allow users to easily extend the functionality of the libraries and, as a consequence, of Falco and any other tool based on the libraries.
-
-This proposal, in particular, focuses on two types of plugins: source plugins and extractor plugins.
-
-## Motivation
-
-[libscap](https://github.com/falcosecurity/libs/tree/master/userspace/libscap) and [libsinsp](https://github.com/falcosecurity/libs/tree/master/userspace/libsinsp) provide a powerful data capture framework, with a rich set of features that includes:
-
-- data capture
-- trace files management
-- enrichment
-- filtering
-- formatting and screen rendering
-- Lua scripting (chisels)
-
-These features have been designed with one specific input in mind: system calls. However, they are generically adaptable to a broad set of inputs, such as cloud logs.
-
-With this proposal, we want to dramatically extend the scope of what the libraries, Falco and other tools can be applied to. We want to do it in a way that is easy, efficient and empowers anyone in the community to write a plugin.
-
-## Goals
-
-- To design and implement a plugin framework that makes the libraries more modular and extensible
-- To have a framework that is easy to use
-- To support dynamic loading of plugins, so that the libraries can be extended without having to be recompiled and relinked
-- To enable users to write plugins in any language, with a particular focus on Go, C and C++
-- To have an efficient plugin framework so that, performance-wise, writing a plugin is as close as possible as extending the libraries internal source code
-- To make it possible to write plugins for Linux, MacOS and Windows
-
-## Non-Goals
-
-- To implement plugins other than source and extractor: to be approached as separate task
-- To document the plugin framework and interface: to be approached as separate task
-
-## Proposal
-
-### Plugin Common Information
-
-Both source and extractor plugins have the following:
-
-- A required api version, to ensure compatibility with the plugin framework.
-- A name
-- A description
-- A version
-- A contact field for the plugin authors (website, github repo, twitter, etc).
-- Functions to initialize and destroy the plugin internal state.
-
-### Plugin types
-
-Initially, we will implement support for two types of plugins: source plugins and extractor plugins.
-
-#### Source Plugin
-
-A source plugin implements a new sinsp/scap event source. It has the ability to "open" and "close" a session that provides events. It also has the ability to return an event to the plugin framework via a next() method. Events returned by source plugins have an "event source", which describes the information in the event. This is distinct from the plugin name to allow for multiple kinds of plugins to generate the same kind of events. For example, there might be plugins gke-audit-bridge, eks-audit-bridge, ibmcloud-audit-bridge, etc. that all fetch [K8s Audit](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) information. The plugins would have different names but would have the same event source "k8s_audit".
-
-Source plugins also have the ability to extract information from events based on fields. For example, a field proc.name extracts a process name from a syscall event. The plugin returns a set of supported fields, and there are functions to extract a value given an event and field. The plugin framework can then build filtering expressions/Falco rule conditions based on these fields combined with relational and/or logical operators. For example, given an expression "ct.name=root and ct.region=us-east-1", the plugin framework handles parsing the expression, calling the plugin to extract values for a given event, and determining the result of the expression. In a Falco output string like "An EC2 Node was created (name=%ec2.name region=%ct.region)", the plugin framework handles parsing the output string, calling the plugin to extract values for a given event, and building the resolved string.
-
-Source plugins also provide an "id", which is globally unique and is used in capture files (see below).
-
-#### Extractor Plugin
-
-An extractor plugin focuses only on field extraction from events generated by other plugins, or by the core libraries. It does *not* provide an event source, but can extract fields from other event sources. An example is json field extraction, where a plugin might be able to extract fields from arbitrary json payloads.
-
-An extractor plugin provides an optional set of event sources. When the framework receives an event with an event source in the plugin's set of event sources, fields in expressions/Falco outputs will be extracted from events using the plugin. An extractor plugin can also *not* name a set of event sources. In this case, fields will be extracted from *all* events, regardless of source. In this case, the exctractor plugin must detect the format of arbitrary payloads and be able to return NULL/no value when the payload is not supported.
-
-### Support for Plugin Events in Capture Files.
-
-libscap will define a new event type called "pluginevent" that contains two fields:
-
-* "plugin ID": This uniquely identifies the plugin that generated this event.
-* "event_data": This is a variable-length data buffer containing the event data, as returned by the plugin.
-
-Defining an event for plugins allows creating capture files from plugins. These capture files can be saved, read, filtered, etc, like any other capture file, allowing for later analysis/display/etc.
-
-### Plugins format
-
-Plugins are dynamic libraries (.so files in Unix, .dll files in windows) that export a minimum set of functions that the libraries will recognize.
-
-Plugins are versioned using semantic versioning to minimize regressions and compatibility issues.
-
-Plugins can be written in any language, as long as they export the required functions. Go, however, is the preferred language to write plugins, followed by C/C++.
-
-### Protecting from plugin issues
-
-The libraries will do everything possible to validate the data coming from the plugins and protect Falco and the other consumers from corrupted data. However, for performance reasons, plugins will be "trusted": they will run in the same thread and address space as Falco and they could crash the program. We assume that the user will be in control of plugin loading and will make sure only trusted plugins are loaded/packaged with Falco.
-
-### Plugin/Event Source registries
-
-Every source plugin requires its own, unique plugin ID to interoperate with Falco and the other plugins. The plugin ID will be used by the libs to properly process incoming events (for example, when saving events to file and loading them back), and by plugins to unuambiguosly recognize their dependencies.
-
-To facilitate the allocation and distribution of plugin IDs, we will require that plugin developers request IDs for their plugins to the Falco organization. The mechanism used for plugin allocation is not determined yet and will be discussed in the future.
-
-Similarly, plugin developers must register event sources with the Falco organization. This allows coordination between plugins that wish to provide compatible payloads, and to allow extractor plugins to know what data format is associated with a given event source.
-
-### golang plugin SDK
-
-To facilitate the development of plugins written in go, an SDK has been developed. We intend this SDK (and future SDKs for other languages) to be part of the Falco organization. For this reason, we submitted the following incubation request: https://github.com/falcosecurity/evolution/issues/62
-
-### Proposed API (subject to change)
-
-```c
-// This struct represents an event returned by the plugin, and is used
-// below in next()/next_batch().
-// - data: pointer to a memory buffer pointer. The plugin will set it
-//   to point to the memory containing the next event. Once returned,
-//   the memory is owned by the plugin framework and will be freed via
-//   a call to free().
-// - datalen: pointer to a 32bit integer. The plugin will set it the size of the
-//   buffer pointed by data.
-// - ts: the event timestamp. Can be (uint64_t)-1, in which case the engine will
-//   automatically fill the event time with the current time.
-typedef struct ss_plugin_event
-{
-	uint8_t *data;
-	uint32_t datalen;
-	uint64_t ts;
-} ss_plugin_event;
-
-//
-// This is the opaque pointer to the state of a plugin.
-// It points to any data that might be needed plugin-wise. It is
-// allocated by init() and must be destroyed by destroy().
-// It is defined as void because the engine doesn't care what it is
-// and it treats is as opaque.
-//
-typedef void ss_plugin_t;
-
-//
-// This is the opaque pointer to the state of an open instance of the source
-// plugin.
-// It points to any data that is needed while a capture is running. It is
-// allocated by open() and must be destroyed by close().
-// It is defined as void because the engine doesn't care what it is
-// and it treats is as opaque.
-//
-typedef void ss_instance_t;
-
-//
-// Interface for a sinsp/scap source plugin
-//
-//
-// NOTE: For all functions below that return a char *, the memory
-// pointed to by the char * must be allocated by the plugin using
-// malloc() and should be freed by the caller using free().
-//
-// For each function below, the exported symbol from the dynamic
-// library should have a prefix of "plugin_"
-// (e.g. plugin_get_required_api_version, plugin_init, etc.)
-//
-typedef struct
-{
-	//
-	// Return the version of the plugin API used by this plugin.
-	// Required: yes
-	// Return value: the API version string, in the following format:
-	//        "<major>.<minor>.<patch>", e.g. "1.2.3".
-	// NOTE: to ensure correct interoperability between the engine and the plugins,
-	//       we use a semver approach. Plugins are required to specify the version
-	//       of the API they run against, and the engine will take care of checking
-	//       and enforcing compatibility.
-	//
-	char* (*get_required_api_version)();
-	//
-	// Return the plugin type.
-	// Required: yes
-	// Should return TYPE_SOURCE_PLUGIN. It still makes sense to
-	// have a function get_type() as the plugin interface will
-	// often dlsym() functions from shared libraries, and can't
-	// inspect any C struct type.
-	//
-	uint32_t (*get_type)();
-	//
-	// Initialize the plugin and, if needed, allocate its state.
-	// Required: yes
-	// Arguments:
-	// - config: a string with the plugin configuration. The format of the
-	//   string is chosen by the plugin itself.
-	// - rc: pointer to an integer that will contain the initialization result,
-	//   as a SCAP_* value (e.g. SCAP_SUCCESS=0, SCAP_FAILURE=1)
-	// Return value: pointer to the plugin state that will be treated as opaque
-	//   by the engine and passed to the other plugin functions.
-	//   If rc is SCAP_FAILURE, this function should return NULL.
-	//
-	ss_plugin_t* (*init)(char* config, int32_t* rc);
-	//
-	// Destroy the plugin and, if plugin state was allocated, free it.
-	// Required: yes
-	//
-	void (*destroy)(ss_plugin_t* s);
-	//
-	// Return a string with the error that was last generated by
-	// the plugin.
-        // Required: yes
-	//
-	// In cases where any other api function returns an error, the
-	// plugin should be prepared to return a human-readable error
-	// string with more context for the error. The plugin manager
-	// calls get_last_error() to access that string.
-	//
-	char* (*get_last_error)(ss_plugin_t* s);
-	//
-	// Return the unique ID of the plugin.
-	// Required: yes
-	// EVERY SOURCE PLUGIN (see get_type()) MUST OBTAIN AN OFFICIAL ID FROM THE
-	// FALCOSECURITY ORGANIZATION, OTHERWISE IT WON'T PROPERLY COEXIST WITH OTHER PLUGINS.
-	//
-	uint32_t (*get_id)();
-	//
-	// Return the name of the plugin, which will be printed when displaying
-	// information about the plugin.
-	// Required: yes
-	//
-	char* (*get_name)();
-	//
-	// Return the descriptions of the plugin, which will be printed when displaying
-	// information about the plugin or its events.
-	// Required: yes
-	//
-	char* (*get_description)();
-	//
-	// Return a string containing contact info (url, email, twitter, etc) for
-	// the plugin authors.
-	// Required: yes
-	//
-	char* (*get_contact)();
-	//
-	// Return the version of this plugin itself
-	// Required: yes
-	// Return value: a string with a version identifier, in the following format:
-	//        "<major>.<minor>.<patch>", e.g. "1.2.3".
-	// This differs from the api version in that this versions the
-	// plugin itself, as compared to the plugin interface. When
-	// reading capture files, the major version of the plugin that
-	// generated events must match the major version of the plugin
-	// used to read events.
-	//
-	char* (*get_version)();
-	//
-	// Return a string describing the events generated by this source plugin.
-	// Required: yes
-	// Example event sources would be strings like "syscall",
-	// "k8s_audit", etc.  The source can be used by extractor
-	// plugins to filter the events they receive.
-	//
-	char* (*get_event_source)();
-	//
-	// Return the list of extractor fields exported by this plugin. Extractor
-	// fields can be used in Falco rule conditions and sysdig filters.
-	// Required: no
-	// Return value: a string with the list of fields encoded as a json
-	//   array.
-	//   Each field entry is a json object with the following properties:
-	//     "type": one of "string", "uint64"
-	//     "name": a string with a name for the field
-	//     "desc": a string with a description of the field
-	// Example return value:
-	// [
-	//    {"type": "string", "name": "field1", "desc": "Describing field 1"},
-	//    {"type": "uint64", "name": "field2", "desc": "Describing field 2"}
-	// ]
-	char* (*get_fields)();
-	//
-	// Open the source and start a capture.
-	// Required: yes
-	// Arguments:
-	// - s: the plugin state returned by init()
-	// - params: the open parameters, as a string. The format is defined by the plugin
-	//   itsef
-	// - rc: pointer to an integer that will contain the open result, as a SCAP_* value
-	//   (e.g. SCAP_SUCCESS=0, SCAP_FAILURE=1)
-	// Return value: a pointer to the open context that will be passed to next(),
-	//   close(), event_to_string() and extract_*.
-	//
-	ss_instance_t* (*open)(ss_plugin_t* s, char* params, int32_t* rc);
-	//
-	// Close a capture.
-	// Required: yes
-	// Arguments:
-	// - s: the plugin context, returned by init(). Can be NULL.
-	// - h: the capture context, returned by open(). Can be NULL.
-	//
-	void (*close)(ss_plugin_t* s, ss_instance_t* h);
-	//
-	// Return the next event.
-	// Required: yes
-	// Arguments:
-	// - s: the plugin context, returned by init(). Can be NULL.
-	// - h: the capture context, returned by open(). Can be NULL.
-	//
-        // - evt: pointer to a ss_plugin_event pointer. The plugin should
-        //   allocate a ss_plugin_event struct using malloc(), as well as
-	//   allocate the data buffer within the ss_plugin_event struct.
-	//   Both the struct and data buffer are owned by the plugin framework
-	//   and will free them using free().
-	//
-	// Return value: the status of the operation (e.g. SCAP_SUCCESS=0, SCAP_FAILURE=1,
-	//   SCAP_TIMEOUT=-1)
-	//
-	int32_t (*next)(ss_plugin_t* s, ss_instance_t* h, ss_plugin_event **evt);
-	//
-	// Return the read progress.
-	// Required: no
-	// Arguments:
-	// - progress_pct: the read progress, as a number between 0 (no data has been read)
-	//   and 10000 (100% of the data has been read). This encoding allows the engine to
-	//   print progress decimals without requiring to deal with floating point numbers
-	//   (which could cause incompatibility problems with some languages).
-	// Return value: a string representation of the read
-	//   progress. This might include the progress percentage
-	//   combined with additional context added by the plugin. If
-	//   NULL, progress_pct should be used.
-	// NOTE: reporting progress is optional and in some case could be impossible. However,
-	//       when possible, it's recommended as it provides valuable information to the
-	//       user.
-	//
-	char* (*get_progress)(ss_plugin_t* s, ss_instance_t* h, uint32_t* progress_pct);
-	//
-	// Return a text representation of an event generated by this source plugin.
-	// Required: yes
-	// Arguments:
-	// - data: the buffer from an event produced by next().
-	// - datalen: the length of the buffer from an event produced by next().
-	// Return value: the text representation of the event. This is used, for example,
-	//   by sysdig to print a line for the given event.
-	//
-	char *(*event_to_string)(ss_plugin_t *s, const uint8_t *data, uint32_t datalen);
-	//
-	// Extract a filter field value from an event.
-	// We offer multiple versions of extract(), differing from each other only in
-	// the type of the value they return (string, integer...).
-	// Required: no
-	// Arguments:
-	// - evtnum: the number of the event that is bein processed
-	// - id: the numeric identifier of the field to extract. It corresponds to the
-	//   position of the field in the array returned by get_fields().
-	// - arg: the field argument, if an argument has been specified for the field,
-	//   otherwise it's NULL. For example:
-	//    * if the field specified by the user is foo.bar[pippo], arg will be the
-	//      string "pippo"
-	//    * if the field specified by the user is foo.bar, arg will be NULL
-	// - data: the buffer produced by next().
-	// - datalen: the length of the buffer produced by next().
-	// - field_present: nonzero if the field is present for the given event.
-	// Return value: the produced value of the filter field. For extract_str(), a
-	//   NULL return value means that the field is missing for the given event.
-	//
-	char *(*extract_str)(ss_plugin_t *s, uint64_t evtnum, const char * field, const char *arg, uint8_t *data, uint32_t datalen);
-	uint64_t (*extract_u64)(ss_plugin_t *s, uint64_t evtnum, const char *field, const char *arg, uint8_t *data, uint32_t datalen, uint32_t *field_present);
-	//
-	// This is an optional, internal, function used to speed up event capture by
-	// batching the calls to next().
-	// On success:
-	//   - nevts will be filled in with the number of events.
-        //   - evts: pointer to an ss_plugin_event pointer. The plugin should
-        //     allocate an array of contiguous ss_plugin_event structs using malloc(),
-	//     as well as allocate each data buffer within each ss_plugin_event
-	//     struct using malloc(). Both the array of structs and each data buffer are
-	//     owned by the plugin framework and will free them using free().
-	// Required: no
-	//
-	int32_t (*next_batch)(ss_plugin_t* s, ss_instance_t* h, uint32_t *nevts, ss_plugin_event **evts);
-	//
-	// This is an optional, internal, function used to speed up value extraction
-	// Required: no
-	//
-	int32_t (*register_async_extractor)(ss_plugin_t *s, async_extractor_info *info);
-
-	//
-	// The following members are PRIVATE for the engine and should not be touched.
-	//
-	ss_plugin_t* state;
-	ss_instance_t* handle;
-	uint32_t id;
-	char *name;
-} source_plugin_info;
-
-//
-// Interface for a sinsp/scap extractor plugin
-//
-//
-// NOTE: For all functions below that return a char *, the memory
-// pointed to by the char * must be allocated by the plugin using
-// malloc() and should be freed by the caller using free().
-//
-typedef struct
-{
-	//
-	// Return the version of the plugin API used by this plugin.
-	// Required: yes
-	// Return value: the API version string, in the following format:
-	//        "<major>.<minor>.<patch>", e.g. "1.2.3".
-	// NOTE: to ensure correct interoperability between the engine and the plugins,
-	//       we use a semver approach. Plugins are required to specify the version
-	//       of the API they run against, and the engine will take care of checking
-	//       and enforcing compatibility.
-	//
-	char* (*get_required_api_version)();
-	//
-	// Return the plugin type.
-	// Required: yes
-	// Should return TYPE_EXTRACTOR_PLUGIN. It still makes sense to
-	// have a function get_type() as the plugin interface will
-	// often dlsym() functions from shared libraries, and can't
-	// inspect any C struct type.
-	//
-	uint32_t (*get_type)();
-	//
-	// Initialize the plugin and, if needed, allocate its state.
-	// Required: yes
-	// Arguments:
-	// - config: a string with the plugin configuration. The format of the
-	//   string is chosen by the plugin itself.
-	// - rc: pointer to an integer that will contain the initialization result,
-	//   as a SCAP_* value (e.g. SCAP_SUCCESS=0, SCAP_FAILURE=1)
-	// Return value: pointer to the plugin state that will be treated as opaque
-	//   by the engine and passed to the other plugin functions.
-	//
-	ss_plugin_t* (*init)(char* config, int32_t* rc);
-	//
-	// Destroy the plugin and, if plugin state was allocated, free it.
-	// Required: yes
-	//
-	void (*destroy)(ss_plugin_t* s);
-	//
-	// Return a string with the error that was last generated by
-	// the plugin.
-        // Required: yes
-	//
-	// In cases where any other api function returns an error, the
-	// plugin should be prepared to return a human-readable error
-	// string with more context for the error. The plugin manager
-	// calls get_last_error() to access that string.
-	//
-	char* (*get_last_error)(ss_plugin_t* s);
-	//
-	// Return the name of the plugin, which will be printed when displaying
-	// information about the plugin.
-	// Required: yes
-	//
-	char* (*get_name)();
-	//
-	// Return the descriptions of the plugin, which will be printed when displaying
-	// information about the plugin or its events.
-	// Required: yes
-	//
-	char* (*get_description)();
-	//
-	// Return a string containing contact info (url, email, twitter, etc) for
-	// the plugin author.
-	// Required: yes
-	//
-	char* (*get_contact)();
-	//
-	// Return the version of this plugin itself
-	// Required: yes
-	// Return value: a string with a version identifier, in the following format:
-	//        "<major>.<minor>.<patch>", e.g. "1.2.3".
-	// This differs from the api version in that this versions the
-	// plugin itself, as compared to the plugin interface. When
-	// reading capture files, the major version of the plugin that
-	// generated events must match the major version of the plugin
-	// used to read events.
-	//
-	char* (*get_version)();
-	//
-	// Return a string describing the event sources that this
-	// extractor plugin can consume.
-	// Required: no
-	// Return value: a json array of strings containing event
-	//   sources returned by a source plugin's get_event_source()
-	//   function.
-	// This function is optional--if NULL then the exctractor
-	// plugin will receive every event.
-	//
-	char* (*get_extract_event_sources)();
-	//
-	// Return the list of extractor fields exported by this plugin. Extractor
-	// fields can be used in Falco rules and sysdig filters.
-	// Required: yes
-	// Return value: a string with the list of fields encoded as a json
-	//   array.
-	//
-	char* (*get_fields)();
-	//
-	// Extract a filter field value from an event.
-	// We offer multiple versions of extract(), differing from each other only in
-	// the type of the value they return (string, integer...).
-	// Required: for plugins of type TYPE_EXTRACTOR_PLUGIN only
-	// Arguments:
-	// - evtnum: the number of the event that is being processed
-	// - id: the numeric identifier of the field to extract. It corresponds to the
-	//   position of the field in the array returned by get_fields().
-	// - arg: the field argument, if an argument has been specified for the field,
-	//   otherwise it's NULL. For example:
-	//    * if the field specified by the user is foo.bar[pippo], arg will be the
-	//      string "pippo"
-	//    * if the field specified by the user is foo.bar, arg will be NULL
-	// - data: the buffer produced by next().
-	// - datalen: the length of the buffer produced by next().
-	// - field_present: nonzero if the field is present for the given event.
-	// Return value: the produced value of the filter field. For extract_str(), a
-	// NULL return value means that the field is missing for the given event.
-	//
-	char *(*extract_str)(ss_plugin_t *s, uint64_t evtnum, const char *field, const char *arg, uint8_t *data, uint32_t datalen);
-	uint64_t (*extract_u64)(ss_plugin_t *s, uint64_t evtnum, const char *field, const char *arg, uint8_t *data, uint32_t datalen, uint32_t *field_present);
-} extractor_plugin_info;
-
-```
-
-### Event Sources and Falco Rules
-
-Falco rules already have the notion of a "source", using the source property in rules objects, and there are currently two kinds of event sources: "syscall" and "k8s_audit". We will use the source property in Falco rules to map a given rule to the event source on which the rule runs.
-
-For example, given a plugin with source "aws_cloudtrail", and a Falco rule with source "aws_cloudtrail", the rule will be evaluated for any events generated by the plugin.
-
-Similarly, an extractor plugin that includes "aws_cloudtrail" in its set of event sources will have the opportunity to extract information from aws_cloudtrail events if a matching field is found in the rule's condition, exception, or output properties.
-
-This, combined with the restrictions below, allows a set of loaded rules files to contain a mix of rules for plugins as well as "core" syscall/k8s_audit events.
-
-We will also make a change to compile rules/macros/lists selectively based on the set of loaded plugins (specifically, their event sources), instead of unconditionally as Falco is started. This is especially important for macros, which do not contain a source property, but might contain fields that are only implemented by a given plugin.
-
-### Handling Duplicate/Overlapping Fields in Plugins/Libraries Core
-
-At an initial glance, adding plugins introduces the possibility of tens/hundreds of new filtercheck fields that could potentially overlap/conflict. For example, what happens if a plugin defines a "proc.name" field? However, the notion of "event source" makes these potential conflicts managable.
-
-Remember that field extraction is always done in the context of an event, and each event can be mapped back to an event source. So we only need to ensure that filtercheck fields are distinct for a given event source. For example, it's perfectly valid for an AWS Cloudtrail plugin to define a proc.name field, as the events generated by that plugin are wholly separate from syscall events. For syscall events, the AWS Cloudtrail plugin is not involved and the core libraries extract the process name for the tid performing a syscall. For AWS Cloudtrail events, the core libraries are not involved in field extraction and is performed by the AWS Cloudtrail plugin instead.
-
-We only need to ensure the following:
-
-* That only one plugin is loaded at a time that exports a given event source. For example, the libraries can load either a gke-audit-bridge plugin with event source k8s_audit, or eks-audit-bridge with event source k8s_audit, but not both.
-* That for a mix of source and extractor plugins having the same event source, that the fields are distinct. For example, a source plugin with source k8s_audit can export ka.* fields, and an extractor plugin with event source k8s_audit can export a jevt.value[/...] field, and the appropriate plugin will be used to extract fields from k8s_audit events as fields are parsed from condition expressions/output format strings.
-
-### Plugin Versions and Falco Rules
-
-To allow rules files to document the plugin versions they are compatible with, we will add a new top-level field `required_plugin_versions` to the Falco rules file format. The field is optional, and if not provided no plugin compatibility checks will be performed. The syntax of `required_plugin_versions` will be the following:
-
-```yaml
-- required_plugin_versions:
-  - name: <plugin_name>
-    version: x.y.z
-  ...
-```
-
-Below required_plugin_versions is a list of objects, where each object has `name` and `version` properties. If a plugin is loaded, and if an entry in `required_plugin_versions` has a matching name, then the loaded plugin version must be semver compatible with the version property.
-
-Falco can load multiple rules files, and each file may contain its own `required_plugin_versions` property. In this case, name+version pairs across all files will be merged, and in the case of duplicate names all provided versions must be compatible.
-
-### Loading the plugins
-
-The mechanics of loading a plugin are implemented in the libraries and leverage the dynamic library functionality of the operating system (dlopen/dlsym in unix, LoadLibrary/GetProcAddress in Windows). The plugin loading code also ensures that:
-
-- the plugin is valid, i.e. that it exports the set of expected symbols
-- the plugin has an api version number that is compatible with the libraries instance
-- that only one source plugin is loaded at a time for a given event source
-- if a mix of source and extractor plugins are loaded for a given event source, that the exported fields have unique names that don't overlap across plugins
-
-#### Loading plugins in falcosecurity/libs
-
-At the libraries level, loading plugins is handled via the static method:
-
-```c++
-void sinsp_plugin::register_plugin(sinsp* inspector, string filepath, char* config, ...)
-```
-
-filepath points to a dynamic library containing code that exports plugin API functions. config contains arbitrary config content which is passed to init().
-
-Note that the code using the libraries is responsible for determining the location of plugin libraries and their configuration.
-
-#### Loading plugins in falcosecurity/falco
-
-Falco will control/configure loading plugins via the new "plugins" property in falco.yaml. Here's an example:
-
-```yaml
-plugins:
-  - name: aws_cloudtrail
-    library_path: aws_cloudtrail/plugin.so
-    init_config: "..."
-    open_params: "..."
-  - name: http_json
-    library_path: http_json/plugin.so
-    init_config_file: http_json/config.txt
-    open_params_file: http_json/params.txt
-
-# Optional
-load_plugins: [aws_cloudtrail]
-```
-
-A new "plugins" property in falco.yaml will define the set of plugins that can be loaded by Falco. The property contains a list of objects, with the following properties:
-
-* name: Only used for load_plugins, but by convention should be the same as the value returned by the name() api function.
-* library_path: a path to the shared library. The path can be relative, in which case it is relative to Falco's "share" directory under a "plugins" subdirectory e.g. /usr/share/falco/plugins.
-* init_config: If present, the exact configuration text that will be provided as an argument to the init() function.
-* init_config_file: If present, the provided file will be read and the contents will be provided as an argument to the init() function.
-* open_params: If present, the exact params text that will be provided as an argument to the open() function.
-* open_params_file: If present, the provided file will be read and the contents will be provided as an argument to the open() function.
-
-For a given yaml object in the plugins list, only one of init_config/init_config_file and one of open_params/open_params_file can be provided at a time.
-
-A new "load_plugins" property in falco.yaml will allow for loading a subset of the plugins defined in plugins. If present, only the plugins with the provided names will be loaded.
-
-### Examples
-
-We have an initial version working, consisting of:
-
-* A version of falcosecurity/libs that supports the [plugin framework](https://github.com/falcosecurity/libs/tree/new/plugin-system-api-additions)
-* Support code and examples for [writing plugins in go](https://github.com/mstemm/libsinsp-plugin-sdk-go/tree/new/plugin-system-api-additions)
-* A [cloudtrail](https://github.com/mstemm/plugins/tree/new/plugin-system-api-additions) plugin that can generate events from cloudtrail logs and extract fields from those events.
-* A version of Falco that uses all of the above to [load and evaluate rules with plugins](https://github.com/leogr/falco/tree/new/plugin-system-api-additions)

rules/falco_rules.yaml

@@ -1755,6 +1755,7 @@
     (container.image.repository startswith "602401143452.dkr.ecr" or
      container.image.repository startswith "877085696533.dkr.ecr" or
      container.image.repository startswith "800184023465.dkr.ecr" or
+     container.image.repository startswith "602401143452.dkr.ecr" or
      container.image.repository startswith "918309763551.dkr.ecr" or
      container.image.repository startswith "961992271922.dkr.ecr" or
      container.image.repository startswith "590381155156.dkr.ecr" or
@@ -2746,12 +2747,7 @@
       "xmr-eu1.nanopool.org","xmr-eu2.nanopool.org",
       "xmr-jp1.nanopool.org","xmr-us-east1.nanopool.org",
       "xmr-us-west1.nanopool.org","xmr.crypto-pool.fr",
-      "xmr.pool.minergate.com", "rx.unmineable.com",
-      "ss.antpool.com","dash.antpool.com",
-      "eth.antpool.com","zec.antpool.com",
-      "xmc.antpool.com","btm.antpool.com",
-      "stratum-dash.antpool.com","stratum-xmc.antpool.com",
-      "stratum-btm.antpool.com"
+      "xmr.pool.minergate.com", "rx.unmineable.com"
       ]
 
 - list: https_miner_domains
@@ -2768,12 +2764,7 @@
     "stratum-ltc.antpool.com",
     "stratum-zec.antpool.com",
     "stratum.antpool.com",
-    "xmr.crypto-pool.fr",
-    "ss.antpool.com",
-    "stratum-dash.antpool.com",
-    "stratum-xmc.antpool.com",
-    "stratum-btm.antpool.com",
-    "btm.antpool.com"
+    "xmr.crypto-pool.fr"
   ]
 
 - list: http_miner_domains

scripts/falco-driver-loader

@@ -154,7 +154,7 @@ load_kernel_module_compile() {
 	fi
 
 	# Try to compile using all the available gcc versions
-	for CURRENT_GCC in $(which gcc) $(ls "$(dirname "$(which gcc)")"/gcc-* | grep 'gcc-[0-9]\+' | sort -n -r -k 2 -t -); do
+	for CURRENT_GCC in $(which gcc) $(ls "$(dirname "$(which gcc)")"/gcc-* | grep 'gcc-[0-9]\+' | sort -r); do
 		echo "* Trying to dkms install ${DRIVER_NAME} module with GCC ${CURRENT_GCC}"
 		echo "#!/usr/bin/env bash" > /tmp/falco-dkms-make
 		echo "make CC=${CURRENT_GCC} \$@" >> /tmp/falco-dkms-make

scripts/rpm/falco.service

@@ -19,7 +19,6 @@ ProtectSystem=full
 ProtectKernelTunables=true
 RestrictRealtime=true
 RestrictAddressFamilies=~AF_PACKET
-StandardOutput=null
 
 [Install]
 WantedBy=multi-user.target

test/confs/psp.yaml

@@ -43,12 +43,6 @@ json_output: false
 # (user=root ....") in the json output.
 json_include_output_property: true
 
-# When using json output, whether or not to include the "tags" property
-# itself in the json output. If set to true, outputs caused by rules
-# with no tags will have a "tags" field set to an empty array. If set to
-# false, the "tags" field will not be included in the json output at all.
-json_include_tags_property: true
-
 # Send information logs to stderr and/or syslog Note these are *not* security
 # notification logs! These are just Falco lifecycle (and possibly error) logs.
 log_stderr: true

test/falco_test.py

@@ -90,8 +90,6 @@ class FalcoTest(Test):
         self.json_output = self.params.get('json_output', '*', default=False)
         self.json_include_output_property = self.params.get(
             'json_include_output_property', '*', default=True)
-        self.json_include_tags_property = self.params.get(
-            'json_include_tags_property', '*', default=True)
         self.all_events = self.params.get('all_events', '*', default=False)
         self.priority = self.params.get('priority', '*', default='debug')
         self.rules_file = self.params.get(
@@ -390,11 +388,10 @@ class FalcoTest(Test):
             for line in res.stdout.decode("utf-8").splitlines():
                 if line.startswith('{'):
                     obj = json.loads(line)
-                    attrs = ['time', 'rule', 'priority']
                     if self.json_include_output_property:
-                        attrs.append('output')
-                    if self.json_include_tags_property:
-                        attrs.append('tags')
+                        attrs = ['time', 'rule', 'priority', 'output']
+                    else:
+                        attrs = ['time', 'rule', 'priority']
                     for attr in attrs:
                         if not attr in obj:
                             self.fail(
@@ -412,15 +409,10 @@ class FalcoTest(Test):
             else:
                 actual = open(output['actual']).read()
 
-            actual_cursor = actual
-            expected_lines = expected.splitlines()
-            for line in expected_lines:
-                pos = actual_cursor.find(line)
-                if pos < 0:
-                    self.fail("Output '{}' does not strictly contains the expected content '{}'".format(
-                        output['actual'], output['expected']))
-                    return False
-                actual_cursor = actual_cursor[pos + len(line):]
+            if expected not in actual:
+                self.fail("Output '{}' does not strictly contains the expected content '{}'".format(
+                    output['actual'], output['expected']))
+                return False
 
         return True
 
@@ -617,9 +609,8 @@ class FalcoTest(Test):
                 self.log.debug("Converted Rules: {}".format(psp_rules))
 
         # Run falco
-        cmd = '{} {} {} -c {} {} -o json_output={} -o json_include_output_property={} -o json_include_tags_property={} -o priority={} -v'.format(
-            self.falco_binary_path, self.rules_args, self.disabled_args, self.conf_file, trace_arg, self.json_output,
-            self.json_include_output_property, self.json_include_tags_property, self.priority)
+        cmd = '{} {} {} -c {} {} -o json_output={} -o json_include_output_property={} -o priority={} -v'.format(
+            self.falco_binary_path, self.rules_args, self.disabled_args, self.conf_file, trace_arg, self.json_output, self.json_include_output_property, self.priority)
 
         for tag in self.disable_tags:
             cmd += ' -T {}'.format(tag)

test/falco_tests.yaml

@@ -688,7 +688,7 @@ trace_files: !mux
     detect: True
     detect_level: WARNING
     rules_file:
-      - rules/single_rule_with_tags.yaml
+      - rules/single_rule.yaml
     conf_file: confs/stdout_output.yaml
     trace_file: trace_files/cat_write.scap
     time_iso_8601: true
@@ -721,7 +721,7 @@ trace_files: !mux
     detect: True
     detect_level: WARNING
     rules_file:
-      - rules/single_rule_with_tags.yaml
+      - rules/single_rule.yaml
     conf_file: confs/grpc_unix_socket.yaml
     trace_file: trace_files/cat_write.scap
     run_duration: 5
@@ -745,10 +745,6 @@ trace_files: !mux
         # For the hostname, since we don't know that beforehand,
         # only check the field presence
         - "hostname: "
-        #tags
-        - "tags: \"filesystem\""
-        - "tags: \"process\""
-        - "tags: \"testing\""
 
   detect_counts:
     detect: True
@@ -1111,25 +1107,6 @@ trace_files: !mux
     trace_file: trace_files/cat_write.scap
     stdout_contains: "^(?!.*Warning An open of /dev/null was seen.*)"
 
-  json_output_no_tags_property:
-    json_output: True
-    json_include_tags_property: False
-    detect: True
-    detect_level: WARNING
-    rules_file:
-      - rules/rule_append.yaml
-    trace_file: trace_files/cat_write.scap
-    stdout_contains: "^(?!.*\"tags\":[ ]*\\[.*\\],.*)"
-
-  json_output_empty_tags_property:
-    json_output: True
-    detect: True
-    detect_level: WARNING
-    rules_file:
-      - rules/rule_append.yaml
-    trace_file: trace_files/cat_write.scap
-    stdout_contains: "^(.*\"tags\":[ ]*\\[\\],.*)"
-
   in_operator_netmasks:
     detect: True
     detect_level: INFO

test/output_files/single_rule_with_cat_write.json

@@ -1,8 +1,8 @@
-{"output":"2016-08-04T16:17:57.881781397+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","source":"syscall","tags":["filesystem","process","testing"],"time":"2016-08-04T16:17:57.881781397Z", "output_fields": {"evt.time.iso8601":1470327477881781397,"proc.cmdline":"cat /dev/null"}}
-{"output":"2016-08-04T16:17:57.881785348+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","source":"syscall","tags":["filesystem","process","testing"],"time":"2016-08-04T16:17:57.881785348Z", "output_fields": {"evt.time.iso8601":1470327477881785348,"proc.cmdline":"cat /dev/null"}}
-{"output":"2016-08-04T16:17:57.881796705+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","source":"syscall","tags":["filesystem","process","testing"],"time":"2016-08-04T16:17:57.881796705Z", "output_fields": {"evt.time.iso8601":1470327477881796705,"proc.cmdline":"cat /dev/null"}}
-{"output":"2016-08-04T16:17:57.881799840+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","source":"syscall","tags":["filesystem","process","testing"],"time":"2016-08-04T16:17:57.881799840Z", "output_fields": {"evt.time.iso8601":1470327477881799840,"proc.cmdline":"cat /dev/null"}}
-{"output":"2016-08-04T16:17:57.882003104+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","source":"syscall","tags":["filesystem","process","testing"],"time":"2016-08-04T16:17:57.882003104Z", "output_fields": {"evt.time.iso8601":1470327477882003104,"proc.cmdline":"cat /dev/null"}}
-{"output":"2016-08-04T16:17:57.882008208+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","source":"syscall","tags":["filesystem","process","testing"],"time":"2016-08-04T16:17:57.882008208Z", "output_fields": {"evt.time.iso8601":1470327477882008208,"proc.cmdline":"cat /dev/null"}}
-{"output":"2016-08-04T16:17:57.882045694+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","source":"syscall","tags":["filesystem","process","testing"],"time":"2016-08-04T16:17:57.882045694Z", "output_fields": {"evt.time.iso8601":1470327477882045694,"proc.cmdline":"cat /dev/null"}}
-{"output":"2016-08-04T16:17:57.882054739+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","source":"syscall","tags":["filesystem","process","testing"],"time":"2016-08-04T16:17:57.882054739Z", "output_fields": {"evt.time.iso8601":1470327477882054739,"proc.cmdline":"cat /dev/null"}}
+{"output":"2016-08-04T16:17:57.881781397+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","time":"2016-08-04T16:17:57.881781397Z", "output_fields": {"evt.time.iso8601":1470327477881781397,"proc.cmdline":"cat /dev/null"}}
+{"output":"2016-08-04T16:17:57.881785348+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","time":"2016-08-04T16:17:57.881785348Z", "output_fields": {"evt.time.iso8601":1470327477881785348,"proc.cmdline":"cat /dev/null"}}
+{"output":"2016-08-04T16:17:57.881796705+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","time":"2016-08-04T16:17:57.881796705Z", "output_fields": {"evt.time.iso8601":1470327477881796705,"proc.cmdline":"cat /dev/null"}}
+{"output":"2016-08-04T16:17:57.881799840+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","time":"2016-08-04T16:17:57.881799840Z", "output_fields": {"evt.time.iso8601":1470327477881799840,"proc.cmdline":"cat /dev/null"}}
+{"output":"2016-08-04T16:17:57.882003104+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","time":"2016-08-04T16:17:57.882003104Z", "output_fields": {"evt.time.iso8601":1470327477882003104,"proc.cmdline":"cat /dev/null"}}
+{"output":"2016-08-04T16:17:57.882008208+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","time":"2016-08-04T16:17:57.882008208Z", "output_fields": {"evt.time.iso8601":1470327477882008208,"proc.cmdline":"cat /dev/null"}}
+{"output":"2016-08-04T16:17:57.882045694+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","time":"2016-08-04T16:17:57.882045694Z", "output_fields": {"evt.time.iso8601":1470327477882045694,"proc.cmdline":"cat /dev/null"}}
+{"output":"2016-08-04T16:17:57.882054739+0000: Warning An open was seen (command=cat /dev/null)","priority":"Warning","rule":"open_from_cat","time":"2016-08-04T16:17:57.882054739Z", "output_fields": {"evt.time.iso8601":1470327477882054739,"proc.cmdline":"cat /dev/null"}}

test/rules/single_rule_with_tags.yaml

@@ -1,34 +0,0 @@
-#
-# Copyright (C) 2021 The Falco Authors.
-#
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-#
-
-- required_engine_version: 2
-
-- list: cat_binaries
-  items: [cat]
-
-- list: cat_capable_binaries
-  items: [cat_binaries]
-
-- macro: is_cat
-  condition: proc.name in (cat_capable_binaries)
-
-- rule: open_from_cat
-  desc: A process named cat does an open
-  condition: evt.type=open and is_cat
-  output: "An open was seen (command=%proc.cmdline)"
-  priority: WARNING
-  tags: [filesystem, process, testing]

userspace/engine/CMakeLists.txt

@@ -70,3 +70,5 @@ else()
     FILES_MATCHING
     PATTERN *.lua)
 endif()
+
+add_subdirectory(embeddable)

userspace/engine/embeddable/CMakeLists.txt

@@ -0,0 +1,43 @@
+#
+# Copyright (C) 2021 The Falco Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations under the License.
+
+set(FALCO_ENGINE_EMBEDDABLE_SOURCE_FILES
+    falco_engine_embeddable.cpp)
+
+set(
+  FALCO_LIBRARIES
+  falco_engine
+  sinsp
+  "${LIBYAML_LIB}"
+  "${YAMLCPP_LIB}"
+)
+
+add_library(falco_engine_embeddable SHARED ${FALCO_ENGINE_EMBEDDABLE_SOURCE_FILES})
+add_dependencies(falco_engine_embeddable falco_engine)
+
+target_include_directories(
+  falco_engine_embeddable
+  PUBLIC
+    "${PROJECT_SOURCE_DIR}/userspace/engine"
+    "${LUAJIT_INCLUDE}"
+    "${NJSON_INCLUDE}"
+    "${TBB_INCLUDE_DIR}"
+    "${STRING_VIEW_LITE_INCLUDE}"
+    "${LIBSCAP_INCLUDE_DIRS}"
+    "${LIBSINSP_INCLUDE_DIRS}"
+    "${PROJECT_BINARY_DIR}/userspace/engine")
+
+target_link_libraries(falco_engine_embeddable ${FALCO_LIBRARIES})
+
+#add_custom_target(example ALL DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/example)
+#add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/example COMMAND go build ${CMAKE_CURRENT_SOURCE_DIR}/example.go -o ${CMAKE_CURRENT_BINARY_DIR}/example)
+

userspace/engine/embeddable/example/example.go

@@ -0,0 +1,102 @@
+package main
+
+//#cgo CFLAGS: -I../
+//#cgo LDFLAGS: -L/home/mstemm/work/falco-build/userspace/engine/embeddable -lfalco_engine_embeddable -Wl,-rpath=/home/mstemm/work/falco-build/userspace/engine/embeddable
+/*
+#include "stdio.h"
+#include "falco_engine_embeddable.h"
+
+int open_engine(void **engine, void *rules_content)
+{
+   int32_t rc;
+   *engine = falco_engine_embed_init(&rc);
+
+   if (rc != 0)
+   {
+   return rc;
+   }
+
+   char *errstr;
+   rc = falco_engine_embed_load_rules_content(*engine, (const char *) rules_content, &errstr);
+
+   if (rc != 0)
+   {
+   fprintf(stderr, "%s", errstr);
+   return rc;
+   }
+
+   rc = falco_engine_embed_open(*engine, &errstr);
+
+   if (rc != 0)
+   {
+   fprintf(stderr, "%s", errstr);
+   return rc;
+   }
+
+   return rc;
+}
+
+int next_result(void *engine, char **output)
+{
+
+   int32_t rc;
+   falco_engine_embed_result *res;
+   char *errstr;
+
+   rc = falco_engine_embed_next_result(engine, &res, &errstr);
+
+   if (rc != 0)
+   {
+   fprintf(stderr, "NEXT ERROR %s", errstr);
+   return rc;
+   }
+
+   *output = res->output_str;
+   return rc;
+
+}
+
+*/
+import "C"
+
+import (
+	"fmt"
+	"io/ioutil"
+	"os"
+	"unsafe"
+)
+
+func doMain(rules_filename string) int {
+
+	rules_content, err := ioutil.ReadFile(rules_filename)
+	if err != nil {
+		fmt.Printf("Could not open rules file %s: %v", rules_filename, err)
+		return 1
+	}
+
+	var handle unsafe.Pointer
+	rc := C.open_engine(&handle, C.CBytes(rules_content))
+
+	if rc != 0 {
+		fmt.Printf("Could not open falco engine")
+		return 1
+	}
+
+	for true {
+		var output *C.char
+		rc := C.next_result(handle, &output)
+		if rc != 0 {
+			fmt.Printf("Could not get next result")
+			return 1
+		}
+		fmt.Printf("GOT RESULT %s\n", C.GoString(output))
+	}
+
+	return 0
+}
+
+func main() {
+	os.Exit(doMain(os.Args[1]))
+}
+
+

userspace/engine/embeddable/example/go.mod

@@ -0,0 +1,6 @@
+module github.com/falcosecurity/falco/embedded/example
+
+go 1.16
+
+require (
+)

userspace/engine/embeddable/falco_engine_embeddable.cpp

@@ -0,0 +1,356 @@
+/*
+Copyright (C) 2021 The Falco Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+#include <memory>
+#include <atomic>
+
+#include <sinsp.h>
+#include <event.h>
+
+#include <falco_engine.h>
+#include "falco_engine_embeddable.h"
+
+using namespace std;
+
+class falco_engine_embed_int {
+public:
+	falco_engine_embed_int();
+	virtual ~falco_engine_embed_int();
+
+	bool load_rules_content(const char *rules_content, string &err);
+	bool is_open();
+	bool open(string &err);
+	bool close(string &err);
+	falco_engine_embed_rc next_result(falco_engine_embed_result **result, string &err);
+
+private:
+
+	falco_engine_embed_result *rule_result_to_embed_result(sinsp_evt *ev,
+							       unique_ptr<falco_engine::rule_result> &res);
+
+	static void add_output_pair(const string &field, const string &val,
+				    char **&fields, char **&vals,
+				    uint32_t &len);
+
+	unique_ptr<sinsp_evt_formatter_cache> m_formatters;
+	bool m_open;
+	unique_ptr<sinsp> m_inspector;
+	unique_ptr<falco_engine> m_falco_engine;
+	atomic<bool> m_shutdown;
+};
+
+falco_engine_embed_int::falco_engine_embed_int()
+        : m_open(false),
+	m_shutdown(false)
+{
+	m_inspector.reset(new sinsp());
+	m_falco_engine.reset(new falco_engine());
+	m_falco_engine->set_inspector(m_inspector.get());
+
+	m_formatters.reset(new sinsp_evt_formatter_cache(m_inspector.get()));
+}
+
+falco_engine_embed_int::~falco_engine_embed_int()
+{
+}
+
+bool falco_engine_embed_int::load_rules_content(const char *rules_content, string &err)
+{
+	bool verbose = false;
+	bool all_events = true;
+
+	try {
+		m_falco_engine->load_rules(string(rules_content), verbose, all_events);
+	}
+	catch(falco_exception &e)
+	{
+		err = e.what();
+		return false;
+	}
+
+	return true;
+}
+
+bool falco_engine_embed_int::is_open()
+{
+	return m_open;
+}
+
+bool falco_engine_embed_int::open(string &err)
+{
+	try {
+		m_inspector->open();
+	}
+	catch(exception &e)
+	{
+		err = e.what();
+		return false;
+	}
+
+	m_open = true;
+	return true;
+}
+
+bool falco_engine_embed_int::close(string &err)
+{
+	m_shutdown = true;
+	m_open = false;
+
+	return true;
+}
+
+falco_engine_embed_rc falco_engine_embed_int::next_result(falco_engine_embed_result **result, string &err)
+{
+	*result = NULL;
+
+	while(!m_shutdown)
+	{
+		sinsp_evt* ev;
+
+		int32_t rc = m_inspector->next(&ev);
+
+		if (rc == SCAP_TIMEOUT)
+		{
+			continue;
+		}
+		else if (rc == SCAP_EOF)
+		{
+			break;
+		}
+		else if (rc != SCAP_SUCCESS)
+		{
+			err = m_inspector->getlasterr();
+			return FE_EMB_RC_ERROR;
+		}
+
+		if(!ev->simple_consumer_consider())
+		{
+			continue;
+		}
+
+		unique_ptr<falco_engine::rule_result> res = m_falco_engine->process_sinsp_event(ev);
+		if(!res)
+		{
+			continue;
+		}
+
+		*result = rule_result_to_embed_result(ev, res);
+
+		return FE_EMB_RC_OK;
+	}
+
+	// Can only get here if shut down/eof.
+	return FE_EMB_RC_EOF;
+}
+
+falco_engine_embed_result * falco_engine_embed_int::rule_result_to_embed_result(sinsp_evt *ev,
+										unique_ptr<falco_engine::rule_result> &res)
+{
+	falco_engine_embed_result *result;
+
+	result = (falco_engine_embed_result *) malloc(sizeof(falco_engine_embed_result));
+
+	result->rule = strdup(res->rule.c_str());
+	result->event_source = strdup(res->source.c_str());
+	result->priority_num = res->priority_num;
+
+	// Copy output format string without resolving fields.
+	result->output_format_str = strdup(res->format.c_str());
+
+	// Resolve output format string into resolved output
+	string output;
+	m_formatters->tostring(ev, res->format, &output);
+	result->output_str = strdup(output.c_str());
+
+	result->output_fields = NULL;
+	result->output_values = NULL;
+	result->num_output_values = 0;
+
+	map<string, string> rule_output_fields;
+	m_formatters->resolve_tokens(ev, res->format, rule_output_fields);
+	for(auto &pair : rule_output_fields)
+	{
+		add_output_pair(pair.first, pair.second,
+				result->output_fields, result->output_values,
+				result->num_output_values);
+	}
+
+	// Preceding * makes the formatting permissive (not ending at first empty value)
+	std::string exformat = "*";
+	for (const auto& exfield : res->exception_fields)
+	{
+		exformat += " %" + exfield;
+	}
+
+	map<string, string> exception_output_fields;
+	m_formatters->resolve_tokens(ev, exformat, exception_output_fields);
+	for(auto &pair : exception_output_fields)
+	{
+		add_output_pair(pair.first, pair.second,
+				result->output_fields, result->output_values,
+				result->num_output_values);
+	}
+
+	return result;
+}
+
+void falco_engine_embed_int::add_output_pair(const string &field, const string &val,
+					     char **&fields, char **&vals,
+					     uint32_t &len)
+{
+	len++;
+	fields = (char **) realloc(fields, len*sizeof(char *));
+	vals = (char **) realloc(vals, len*sizeof(char *));
+	fields[len-1] = strdup(field.c_str());
+	vals[len-1] = strdup(val.c_str());
+}
+
+static const char *FALCO_ENGINE_EMBED_VERSION = "1.0.0";
+
+char *falco_engine_embed_get_version()
+{
+	return strdup(FALCO_ENGINE_EMBED_VERSION);
+}
+
+void falco_engine_embed_free_result(falco_engine_embed_result *result)
+{
+	free(result->rule);
+	free(result->event_source);
+	free(result->output_format_str);
+	free(result->output_str);
+
+	for(int32_t i; i < result->num_output_values; i++)
+	{
+		free(result->output_fields[i]);
+		free(result->output_values[i]);
+	}
+	free(result->output_fields);
+	free(result->output_values);
+	free(result);
+}
+
+falco_engine_embed_t* falco_engine_embed_init(int32_t *rc)
+{
+	falco_engine_embed_int *eengine = new falco_engine_embed_int();
+
+	*rc = FE_EMB_RC_OK;
+
+	return eengine;
+}
+
+int32_t falco_engine_embed_destroy(falco_engine_embed_t *engine, char *errstr)
+{
+	falco_engine_embed_int *eengine = (falco_engine_embed_int *) engine;
+
+	if(eengine->is_open())
+	{
+		errstr = strdup("Engine is open--must call close() first");
+		return FE_EMB_RC_ERROR;
+	}
+
+	delete(eengine);
+
+	return FE_EMB_RC_OK;
+}
+
+int32_t falco_engine_embed_load_plugin(falco_engine_embed_t *engine,
+				       const char *path,
+				       const char* init_config,
+				       const char* open_params,
+				       char **errstr)
+{
+	falco_engine_embed_int *eengine = (falco_engine_embed_int *) engine;
+
+	// XXX/mstemm fill in
+	return FE_EMB_RC_OK;
+}
+
+int32_t falco_engine_embed_load_rules_content(falco_engine_embed_t *engine,
+					      const char *rules_content,
+					      char **errstr)
+{
+	falco_engine_embed_int *eengine = (falco_engine_embed_int *) engine;
+	std::string err;
+
+	if (!eengine->load_rules_content(rules_content, err))
+	{
+		*errstr = strdup(err.c_str());
+		return FE_EMB_RC_ERROR;
+	}
+
+	return FE_EMB_RC_OK;
+}
+
+int32_t falco_engine_embed_enable_source(falco_engine_embed_t *engine,
+					 int32_t source,
+					 bool enabled,
+					 char **errstr)
+{
+	falco_engine_embed_int *eengine = (falco_engine_embed_int *) engine;
+
+	// XXX/mstemm fill in
+	return FE_EMB_RC_OK;
+}
+
+int32_t falco_engine_embed_open(falco_engine_embed_t *engine,
+				char **errstr)
+{
+	falco_engine_embed_int *eengine = (falco_engine_embed_int *) engine;
+	std::string err;
+
+	if (!eengine->open(err))
+	{
+		*errstr = strdup(err.c_str());
+		return FE_EMB_RC_ERROR;
+	}
+
+	return FE_EMB_RC_OK;
+}
+
+int32_t falco_engine_embed_close(falco_engine_embed_t *engine,
+				 char **errstr)
+{
+	falco_engine_embed_int *eengine = (falco_engine_embed_int *) engine;
+	std::string err;
+
+	if (!eengine->close(err))
+	{
+		*errstr = strdup(err.c_str());
+		return FE_EMB_RC_ERROR;
+	}
+
+	return FE_EMB_RC_OK;
+}
+
+int32_t falco_engine_embed_next_result(falco_engine_embed_t *engine,
+				       falco_engine_embed_result **result,
+				       char **errstr)
+{
+	falco_engine_embed_int *eengine = (falco_engine_embed_int *) engine;
+	std::string err;
+	falco_engine_embed_rc rc;
+
+	rc = eengine->next_result(result, err);
+
+	if(rc == FE_EMB_RC_ERROR)
+	{
+		*errstr = strdup(err.c_str());
+	}
+
+	return rc;
+}
+
+

userspace/engine/embeddable/falco_engine_embeddable.h

@@ -0,0 +1,268 @@
+/*
+  Copyright (C) 2021 The Falco Authors.
+
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License.
+*/
+
+/*
+  This header file provides a C-only interface to the falco engine,
+  suitable for embedding in other programs as a shared library. This
+  interface handles:
+  - Loading Rules Content
+  - Enabling/Disabling syscall/k8s_audit event sources.
+  - Loading and configuring source/extractor plugins
+  - Starting/Stopping the event processing loop.
+
+  After setup, the main interface involves receiving "results" when
+  syscall/k8s_audit/plugin events match rules.
+
+  This interface does not provide as many features as the c++
+  falco_engine interface, such as interfaces to list rules, segregate
+  rules by "ruleset", enabling/disabling specific rules etc.
+
+  Output handling (e.g. routing alerts to files, stdout, webhook,
+  slack, etc) is not covered by this interface. After receiving a
+  result, a program could use a program like falcosidekick for a rich
+  set of output handling methods.
+
+*/
+
+#pragma once
+
+#include <stdint.h>
+#include <stdbool.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+/* A handle to an embeddable falco engine */
+typedef void falco_engine_embed_t;
+
+/* Defined return values from API functions. */
+enum falco_engine_embed_rc
+{
+	/* No Error */
+	FE_EMB_RC_OK = 0,
+	FE_EMB_RC_ERROR = 1,
+	FE_EMB_RC_EOF = 2,
+};
+
+/* Defined event sources. */
+enum falco_engine_embed_evt_source
+{
+	FE_EMB_SRC_NONE = 0,
+	FE_EMB_SRC_SYSCALL = 1,
+	FE_EMB_K8S_AUDIT = 2,
+	FE_EMB_PLUGINS = 3,   // This includes any event from any plugin
+};
+
+/* Represents a result (e.g. an event matching a falco rule)
+
+   When returned by a call to next_result(), the struct, as well as
+   every allocated char * within the struct, is allocated via a call
+   to malloc() and must be freed via a call to free().
+*/
+typedef struct falco_engine_embed_result
+{
+	// The rule that matched the event
+	char *rule;
+
+	// The event source of the event that matched the rule
+	char *event_source;
+
+	// An int containing a falco_common::priority_type value of
+	// the priority of the matching rule.
+	int32_t priority_num;
+
+	// A copy of the rule's output string, *without* any
+	// fields (e.g. %proc.name, ...) resolved to values.
+	char *output_format_str;
+
+	// An output string, starting with the rule's output string
+	// with all fields resolved to values.
+	char *output_str;
+
+	// An allocated array of allocated field names from the output
+	// string. Additional fields + values may be included in
+	// addition to those in the output string, to aid in
+	// debugging. Item i in this array maps to item i in
+	// output_values.
+	char **output_fields;
+
+	// An allocated array of allocated field values from the
+	// output string. Additional fields + values may be included in
+	// addition to those in the output string, to aid in
+	// debugging. Item i in this array maps to item i in
+	// output_fields.
+	char **output_values;
+
+	// The length of output_fields/output_values
+	uint32_t num_output_values;
+} falco_engine_embed_result;
+
+/* A utility function to free a falco_engine_embed_result struct and
+ * its allocated strings returned by a call to next_result() */
+void falco_engine_embed_free_result(falco_engine_embed_result *result);
+
+// Interface to interact with an embeddable falco engine.
+
+// NOTE: For all functions below that return a char *, the memory
+// pointed to by the char * is allocated using malloc() and should be
+// freed by the caller using free().
+
+// Return the embedded engine version.
+//
+// Return value: a version string, in the following format:
+//        "<major>.<minor>.<patch>", e.g. "1.2.3".
+// This interface is compatible following semver conventions:
+// <major> changes for incompatible api changes, <minor> for
+// backwards-compatible additions, <patch> for compatible bug
+// fixes.
+char* falco_engine_embed_get_version();
+
+// Initialize a falco engine.
+//
+// Arguments:
+// - rc: pointer to an integer containing a falco_engine_embed_rc value.
+//
+// Return value: pointer to the engine state that is passed to
+// other API functions.
+falco_engine_embed_t* falco_engine_embed_init(int32_t *rc);
+
+// Destroy a falco engine. This frees any resources allocated in
+// init(). If open() has been called, close() should be called before
+// destroy().
+//
+// Arguments:
+// - engine: returned by a prior succesful call to init().
+// - errstr: on error, errstr will point to an allocated
+//   string with additional details on the errror. The string
+//   must be freed via a call to free().
+//
+// Return value: an integer containing a falco_engine_embed_rc
+// value.
+int32_t falco_engine_embed_destroy(falco_engine_embed_t *engine, char *errstr);
+
+// Load either a falco source or extractor plugin.
+//
+// Arguments:
+// - engine: returned by a prior succesful call to init().
+// - path: a file path pointing to a dynamic library that
+//   can be dlopen()ed.
+// - init_config: a string that will be passed to the plugin's
+//   init() function.
+// - open_params: a string that will be passed to the
+//   plugin's open() function.
+// - errstr: on error, errstr will point to an allocated
+//   string with additional details on the errror. The string
+//   must be freed via a call to free().
+//
+// Return value: an integer containing a falco_engine_embed_rc
+// value.
+int32_t falco_engine_embed_load_plugin(falco_engine_embed_t *engine,
+				       const char *path,
+				       const char* init_config,
+				       const char* open_params,
+				       char **errstr);
+
+// Load the provided rules content. These rules are applied on
+// top of any previously loaded rules content
+// (e.g. appending/overriding rule/macro/list objects as
+// specified via "append:" properties)
+//
+// NOTE: Plugins should be loaded before any rules are loaded.
+//
+// Arguments:
+// - engine: returned by a prior succesful call to init().
+// - rules_content: a null-terminated string containing
+//   yaml rules content.
+// - errstr: on error, errstr will point to an allocated
+//   string with additional details on the errror. The string
+//   must be freed via a call to free().
+//
+// Return value: an integer containing a falco_engine_embed_rc
+// value.
+int32_t falco_engine_embed_load_rules_content(falco_engine_embed_t *engine,
+					      const char *rules_content,
+					      char **errstr);
+
+// Enable/disable an event source.
+// By default all event sources are enabled. This function
+// enables/disables specific event sources.
+//
+// Arguments:
+// - engine: returned by a prior succesful call to init().
+// - source: an int containing a falco_engine_embed_evt_source value.
+// - enabled: whether to enable or disable the provided source
+// - errstr: on error, errstr will point to an allocated
+//   string with additional details on the errror. The string
+//   must be freed via a call to free().
+//
+// Return value: an integer containing a falco_engine_embed_rc
+// value.
+int32_t falco_engine_embed_enable_source(falco_engine_embed_t *engine,
+					 int32_t source,
+					 bool enabled,
+					 char **errstr);
+
+// Open the engine, which starts event processing and matching
+// against the loaded set of rules.
+//
+// Arguments:
+// - engine: returned by a prior succesful call to init().
+// - errstr: on error, errstr will point to an allocated
+//   string with additional details on the errror. The string
+//   must be freed via a call to free().
+//
+// Return value: an integer containing a falco_engine_embed_rc
+// value.
+int32_t falco_engine_embed_open(falco_engine_embed_t *engine,
+				char **errstr);
+
+// Close the engine, which stops event processing.
+//
+// Arguments:
+// - engine: returned by a prior succesful call to init().
+// - errstr: on error, errstr will point to an allocated
+//   string with additional details on the errror. The string
+//   must be freed via a call to free().
+//
+// Return value: an integer containing a falco_engine_embed_rc
+// value.
+int32_t falco_engine_embed_close(falco_engine_embed_t *engine,
+				 char **errstr);
+
+// Receive the next result (e.g. an event that matched a
+// rule). This function blocks until the next result is
+// available. close() is called, or an error occurs.
+//
+// Arguments:
+// - engine: returned by a prior succesful call to init().
+// - result: a pointer to a falco_engine_embed_result struct
+//   pointer. On success, a struct will be allocated, and filled in
+//   with allocated char* values, and the pointer updated to point to
+//   the allocated struct.
+// - errstr: on error, errstr will point to an allocated
+//   string with additional details on the errror. The string
+//   must be freed via a call to free().
+//
+// Return value: an integer containing a falco_engine_embed_rc
+// value.
+int32_t falco_engine_embed_next_result(falco_engine_embed_t *engine,
+				       falco_engine_embed_result **result,
+				       char **errstr);
+#ifdef __cplusplus
+} // extern "C"
+#endif
+

userspace/engine/falco_engine.cpp

@@ -177,8 +177,7 @@ void falco_engine::load_rules(const string &rules_content, bool verbose, bool al
 	// json_output to false.
 	bool json_output = false;
 	bool json_include_output_property = false;
-	bool json_include_tags_property = false;
-	falco_formats::init(m_inspector, this, m_ls, json_output, json_include_output_property, json_include_tags_property);
+	falco_formats::init(m_inspector, this, m_ls, json_output, json_include_output_property);
 
 	m_rules->load_rules(rules_content, verbose, all_events, m_extra, m_replace_container_info, m_min_priority, required_engine_version);
 }
@@ -343,29 +342,18 @@ void falco_engine::populate_rule_result(unique_ptr<struct rule_result> &res, gen
 	if(lua_isfunction(m_ls, -1))
 	{
 		lua_pushnumber(m_ls, ev->get_check_id());
-		if(lua_pcall(m_ls, 1, 5, 0) != 0)
+
+		if(lua_pcall(m_ls, 1, 4, 0) != 0)
 		{
 			const char* lerr = lua_tostring(m_ls, -1);
 			string err = "Error invoking function output: " + string(lerr);
 			throw falco_exception(err);
 		}
-		const char *p =  lua_tostring(m_ls, -5);
+		const char *p =  lua_tostring(m_ls, -4);
 		res->rule = p;
 		res->evt = ev;
-		res->priority_num = (falco_common::priority_type) lua_tonumber(m_ls, -4);
-		res->format = lua_tostring(m_ls, -3);
-
-		// Tags are passed back as a table, and is on the top of the stack
-		lua_pushnil(m_ls);  /* first key */
-		while (lua_next(m_ls, -2) != 0) {
-			// key is at index -2, value is at index
-			// -1. We want the value.
-			res->tags.insert(luaL_checkstring(m_ls, -1));
-
-			// Remove value, keep key for next iteration
-			lua_pop(m_ls, 1);
-		}
-		lua_pop(m_ls, 1); // Clean table leftover
+		res->priority_num = (falco_common::priority_type) lua_tonumber(m_ls, -3);
+		res->format = lua_tostring(m_ls, -2);
 
 		// Exception fields are passed back as a table
 		lua_pushnil(m_ls);  /* first key */

userspace/engine/falco_engine.h

@@ -161,7 +161,6 @@ public:
 		falco_common::priority_type priority_num;
 		std::string format;
 		std::set<std::string> exception_fields;
-		std::set<std::string> tags;
 	};
 
 	//

userspace/engine/falco_engine_version.h

@@ -21,4 +21,4 @@ limitations under the License.
 // This is the result of running "falco --list -N | sha256sum" and
 // represents the fields supported by this version of falco. It's used
 // at build time to detect a changed set of fields.
-#define FALCO_FIELDS_CHECKSUM "2f324e2e66d4b423f53600e7e0fcf2f0ff72e4a87755c490f2ae8f310aba9386"
+#define FALCO_FIELDS_CHECKSUM "8183621f52451d842036eee409e2ed920d9c91bab33e0c4a44e4a871378d103f"

userspace/engine/formats.cpp

@@ -24,7 +24,6 @@ sinsp *falco_formats::s_inspector = NULL;
 falco_engine *falco_formats::s_engine = NULL;
 bool falco_formats::s_json_output = false;
 bool falco_formats::s_json_include_output_property = true;
-bool falco_formats::s_json_include_tags_property = true;
 std::unique_ptr<sinsp_evt_formatter_cache> falco_formats::s_formatters = NULL;
 
 const static struct luaL_Reg ll_falco[] =
@@ -37,14 +36,12 @@ void falco_formats::init(sinsp *inspector,
 			 falco_engine *engine,
 			 lua_State *ls,
 			 bool json_output,
-			 bool json_include_output_property,
-			 bool json_include_tags_property)
+			 bool json_include_output_property)
 {
 	s_inspector = inspector;
 	s_engine = engine;
 	s_json_output = json_output;
 	s_json_include_output_property = json_include_output_property;
-	s_json_include_tags_property = json_include_tags_property;
 
 	// todo(leogr): we should have used std::make_unique, but we cannot since it's not C++14
 	s_formatters = std::unique_ptr<sinsp_evt_formatter_cache>(new sinsp_evt_formatter_cache(s_inspector));
@@ -117,7 +114,7 @@ int falco_formats::lua_free_formatter(lua_State *ls)
 }
 
 string falco_formats::format_event(const gen_event *evt, const std::string &rule, const std::string &source,
-				   const std::string &level, const std::string &format, std::set<std::string> &tags)
+				   const std::string &level, const std::string &format)
 {
 
 	string line;
@@ -184,10 +181,8 @@ string falco_formats::format_event(const gen_event *evt, const std::string &rule
 	if(s_json_output)
 	{
 		Json::Value event;
-		Json::Value rule_tags;
 		Json::FastWriter writer;
 		string full_line;
-		unsigned int rule_tags_idx = 0;
 
 		// Convert the time-as-nanoseconds to a more json-friendly ISO8601.
 		time_t evttime = evt->get_ts() / 1000000000;
@@ -202,30 +197,12 @@ string falco_formats::format_event(const gen_event *evt, const std::string &rule
 		event["time"] = iso8601evttime;
 		event["rule"] = rule;
 		event["priority"] = level;
-		event["source"] = source;
 
 		if(s_json_include_output_property)
 		{
 			// This is the filled-in output line.
 			event["output"] = line;
 		}
-		
-		if(s_json_include_tags_property)
-		{
-			if (tags.size() == 0) 
-			{
-				// This sets an empty array
-				rule_tags = Json::arrayValue;
-			}
-			else
-			{
-				for (auto &tag : tags)
-				{
-					rule_tags[rule_tags_idx++] = tag;
-				}
-			}
-			event["tags"] = rule_tags;
-		}
 
 		full_line = writer.write(event);
 

userspace/engine/formats.h

@@ -37,8 +37,7 @@ public:
 			 falco_engine *engine,
 			 lua_State *ls,
 			 bool json_output,
-			 bool json_include_output_property,
-			 bool json_include_tags_property);
+			 bool json_include_output_property);
 
 	// formatter = falco.formatter(format_string)
 	static int lua_formatter(lua_State *ls);
@@ -47,7 +46,7 @@ public:
 	static int lua_free_formatter(lua_State *ls);
 
 	static string format_event(const gen_event *evt, const std::string &rule, const std::string &source,
-				   const std::string &level, const std::string &format, std::set<std::string> &tags);
+				   const std::string &level, const std::string &format);
 
 	static map<string, string> resolve_tokens(const gen_event *evt, const std::string &source,
 						  const std::string &format);
@@ -57,5 +56,4 @@ public:
 	static std::unique_ptr<sinsp_evt_formatter_cache> s_formatters;
 	static bool s_json_output;
 	static bool s_json_include_output_property;
-	static bool s_json_include_tags_property;
 };

userspace/engine/lua/rule_loader.lua

@@ -1156,7 +1156,7 @@ function on_event(rule_id)
       error ("rule_loader.on_event(): could not find rule by name: ", rule.rule)
    end
 
-   return rule.rule, rule.priority_num, output, combined_rule.exception_fields, rule.tags
+   return rule.rule, rule.priority_num, output, combined_rule.exception_fields
 end
 
 function print_stats()

userspace/falco/configuration.cpp

@@ -71,7 +71,6 @@ void falco_configuration::init(string conf_filename, list<string> &cmdline_optio
 
 	m_json_output = m_config->get_scalar<bool>("json_output", false);
 	m_json_include_output_property = m_config->get_scalar<bool>("json_include_output_property", true);
-	m_json_include_tags_property = m_config->get_scalar<bool>("json_include_tags_property", true);
 
 	falco::outputs::config file_output;
 	file_output.name = "file";
@@ -253,19 +252,6 @@ void falco_configuration::init(string conf_filename, list<string> &cmdline_optio
 	{
 		throw logic_error("Error reading config file(" + m_config_file + "): the maximum consecutive timeouts without an event must be an unsigned integer > 0");
 	}
-
-	m_metadata_download_max_mb = m_config->get_scalar<uint32_t>("metadata_download", "max_mb", 100);
-	if(m_metadata_download_max_mb > 1024)
-	{
-		throw logic_error("Error reading config file(" + m_config_file + "): metadata download maximum size should be < 1024 Mb");
-	}
-	m_metadata_download_chunk_wait_us = m_config->get_scalar<uint32_t>("metadata_download", "chunk_wait_us", 1000);
-	m_metadata_download_watch_freq_sec = m_config->get_scalar<uint32_t>("metadata_download", "watch_freq_sec", 1);
-	if(m_metadata_download_watch_freq_sec == 0)
-	{
-		throw logic_error("Error reading config file(" + m_config_file + "): metadata download watch frequency seconds must be an unsigned integer > 0");
-	}
-	
 }
 
 void falco_configuration::read_rules_file_directory(const string &path, list<string> &rules_filenames)

userspace/falco/configuration.h

@@ -195,7 +195,6 @@ public:
 	std::list<std::string> m_rules_filenames;
 	bool m_json_output;
 	bool m_json_include_output_property;
-	bool m_json_include_tags_property;
 	std::string m_log_level;
 	std::vector<falco::outputs::config> m_outputs;
 	uint32_t m_notifications_rate;
@@ -230,10 +229,6 @@ public:
 
 	uint32_t m_syscall_evt_timeout_max_consecutives;
 
-	uint32_t m_metadata_download_max_mb;
-	uint32_t m_metadata_download_chunk_wait_us;
-	uint32_t m_metadata_download_watch_freq_sec;
-
 private:
 	void init_cmdline_options(std::list<std::string>& cmdline_options);
 

userspace/falco/falco.cpp

@@ -123,9 +123,6 @@ static void usage()
 	   "                               for this option, it will be interpreted as the name of a file containing bearer token.\n"
 	   "                               Note that the format of this command-line option prohibits use of files whose names contain\n"
 	   "                               ':' or '#' characters in the file name.\n"
-	   " --k8s-node <node_name>        The node name will be used as a filter when requesting metadata of pods to the API server.\n"
-	   "                               Usually, it should be set to the current node on which Falco is running.\n"
-	   "                               If empty, no filter is set, which may have a performance penalty on large clusters.\n"
 #endif
 	   " -L                            Show the name and description of all rules and exit.\n"
 	   " -l <rule>                     Show the name and description of the rule with name <rule> and exit.\n"
@@ -374,7 +371,7 @@ uint64_t do_inspect(falco_engine *engine,
 		unique_ptr<falco_engine::rule_result> res = engine->process_sinsp_event(ev);
 		if(res)
 		{
-			outputs->handle_event(res->evt, res->rule, res->source, res->priority_num, res->format, res->tags);
+			outputs->handle_event(res->evt, res->rule, res->source, res->priority_num, res->format);
 		}
 
 		num_evts++;
@@ -472,7 +469,6 @@ int falco_init(int argc, char **argv)
 #ifndef MINIMAL_BUILD
 	string* k8s_api = 0;
 	string* k8s_api_cert = 0;
-	string *k8s_node_name = 0;
 	string* mesos_api = 0;
 #endif
 	string output_format = "";
@@ -521,7 +517,6 @@ int falco_init(int argc, char **argv)
 			{"ignored-events", no_argument, 0, 'i'},
 			{"k8s-api-cert", required_argument, 0, 'K'},
 			{"k8s-api", required_argument, 0, 'k'},
-			{"k8s-node", required_argument, 0},
 			{"list", optional_argument, 0},
 			{"mesos-api", required_argument, 0, 'm'},
 			{"option", required_argument, 0, 'o'},
@@ -698,15 +693,6 @@ int falco_init(int argc, char **argv)
 				{
 				  cri_async = false;
 				}
-#ifndef MINIMAL_BUILD
-				else if(string(long_options[long_index].name) == "k8s-node")
-				{
-					k8s_node_name = new string(optarg);
-					if (k8s_node_name->size() == 0) {
-						throw std::invalid_argument("If --k8s-node is provided, it cannot be an empty string");
-					}
-				}
-#endif
 				else if (string(long_options[long_index].name) == "list")
 				{
 					list_flds = true;
@@ -1122,7 +1108,6 @@ int falco_init(int argc, char **argv)
 
 		outputs->init(config.m_json_output,
 			config.m_json_include_output_property,
-			config.m_json_include_tags_property,
 			config.m_output_timeout,
 			config.m_notifications_rate, config.m_notifications_max_burst,
 			config.m_buffered_outputs,
@@ -1264,7 +1249,7 @@ int falco_init(int argc, char **argv)
 					k8s_api_cert = new string(k8s_cert_env);
 				}
 			}
-			inspector->init_k8s_client(k8s_api, k8s_api_cert, k8s_node_name, verbose);
+			inspector->init_k8s_client(k8s_api, k8s_api_cert, verbose);
 			k8s_api = 0;
 			k8s_api_cert = 0;
 		}
@@ -1280,7 +1265,7 @@ int falco_init(int argc, char **argv)
 					}
 				}
 				k8s_api = new string(k8s_api_env);
-				inspector->init_k8s_client(k8s_api, k8s_api_cert, k8s_node_name, verbose);
+				inspector->init_k8s_client(k8s_api, k8s_api_cert, verbose);
 			}
 			else
 			{
@@ -1309,11 +1294,6 @@ int falco_init(int argc, char **argv)
 		delete mesos_api;
 		mesos_api = 0;
 
-		falco_logger::log(LOG_DEBUG, "Setting metadata download max size to " + to_string(config.m_metadata_download_max_mb) + " MB\n");
-		falco_logger::log(LOG_DEBUG, "Setting metadata download chunk wait time to " + to_string(config.m_metadata_download_chunk_wait_us) + " μs\n");
-		falco_logger::log(LOG_DEBUG, "Setting metadata download watch frequency to " + to_string(config.m_metadata_download_watch_freq_sec) + " seconds\n");
-		inspector->set_metadata_download_params(config.m_metadata_download_max_mb * 1024 * 1024, config.m_metadata_download_chunk_wait_us, config.m_metadata_download_watch_freq_sec);
-		
 		if(trace_filename.empty() && config.m_webserver_enabled && !disable_k8s_audit)
 		{
 			std::string ssl_option = (config.m_webserver_ssl_enabled ? " (SSL)" : "");

userspace/falco/falco_outputs.cpp

@@ -62,7 +62,6 @@ falco_outputs::~falco_outputs()
 
 void falco_outputs::init(bool json_output,
 		  bool json_include_output_property,
-		  bool json_include_tags_property,
 		  uint32_t timeout,
 		  uint32_t rate, uint32_t max_burst, bool buffered,
 		  bool time_format_iso_8601, std::string hostname)
@@ -80,7 +79,6 @@ void falco_outputs::init(bool json_output,
 	// So we can safely update them.
 	falco_formats::s_json_output = json_output;
 	falco_formats::s_json_include_output_property = json_include_output_property;
-	falco_formats::s_json_include_tags_property = json_include_tags_property;
 
 	m_timeout = std::chrono::milliseconds(timeout);
 
@@ -144,7 +142,7 @@ void falco_outputs::add_output(falco::outputs::config oc)
 }
 
 void falco_outputs::handle_event(gen_event *evt, string &rule, string &source,
-				 falco_common::priority_type priority, string &format, std::set<std::string> &tags)
+				 falco_common::priority_type priority, string &format)
 {
 	if(!m_notifications_tb.claim())
 	{
@@ -192,9 +190,8 @@ void falco_outputs::handle_event(gen_event *evt, string &rule, string &source,
 		sformat += " " + format;
 	}
 
-	cmsg.msg = falco_formats::format_event(evt, rule, source, falco_common::priority_names[priority], sformat, tags);
+	cmsg.msg = falco_formats::format_event(evt, rule, source, falco_common::priority_names[priority], sformat);
 	cmsg.fields = falco_formats::resolve_tokens(evt, source, sformat);
-	cmsg.tags.insert(tags.begin(), tags.end());
 
 	cmsg.type = ctrl_msg_type::CTRL_MSG_OUTPUT;
 	m_queue.push(cmsg);

userspace/falco/falco_outputs.h

@@ -40,7 +40,6 @@ public:
 
 	void init(bool json_output,
 		  bool json_include_output_property,
-		  bool json_include_tags_property,
 		  uint32_t timeout,
 		  uint32_t rate, uint32_t max_burst, bool buffered,
 		  bool time_format_iso_8601, std::string hostname);
@@ -49,7 +48,7 @@ public:
 
 	// Format then send the event to all configured outputs (`evt` is an event that has matched some rule).
 	void handle_event(gen_event *evt, std::string &rule, std::string &source,
-			  falco_common::priority_type priority, std::string &format, std::set<std::string> &tags);
+			  falco_common::priority_type priority, std::string &format);
 
 	// Format then send a generic message to all outputs. Not necessarily associated with any event.
 	void handle_msg(uint64_t now,

userspace/falco/outputs.h

@@ -50,7 +50,6 @@ struct message
 	std::string rule;
 	std::string source;
 	map<std::string, std::string> fields;
-	std::set<std::string> tags;
 };
 
 //

userspace/falco/outputs.proto

@@ -50,5 +50,6 @@ message response {
   string output = 5;
   map<string, string> output_fields = 6;
   string hostname = 7;
-  repeated string tags = 8;
+  // TODO(leodido,fntlnz): tags not supported yet, keeping it for reference.
+  // repeated string tags = 8;
 }

userspace/falco/outputs_grpc.cpp

@@ -64,9 +64,5 @@ void falco::outputs::output_grpc::output(const message *msg)
 	auto host = grpc_res.mutable_hostname();
 	*host = m_hostname;
 
-	// tags
-	auto tags = grpc_res.mutable_tags();
-	*tags = {msg->tags.begin(), msg->tags.end()};
-
 	falco::grpc::queue::get().push(grpc_res);
 }

userspace/falco/webserver.cpp

@@ -102,7 +102,7 @@ bool k8s_audit_handler::accept_data(falco_engine *engine,
 			{
 				outputs->handle_event(res->evt, res->rule,
 						      res->source, res->priority_num,
-						      res->format, res->tags);
+						      res->format);
 			}
 			catch(falco_exception &e)
 			{