github/falco
1
0
Fork 0
mirror of https://github.com/falcosecurity/falco.git synced 2025-10-22 20:29:39 +00:00
Code Issues Projects Releases Wiki Activity

K8s audit evts (#450)

Browse Source 
* Add new json/webserver libs, embedded webserver

Add two new external libraries:

 - nlohmann-json is a better json library that has stronger use of c++
   features like type deduction, better conversion from stl structures,
   etc. We'll use it to hold generic json objects instead of jsoncpp.

 - civetweb is an embeddable webserver that will allow us to accept
   posted json data.

New files webserver.{cpp,h} start an embedded webserver that listens for
POSTS on a configurable url and passes the json data to the falco
engine.

New falco config items are under webserver:
  - enabled: true|false. Whether to start the embedded webserver or not.
  - listen_port. Port that webserver listens on
  - k8s_audit_endpoint: uri on which to accept POSTed k8s audit events.

(This commit doesn't compile entirely on its own, but we're grouping
these related changes into one commit for clarity).

* Don't use relative paths to find lua code

You can look directly below PROJECT_SOURCE_DIR.

* Reorganize compiler lua code

The lua compiler code is generic enough to work on more than just
sinsp-based rules, so move the parts of the compiler related to event
types and filterchecks out into a standalone lua file
sinsp_rule_utils.lua.

The checks for event types/filterchecks are now done from rule_loader,
and are dependent on a "source" attribute of the rule being
"sinsp". We'll be adding additional types of events next that come from
sources other than system calls.

* Manage separate syscall/k8s audit rulesets

Add the ability to manage separate sets of rules (syscall and
k8s_audit). Stop using the sinsp_evttype_filter object from the sysdig
repo, replacing it with falco_ruleset/falco_sinsp_ruleset from
ruleset.{cpp,h}. It has the same methods to add rules, associate them
with rulesets, and (for syscall) quickly find the relevant rules for a
given syscall/event type.

At the falco engine level, there are new parallel interfaces for both
types of rules (syscall and k8s_audit) to:
  - add a rule: add_k8s_audit_filter/add_sinsp_filter
  - match an event against rules, possibly returning a result:
    process_sinsp_event/process_k8s_audit_event

At the rule loading level, the mechanics of creating filterchecks
objects is handled two factories (sinsp_filter_factory and
json_event_filter_factory), both of which are held by the engine.

* Handle multiple rule types when parsing rules

Modify the steps of parsing a rule's filter expression to handle
multiple types of rules. Notable changes:

 - In the rule loader/ast traversal, pass a filter api object down,
   which is passed back up in the lua parser api calls like nest(),
   bool_op(), rel_expr(), etc.
 - The filter api object is either the sinsp factory or k8s audit
   factory, depending on the rule type.
 - When the rule is complete, the complete filter is passed to the
   engine using either add_sinsp_filter()/add_k8s_audit_filter().

* Add multiple output formatting types

Add support for multiple output formatters. Notable changes:

 - The falco engine is passed along to falco_formats to gain access to
   the engine's factories.
 - When creating a formatter, the source of the rule is passed along
   with the format string, which controls which kind of output formatter
   is created.

Also clean up exception handling a bit so all lua callbacks catch all
exceptions and convert them into lua errors.

* Add support for json, k8s audit filter fields

With some corresponding changes in sysdig, you can now create general
purpose filter fields and events, which can be tied together with
nesting, expressions, and relational operators. The classes here
represent an instance of these fields devoted to generic json objects as
well as k8s audit events. Notable changes:

 - json_event: holds a json object, used by all of the below

 - json_event_filter_check: Has the ability to extract values out of a
   json_event object and has the ability to define macros that associate
   a field like "group.field" with a json pointer expression that
   extracts a single property's value out of the json object. The basic
   field definition also allows creating an index
   e.g. group.field[index], where a std::function is responsible for
   performing the indexing. This class has virtual void methods so it
   must be overridden.

 - jevt_filter_check: subclass of json_event_filter_check and defines
   the following fields:
     - jevt.time/jevt.rawtime: extracts the time from the underlying json object.
     - jevt.value[<json pointer>]: general purpose way to extract any
       json value out of the underlying object. <json pointer> is a json
       pointer expression
     - jevt.obj: Return the entire object, stringified.

 - k8s_audit_filter_check: implements fields that extract values from
   k8s audit events. Most of the implementation is in the form of macros
   like ka.user.name, ka.uri, ka.target.name, etc. that just use json
   pointers to extact the appropriate value from a k8s audit event. More
   advanced fields like ka.uri.param, ka.req.container.image use
   indexing to extract individual values out of maps or arrays.

 - json_event_filter_factory: used by things like the lua parser api,
   output formatter, etc to create the necessary objects and return
   them.

  - json_event_formatter: given a format string, create the necessary
    fields that will be used to create a resolved string when given a
    json_event object.

* Add ability to list fields

Similar to sysdig's -l option, add --list (<source>) to list the fields
supported by falco. With no source specified, will print all
fields. Source can be "syscall" for inspector fields e.g. what is
supported by sysdig, or "k8s_audit" to list fields supported only by the
k8s audit support in falco.

* Initial set of k8s audit rules

Add an initial set of k8s audit rules. They're broken into 3 classes of
rules:

 - Suspicious activity: this includes things like:
    - A disallowed k8s user performing an operation
    - A disallowed container being used in a pod.
    - A pod created with a privileged pod.
    - A pod created with a sensitive mount.
    - A pod using host networking
    - Creating a NodePort Service
    - A configmap containing private credentials
    - A request being made by an unauthenticated user.
    - Attach/exec to a pod. (We eventually want to also do privileged
      pods, but that will require some state management that we don't
      currently have).
    - Creating a new namespace outside of an allowed set
    - Creating a pod in either of the kube-system/kube-public namespaces
    - Creating a serviceaccount in either of the kube-system/kube-public
      namespaces
    - Modifying any role starting with "system:"
    - Creating a clusterrolebinding to the cluster-admin role
    - Creating a role that wildcards verbs or resources
    - Creating a role with writable permissions/pod exec permissions.
 - Resource tracking. This includes noting when a deployment, service,
    - configmap, cluster role, service account, etc are created or destroyed.
 - Audit tracking: This tracks all audit events.

To support these rules, add macros/new indexing functions as needed to
support the required fields and ways to index the results.

* Add ability to read trace files of k8s audit evts

Expand the use of the -e flag to cover both .scap files containing
system calls as well as jsonl files containing k8s audit events:

If a trace file is specified, first try to read it using the
inspector. If that throws an exception, try to read the first line as
json. If both fail, return an error.

Based on the results of the open, the main loop either calls
do_inspect(), looping over system events, or
read_k8s_audit_trace_file(), reading each line as json and passing it to
the engine and outputs.

* Example showing how to enable k8s audit logs.

An example of how to enable k8s audit logging for minikube.

* Add unit tests for k8s audit support

Initial unit test support for k8s audit events. A new multiplex file
falco_k8s_audit_tests.yaml defines the tests. Traces (jsonl files) are
in trace_files/k8s_audit and new rules files are in
test/rules/k8s_audit.

Current test cases include:

- User outside allowed set
- Creating disallowed pod.
- Creating a pod explicitly on the allowed list
- Creating a pod w/ a privileged container (or second container), or a
  pod with no privileged container.
- Creating a pod w/ a sensitive mount container (or second container), or a
  pod with no sensitive mount.
- Cases for a trace w/o the relevant property + the container being
  trusted, and hostnetwork tests.
- Tests that create a Service w/ and w/o a NodePort type.
- Tests for configmaps: tries each disallowed string, ensuring each is
  detected, and the other has a configmap with no disallowed string,
  ensuring it is not detected.
- The anonymous user creating a namespace.
- Tests for all kactivity rules e.g. those that create/delete
  resources as compared to suspicious activity.
- Exec/Attach to Pod
- Creating a namespace outside of an allowed set
- Creating a pod/serviceaccount in kube-system/kube-public namespaces
- Deleting/modifying a system cluster role
- Creating a binding to the cluster-admin role
- Creating a cluster role binding that wildcards verbs or resources
- Creating a cluster role with write/pod exec privileges

* Don't manually install gcc 4.8

gcc 4.8 should already be installed by default on the vm we use for
travis.
This commit is contained in:
Mark Stemm
2018-11-09 10:15:39 -08:00
committed by GitHub
parent ff4f7ca13b
commit 1f28f85bdf
85 changed files with 4105 additions and 427 deletions
Download Patch File Download Diff File Expand all files Collapse all files

310
userspace/engine/json_evt.h Normal file
View File

@@ -0,0 +1,310 @@
/*
Copyright (C) 2018 Draios inc.
This file is part of falco.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
#pragma once
#include <memory>
#include <list>
#include <map>
#include <string>
#include <vector>
#include <set>
#include <utility>
#include <nlohmann/json.hpp>
#include "gen_filter.h"
class json_event : public gen_event
{
public:
json_event();
virtual ~json_event();
void set_jevt(nlohmann::json &evt, uint64_t ts);
const nlohmann::json &jevt();
uint64_t get_ts();
protected:
nlohmann::json m_jevt;
uint64_t m_event_ts;
};
class json_event_filter_check : public gen_event_filter_check
{
public:
// A struct describing a single filtercheck field ("ka.user")
struct field_info {
std::string name;
std::string desc;
};
// A struct describing a group of filtercheck fields ("ka")
struct check_info {
std::string name;
std::string desc;
std::list<field_info> fields;
};
json_event_filter_check();
virtual ~json_event_filter_check();
virtual int32_t parse_field_name(const char* str, bool alloc_state, bool needed_for_filtering);
void add_filter_value(const char* str, uint32_t len, uint32_t i = 0 );
bool compare(gen_event *evt);
virtual uint8_t* extract(gen_event *evt, uint32_t* len, bool sanitize_strings = true);
// Simpler version that returns a string
std::string extract(json_event *evt);
const std::string &field();
const std::string &idx();
// The combined size of the field, index, and surrounding
// brackets (e.g. ka.image[foo])
size_t parsed_size();
check_info &get_fields();
//
// Allocate a new check of the same type. Must be overridden.
//
virtual json_event_filter_check *allocate_new() = 0;
protected:
static std::string def_format(const nlohmann::json &j, std::string &field, std::string &idx);
static std::string json_as_string(const nlohmann::json &j);
// Subclasses can define field names that act as aliases for
// specific json pointer expressions e.g. ka.user ==
// jevt.value[/user/username]. This struct represents one of
// those aliases.
typedef std::function<std::string (const nlohmann::json &, std::string &field, std::string &idx)> format_t;
struct alias {
// Whether this alias requires an index, allows an
// index, or should not have an index.
enum index_mode {
IDX_REQUIRED,
IDX_ALLOWED,
IDX_NONE
};
enum index_type {
IDX_KEY,
IDX_NUMERIC
};
// The variants allow for brace-initialization either
// with just the pointer or with both the pointer and
// a format function.
alias();
alias(nlohmann::json::json_pointer ptr);
alias(nlohmann::json::json_pointer ptr, format_t format);
alias(nlohmann::json::json_pointer ptr, format_t format, index_mode mode);
alias(nlohmann::json::json_pointer ptr, format_t format, index_mode mode, index_type itype);
virtual ~alias();
// A json pointer used to extract a referenced value
// from a json object.
nlohmann::json::json_pointer m_jptr;
// A function that given the referenced value selected
// above, formats and returns the appropriate string. This
// function might do further selection (e.g. array
// indexing, searches, etc.) or string reformatting to
// trim unnecessary parts of the value.
format_t m_format;
index_mode m_idx_mode;
index_type m_idx_type;
};
// This map defines the aliases defined by this filter check
// class.
//
// The version of parse_field_name in this base class will
// check a field specification against all the aliases.
std::map<std::string, struct alias> m_aliases;
check_info m_info;
// The actual field name parsed in parse_field_name.
std::string m_field;
// The field name itself might include an index component
// e.g. ka.value[idx]. This holds the index.
std::string m_idx;
// The actual json pointer value to use to extract from events.
nlohmann::json::json_pointer m_jptr;
// Temporary storage to hold extracted value
std::string m_tstr;
// Reformatting function
format_t m_format;
private:
std::vector<std::string> m_values;
};
class jevt_filter_check : public json_event_filter_check
{
public:
jevt_filter_check();
virtual ~jevt_filter_check();
int32_t parse_field_name(const char* str, bool alloc_state, bool needed_for_filtering);
virtual uint8_t* extract(gen_event *evt, uint32_t* len, bool sanitize_strings = true);
json_event_filter_check *allocate_new();
private:
static std::string s_jevt_time_field;
static std::string s_jevt_rawtime_field;
static std::string s_jevt_obj_field;
static std::string s_jevt_value_field;
};
class k8s_audit_filter_check : public json_event_filter_check
{
public:
k8s_audit_filter_check();
virtual ~k8s_audit_filter_check();
json_event_filter_check *allocate_new();
// Index to the appropriate container and/or remove any repo,
// port, and tag from the provided image name.
static std::string index_image(const nlohmann::json &j, std::string &field, std::string &idx);
// Extract the value of the provided query parameter
static std::string index_query_param(const nlohmann::json &j, std::string &field, std::string &idx);
// Return true if an object in the provided array has a name property with idx as value
static std::string index_has_name(const nlohmann::json &j, std::string &field, std::string &idx);
// Return whether the ith container (or any container, if an
// index is not specified) is run privileged.
static std::string index_privileged(const nlohmann::json &j, std::string &field, std::string &idx);
// Return whether or not a hostpath mount exists matching the provided index.
static std::string check_hostpath_vols(const nlohmann::json &j, std::string &field, std::string &idx);
// Index to the ith value from the provided array. If no index is provided, return the entire array as a string.
static std::string index_generic(const nlohmann::json &j, std::string &field, std::string &idx);
// Index to the ith value from the provided array, and select
// the property which is the last component of the provided
// field.
static std::string index_select(const nlohmann::json &j, std::string &field, std::string &idx);
};
class json_event_filter : public gen_event_filter
{
public:
json_event_filter();
virtual ~json_event_filter();
std::string m_rule;
uint32_t m_rule_idx;
std::set<std::string> m_tags;
};
class json_event_filter_factory : public gen_event_filter_factory
{
public:
json_event_filter_factory();
virtual ~json_event_filter_factory();
// Create a new filter
gen_event_filter *new_filter();
// Create a new filter_check
gen_event_filter_check *new_filtercheck(const char *fldname);
// All defined field names
std::list<json_event_filter_check::check_info> &get_fields();
private:
std::list<std::shared_ptr<json_event_filter_check>> m_defined_checks;
std::list<json_event_filter_check::check_info> m_info;
};
// Unlike the other classes, this does not inherit from a shared class
// that's used both by json events and sinsp events. It might be
// worthwhile, but it would require a lot of additional work to pull
// up functionality into the generic filtercheck class.
class json_event_formatter
{
public:
json_event_formatter(json_event_filter_factory &factory, std::string &format);
virtual ~json_event_formatter();
std::string tostring(json_event *ev);
std::string tojson(json_event *ev);
private:
void parse_format();
void resolve_tokens(json_event *ev, std::list<std::pair<std::string,std::string>> &resolved);
// A format token is either a combination of a filtercheck
// name (ka.value) and filtercheck object as key, or an empty
// key and a NULL filtercheck object, combined with a value (
//
// For example, given a format string:
// "The value is %ka.value today"
// The tokens would be:
// [("The value is ", NULL), ("ka.value", <an object>), " today", NULL)]
struct fmt_token
{
std::string text;
std::shared_ptr<json_event_filter_check> check;
};
// The original format string
std::string m_format;
// The chunks that make up the format string, in order, broken
// up between text chunks and filterchecks.
std::list<fmt_token> m_tokens;
// All the filterchecks required to resolve tokens in the format string
json_event_filter_factory &m_json_factory;
};