6 Commits

Author	SHA1	Message	Date
Mark Stemm	bdda640da1	Flag excess drops (#561) ... * Make stats file interval configurable New argument --stats_interval=<msec> controls the interval at which statistics are written to the stats file. The default is 5000 ms (5 sec) which matches the prior hardcoded interval. The stats interval is triggered via signals, so an interval below ~250ms will probably interfere with falco's behavior. * Add ability to emit general purpose messages A new method falco_outputs::handle_msg allows emitting generic messages that have a "rule", message, and output fields, but aren't exactly tied to any event and aren't passed through an event formatter. This allows falco to emit "events" based on internal checks like kernel buffer overflow detection. * Clean up newline handling for logging Log messages from falco_logger::log may or may not have trailing newlines. Handle both by always adding a newline to stderr logs and always removing any newline from syslog logs. * Add method to get sequence from subkey New variant of get_sequence that allows fetching a list of items from a key + subkey, for example: key: subkey: - list - items - here Both use a shared method get_sequence_from_node(). * Monitor syscall event drops + optional actions Start actively monitoring the kernel buffer for syscall event drops, which are visible in scap_stats.n_drops, and add the ability to take actions when events are dropped. The -v (verbose) and -s (stats filename) arguments also print out information on dropped events, but they were only printed/logged without any actions. In falco config you can specify one or more of the following actions to take when falco notes system call drops: - ignore (do nothing) - log a critical message - emit an "internal" falco alert. It looks like any other alert with a time, "rule", message, and output fields but is not related to any rule in falco_rules.yaml/other rules files. - exit falco (the idea being that the restart would be monitored elsewhere). A new module syscall_event_drop_mgr is called for every event and collects scap stats every second. If in the prior second there were drops, perform_actions() handles the actions. To prevent potential flooding in high drop rate environments, actions are goverened by a token bucket with a rate of 1 actions per 30 seconds, with a max burst of 10 seconds. We might tune this later based on experience in busy environments. This might be considered a fix for https://github.com/falcosecurity/falco/issues/545. It doesn't specifically flag falco rules alerts when there are drops, but does make it easier to notice when there are drops. * Add unit test for syscall event drop detection Add unit tests for syscall event drop detection. First, add an optional config option that artifically increments the drop count every second. (This is only used for testing). Then add test cases for each of the following: - No dropped events: should not see any log messages or alerts. - ignore action: should note the drops but not log messages or alert. - log action: should only see log messages for the dropped events. - alert action: should only see alerts for the dropped events. - exit action: should see log message noting the dropped event and exit with rc=1 A new trace file ping_sendto.scap has 10 seconds worth of events to allow the periodic tracking of drops to kick in.	2019-03-27 15:50:39 -07:00
Mark Stemm	1f28f85bdf	K8s audit evts (#450) ... * Add new json/webserver libs, embedded webserver Add two new external libraries: - nlohmann-json is a better json library that has stronger use of c++ features like type deduction, better conversion from stl structures, etc. We'll use it to hold generic json objects instead of jsoncpp. - civetweb is an embeddable webserver that will allow us to accept posted json data. New files webserver.{cpp,h} start an embedded webserver that listens for POSTS on a configurable url and passes the json data to the falco engine. New falco config items are under webserver: - enabled: true|false. Whether to start the embedded webserver or not. - listen_port. Port that webserver listens on - k8s_audit_endpoint: uri on which to accept POSTed k8s audit events. (This commit doesn't compile entirely on its own, but we're grouping these related changes into one commit for clarity). * Don't use relative paths to find lua code You can look directly below PROJECT_SOURCE_DIR. * Reorganize compiler lua code The lua compiler code is generic enough to work on more than just sinsp-based rules, so move the parts of the compiler related to event types and filterchecks out into a standalone lua file sinsp_rule_utils.lua. The checks for event types/filterchecks are now done from rule_loader, and are dependent on a "source" attribute of the rule being "sinsp". We'll be adding additional types of events next that come from sources other than system calls. * Manage separate syscall/k8s audit rulesets Add the ability to manage separate sets of rules (syscall and k8s_audit). Stop using the sinsp_evttype_filter object from the sysdig repo, replacing it with falco_ruleset/falco_sinsp_ruleset from ruleset.{cpp,h}. It has the same methods to add rules, associate them with rulesets, and (for syscall) quickly find the relevant rules for a given syscall/event type. At the falco engine level, there are new parallel interfaces for both types of rules (syscall and k8s_audit) to: - add a rule: add_k8s_audit_filter/add_sinsp_filter - match an event against rules, possibly returning a result: process_sinsp_event/process_k8s_audit_event At the rule loading level, the mechanics of creating filterchecks objects is handled two factories (sinsp_filter_factory and json_event_filter_factory), both of which are held by the engine. * Handle multiple rule types when parsing rules Modify the steps of parsing a rule's filter expression to handle multiple types of rules. Notable changes: - In the rule loader/ast traversal, pass a filter api object down, which is passed back up in the lua parser api calls like nest(), bool_op(), rel_expr(), etc. - The filter api object is either the sinsp factory or k8s audit factory, depending on the rule type. - When the rule is complete, the complete filter is passed to the engine using either add_sinsp_filter()/add_k8s_audit_filter(). * Add multiple output formatting types Add support for multiple output formatters. Notable changes: - The falco engine is passed along to falco_formats to gain access to the engine's factories. - When creating a formatter, the source of the rule is passed along with the format string, which controls which kind of output formatter is created. Also clean up exception handling a bit so all lua callbacks catch all exceptions and convert them into lua errors. * Add support for json, k8s audit filter fields With some corresponding changes in sysdig, you can now create general purpose filter fields and events, which can be tied together with nesting, expressions, and relational operators. The classes here represent an instance of these fields devoted to generic json objects as well as k8s audit events. Notable changes: - json_event: holds a json object, used by all of the below - json_event_filter_check: Has the ability to extract values out of a json_event object and has the ability to define macros that associate a field like "group.field" with a json pointer expression that extracts a single property's value out of the json object. The basic field definition also allows creating an index e.g. group.field[index], where a std::function is responsible for performing the indexing. This class has virtual void methods so it must be overridden. - jevt_filter_check: subclass of json_event_filter_check and defines the following fields: - jevt.time/jevt.rawtime: extracts the time from the underlying json object. - jevt.value[<json pointer>]: general purpose way to extract any json value out of the underlying object. <json pointer> is a json pointer expression - jevt.obj: Return the entire object, stringified. - k8s_audit_filter_check: implements fields that extract values from k8s audit events. Most of the implementation is in the form of macros like ka.user.name, ka.uri, ka.target.name, etc. that just use json pointers to extact the appropriate value from a k8s audit event. More advanced fields like ka.uri.param, ka.req.container.image use indexing to extract individual values out of maps or arrays. - json_event_filter_factory: used by things like the lua parser api, output formatter, etc to create the necessary objects and return them. - json_event_formatter: given a format string, create the necessary fields that will be used to create a resolved string when given a json_event object. * Add ability to list fields Similar to sysdig's -l option, add --list (<source>) to list the fields supported by falco. With no source specified, will print all fields. Source can be "syscall" for inspector fields e.g. what is supported by sysdig, or "k8s_audit" to list fields supported only by the k8s audit support in falco. * Initial set of k8s audit rules Add an initial set of k8s audit rules. They're broken into 3 classes of rules: - Suspicious activity: this includes things like: - A disallowed k8s user performing an operation - A disallowed container being used in a pod. - A pod created with a privileged pod. - A pod created with a sensitive mount. - A pod using host networking - Creating a NodePort Service - A configmap containing private credentials - A request being made by an unauthenticated user. - Attach/exec to a pod. (We eventually want to also do privileged pods, but that will require some state management that we don't currently have). - Creating a new namespace outside of an allowed set - Creating a pod in either of the kube-system/kube-public namespaces - Creating a serviceaccount in either of the kube-system/kube-public namespaces - Modifying any role starting with "system:" - Creating a clusterrolebinding to the cluster-admin role - Creating a role that wildcards verbs or resources - Creating a role with writable permissions/pod exec permissions. - Resource tracking. This includes noting when a deployment, service, - configmap, cluster role, service account, etc are created or destroyed. - Audit tracking: This tracks all audit events. To support these rules, add macros/new indexing functions as needed to support the required fields and ways to index the results. * Add ability to read trace files of k8s audit evts Expand the use of the -e flag to cover both .scap files containing system calls as well as jsonl files containing k8s audit events: If a trace file is specified, first try to read it using the inspector. If that throws an exception, try to read the first line as json. If both fail, return an error. Based on the results of the open, the main loop either calls do_inspect(), looping over system events, or read_k8s_audit_trace_file(), reading each line as json and passing it to the engine and outputs. * Example showing how to enable k8s audit logs. An example of how to enable k8s audit logging for minikube. * Add unit tests for k8s audit support Initial unit test support for k8s audit events. A new multiplex file falco_k8s_audit_tests.yaml defines the tests. Traces (jsonl files) are in trace_files/k8s_audit and new rules files are in test/rules/k8s_audit. Current test cases include: - User outside allowed set - Creating disallowed pod. - Creating a pod explicitly on the allowed list - Creating a pod w/ a privileged container (or second container), or a pod with no privileged container. - Creating a pod w/ a sensitive mount container (or second container), or a pod with no sensitive mount. - Cases for a trace w/o the relevant property + the container being trusted, and hostnetwork tests. - Tests that create a Service w/ and w/o a NodePort type. - Tests for configmaps: tries each disallowed string, ensuring each is detected, and the other has a configmap with no disallowed string, ensuring it is not detected. - The anonymous user creating a namespace. - Tests for all kactivity rules e.g. those that create/delete resources as compared to suspicious activity. - Exec/Attach to Pod - Creating a namespace outside of an allowed set - Creating a pod/serviceaccount in kube-system/kube-public namespaces - Deleting/modifying a system cluster role - Creating a binding to the cluster-admin role - Creating a cluster role binding that wildcards verbs or resources - Creating a cluster role with write/pod exec privileges * Don't manually install gcc 4.8 gcc 4.8 should already be installed by default on the vm we use for travis.	2018-11-09 10:15:39 -08:00
Mark Stemm	ac190ca457	Properly support syscalls in filter conditions (#352) ... * Properly support syscalls in filter conditions Syscalls have their own numbers but they weren't really handled within falco. This meant that there wasn't a way to handle filters with evt.type=xxx clauses where xxx was a value that didn't have a corresponding event entry (like "madvise", for examples), or where a syscall like open could also be done indirectly via syscall(__NR_open, ...). First, add a new top-level global syscalls that maps from a string like "madvise" to all the syscall nums for that id, just as we do for event names/numbers. In the compiler, when traversing the AST for evt.type=XXX or evt.type in (XXX, ...) clauses, also try to match XXX against the global syscalls table, and return any ids in a standalone table. Also throw an error if an XXX doesn't match any event name or syscall name. The syscall numbers are passed as an argument to sinsp_evttype_filter so it can preindex the filters by syscall number. This depends on https://github.com/draios/sysdig/pull/1100 * Add unit test for syscall support This does a madvise, which doesn't have a ppm event type, both directly and indirectly via syscall(__NR_madvise, ...), as well as an open directly + indirectly. The corresponding rules file matches on madvise and open. The test ensures that both opens and both madvises are detected.	2018-04-17 17:14:45 -07:00
Mark Stemm	88327abb41	Unit test for fd.net + in operator fixes (#343) ... Tests fix for https://github.com/draios/falco/issues/339. Depends on https://github.com/draios/sysdig/pull/1091.	2018-04-04 14:23:21 -07:00
Mark Stemm	88faa7c1e7	Add automated tests for tagged rules ... Add automated tests that verify the ability to tag sets of rules, disable them with -T, and run them with -t, works: - New test option disable_tags adds -T <tag> arguments to the falco command line, and run_tags adds -t <tag> arguments to the falco command line. - A new trace file open-multiple-files.scap opens 13 different files, and a new rules file has 13 different rules with all combinations of the tags a, b, c (both forward and backward), a rule with an empty list of tags, a rule with no tags field, and a rule with a completely different tag d. Using the above, add tests for: - Both disabling all combations of a, b, c using disable_tags as well as run all combinations of a, b, c, using run_tags. - Specifying both disabled (-T/-D) and enabled (-t) rules. Not allowed. - Specifying a ruleset while having tagged rules enabled, rules based on a name disabled, and no particular rules enabled or disabled.	2017-02-08 11:08:36 -08:00
Mark Stemm	c140b23678	Add tests for multiple files, disabled rules. ... Add test that cover reading from multiple sets of rule files and disabling rules. Specific changes: - Modify falco to allow multiple -r arguments to read from multiple files. - In the test multiplex file, add a disabled_rules attribute, containing a sequence of rules to disable. Result in -D arguments when running falco. - In the test multiplex file, 'rules_file' can be a sequence. It results in multiple -r arguments when running falco. - In the test multiplex file, 'detect_level' can be a squence of multiple severity levels. All levels will be checked for in the output. - Move all test rules files to a rules subdirectory and all trace files to a traces subdirectory. - Add a small trace file for a simple cat of /dev/null. Used by the new tests. - Add the following new tests: - Reading from multiple files, with the first file being empty. Ensure that the rules from the second file are properly loaded. - Reading from multiple files with the last being empty. Ensures that the empty file doesn't overwrite anything from the first file. - Reading from multiple files with varying severity levels for each rule. Ensures that both files are properly read. - Disabling rules from a rules file, both with full rule names and regexes. Will result in not detecting anything.	2016-10-24 15:56:45 -07:00