Add a bunch of additional test cases for validating rules files. Each
has a specific kind of parse failure and checks for the appropriate
error info on stdout.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
New test options stdout_is/stderr_is do a direct comparison between
stdout/stderr and the provided value.
Test option validate_rules_file maps to -V arguments, which validate
rules and exits.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
The main changes are to use falco_rules.yaml when using
k8s_audit_rules.yaml, as it now depends on it, and to modify one of the
tests to add granular exceptions instead of a single trusted list.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
* Update engine fields checksum for fd.dev.*
New fields fd.dev.*, so updating the fields checksum.
* Print a message why the trace file can't be read.
At debug level only, but better than nothing.
* Adjust tests to match new container_started macro
Now that the container_started macro works either on the container event
or the first process being spawned in a container, we need to adjust the
counts for some rules to handle both cases.
* Supporting files to build/test via jenkins
Changes to build/test via jenkins, which also means running all tests in
a container instead of directly on the host:
- Jenkinsfile controls the stages, build.sh does the build and
run-tests.sh does the regression tests.
- Create a new container falcosecurity/falco-tester that includes the
dependencies required to run the regression tests. This is a different
image than falco-builder because it doesn't need to be centos 6 based,
doesn't install any compiler/etc, and installs the test running
framework we use (avocado). We now use a newer version of avocado,
which resulted in some small changes to how it is run and how yaml
options are parsed.
- Modify run_regression_tests.sh to download trace files to the build
directory and only if not present. Also honor BUILD_TYPE/BUILD_DIR,
which is provided via the docker run cmd.
- The package tests are now moved to a separate falco_tests_package.yaml
file. They will use rpm installs by default instead of debian
packages. Also add the ability to install rpms in addition to debian
packages.
- Automate the process of creating the docker local package by: 1)
Adding CMake rules to copy the Dockerfile, entrypoint to the build
directory and 2) Copy test trace files and rules into the build
directory. This allows running the docker build command from
build/docker/local instead of the source directory.
- Modify the way the container test is run a bit to use the trace
files/rules copied into the container directly instead of host-mounted
trace files.
* Use container builder + tester for travis
We'll probably be using jenkins soon, but this will allow switching back
to travis later if we want.
* Use download.draios.com for binutils packages
That way we won't be dependent on snapshot.debian.org.
* Add option to display times in ISO 8601 UTC
ISO 8601 time is useful when, say, running falco in a container, which
may have a different /etc/localtime than the host system.
A new config option time_format_iso_8601 controls whether log message
and event times are displayed in ISO 8601 in UTC or in local time. The
default is false (display times in local time).
This option is passed to logger init as well as outputs. For outputs it
eventually changes the time format field from %evt.time/%jevt.time to
%evt.time.iso8601/%jevt.time.iso8601.
Adding this field changes the falco engine version so increment it.
This depends on https://github.com/draios/sysdig/pull/1317.
* Unit test for ISO 8601 output
A unit test for ISO 8601 output ensures that both the log and event time
is in ISO 8601 format.
* Use ISO 8601 output by default in containers
Now that we have an option that controls iso 8601 output, use it by
default in containers. We do this by changing the value of
time_format_iso_8601 in falco.yaml in the container.
* Handle errors in strftime/asctime/gmtime
A placeholder "N/A" is used in log messages instead.
* Skip incomplete container info for container start
In the container_started macro, ensure that the container metadata is
complete after either the container event (very unlikely) or after the
exec of the first process into the container (very likely now that
container metadata fetches are async).
When using these rules with older falco versions, this macro will still
work as the synchronous container metadata fetch will result in a
repository that isn't "incomplete".
* Update test traces to have full container info
Some test trace files used for regression tests didn't have full
container info, and once we started looking for those fields, the tests
stopped working.
So update the traces, and event counts to match.
* Make stats file interval configurable
New argument --stats_interval=<msec> controls the interval at which
statistics are written to the stats file. The default is 5000 ms (5 sec)
which matches the prior hardcoded interval.
The stats interval is triggered via signals, so an interval below ~250ms
will probably interfere with falco's behavior.
* Add ability to emit general purpose messages
A new method falco_outputs::handle_msg allows emitting generic messages
that have a "rule", message, and output fields, but aren't exactly tied
to any event and aren't passed through an event formatter.
This allows falco to emit "events" based on internal checks like kernel
buffer overflow detection.
* Clean up newline handling for logging
Log messages from falco_logger::log may or may not have trailing
newlines. Handle both by always adding a newline to stderr logs and
always removing any newline from syslog logs.
* Add method to get sequence from subkey
New variant of get_sequence that allows fetching a list of items from a
key + subkey, for example:
key:
subkey:
- list
- items
- here
Both use a shared method get_sequence_from_node().
* Monitor syscall event drops + optional actions
Start actively monitoring the kernel buffer for syscall event drops,
which are visible in scap_stats.n_drops, and add the ability
to take actions when events are dropped. The -v (verbose) and
-s (stats filename) arguments also print out information on dropped
events, but they were only printed/logged without any actions.
In falco config you can specify one or more of the following actions to
take when falco notes system call drops:
- ignore (do nothing)
- log a critical message
- emit an "internal" falco alert. It looks like any other alert with a
time, "rule", message, and output fields but is not related to any
rule in falco_rules.yaml/other rules files.
- exit falco (the idea being that the restart would be monitored
elsewhere).
A new module syscall_event_drop_mgr is called for every event and
collects scap stats every second. If in the prior second there were
drops, perform_actions() handles the actions.
To prevent potential flooding in high drop rate environments, actions
are goverened by a token bucket with a rate of 1 actions per 30 seconds,
with a max burst of 10 seconds. We might tune this later based on
experience in busy environments.
This might be considered a fix for
https://github.com/falcosecurity/falco/issues/545. It doesn't
specifically flag falco rules alerts when there are drops, but does
make it easier to notice when there are drops.
* Add unit test for syscall event drop detection
Add unit tests for syscall event drop detection. First, add an optional
config option that artifically increments the drop count every
second. (This is only used for testing).
Then add test cases for each of the following:
- No dropped events: should not see any log messages or alerts.
- ignore action: should note the drops but not log messages or alert.
- log action: should only see log messages for the dropped events.
- alert action: should only see alerts for the dropped events.
- exit action: should see log message noting the dropped event and exit
with rc=1
A new trace file ping_sendto.scap has 10 seconds worth of events to
allow the periodic tracking of drops to kick in.
* Add ability to print field names only
Add ability to print field names only instead of all information about
fields (description, etc) using -N cmdline option.
This will be used to add some versioning support steps that check for a
changed set of fields.
* Add an engine version that changes w/ filter flds
Add a method falco_engine::engine_version() that returns the current
engine version (e.g. set of supported fields, rules objects, operators,
etc.). It's defined in falco_engine_version.h, starts at 2 and should be
updated whenever a breaking change is made.
The most common reason for an engine change will be an update to the set
of filter fields. To make this easy to diagnose, add a build time check
that compares the sha256 output of "falco --list -N" against a value
that's embedded in falco_engine_version.h. A mismatch fails the build.
* Check engine version when loading rules
A rules file can now have a field "required_engine_version N". If
present, the number is compared to the falco engine version. If the
falco engine version is less, an error is thrown.
* Unit tests for engine versioning
Add a required version: 2 to one trace file to check the positive case
and add a new test that verifies that a too-new rules file won't be loaded.
* Rename falco test docker image
Rename sysdig/falco to falcosecurity/falco in unit tests.
* Don't pin falco_rules.yaml to an engine version
Currently, falco_rules.yaml is compatible with versions <= 0.13.1 other
than the required_engine_version object itself, so keep that line
commented out so users can use this rules file with older falco
versions.
We'll uncomment it with the first incompatible falco engine change.
* Add new json/webserver libs, embedded webserver
Add two new external libraries:
- nlohmann-json is a better json library that has stronger use of c++
features like type deduction, better conversion from stl structures,
etc. We'll use it to hold generic json objects instead of jsoncpp.
- civetweb is an embeddable webserver that will allow us to accept
posted json data.
New files webserver.{cpp,h} start an embedded webserver that listens for
POSTS on a configurable url and passes the json data to the falco
engine.
New falco config items are under webserver:
- enabled: true|false. Whether to start the embedded webserver or not.
- listen_port. Port that webserver listens on
- k8s_audit_endpoint: uri on which to accept POSTed k8s audit events.
(This commit doesn't compile entirely on its own, but we're grouping
these related changes into one commit for clarity).
* Don't use relative paths to find lua code
You can look directly below PROJECT_SOURCE_DIR.
* Reorganize compiler lua code
The lua compiler code is generic enough to work on more than just
sinsp-based rules, so move the parts of the compiler related to event
types and filterchecks out into a standalone lua file
sinsp_rule_utils.lua.
The checks for event types/filterchecks are now done from rule_loader,
and are dependent on a "source" attribute of the rule being
"sinsp". We'll be adding additional types of events next that come from
sources other than system calls.
* Manage separate syscall/k8s audit rulesets
Add the ability to manage separate sets of rules (syscall and
k8s_audit). Stop using the sinsp_evttype_filter object from the sysdig
repo, replacing it with falco_ruleset/falco_sinsp_ruleset from
ruleset.{cpp,h}. It has the same methods to add rules, associate them
with rulesets, and (for syscall) quickly find the relevant rules for a
given syscall/event type.
At the falco engine level, there are new parallel interfaces for both
types of rules (syscall and k8s_audit) to:
- add a rule: add_k8s_audit_filter/add_sinsp_filter
- match an event against rules, possibly returning a result:
process_sinsp_event/process_k8s_audit_event
At the rule loading level, the mechanics of creating filterchecks
objects is handled two factories (sinsp_filter_factory and
json_event_filter_factory), both of which are held by the engine.
* Handle multiple rule types when parsing rules
Modify the steps of parsing a rule's filter expression to handle
multiple types of rules. Notable changes:
- In the rule loader/ast traversal, pass a filter api object down,
which is passed back up in the lua parser api calls like nest(),
bool_op(), rel_expr(), etc.
- The filter api object is either the sinsp factory or k8s audit
factory, depending on the rule type.
- When the rule is complete, the complete filter is passed to the
engine using either add_sinsp_filter()/add_k8s_audit_filter().
* Add multiple output formatting types
Add support for multiple output formatters. Notable changes:
- The falco engine is passed along to falco_formats to gain access to
the engine's factories.
- When creating a formatter, the source of the rule is passed along
with the format string, which controls which kind of output formatter
is created.
Also clean up exception handling a bit so all lua callbacks catch all
exceptions and convert them into lua errors.
* Add support for json, k8s audit filter fields
With some corresponding changes in sysdig, you can now create general
purpose filter fields and events, which can be tied together with
nesting, expressions, and relational operators. The classes here
represent an instance of these fields devoted to generic json objects as
well as k8s audit events. Notable changes:
- json_event: holds a json object, used by all of the below
- json_event_filter_check: Has the ability to extract values out of a
json_event object and has the ability to define macros that associate
a field like "group.field" with a json pointer expression that
extracts a single property's value out of the json object. The basic
field definition also allows creating an index
e.g. group.field[index], where a std::function is responsible for
performing the indexing. This class has virtual void methods so it
must be overridden.
- jevt_filter_check: subclass of json_event_filter_check and defines
the following fields:
- jevt.time/jevt.rawtime: extracts the time from the underlying json object.
- jevt.value[<json pointer>]: general purpose way to extract any
json value out of the underlying object. <json pointer> is a json
pointer expression
- jevt.obj: Return the entire object, stringified.
- k8s_audit_filter_check: implements fields that extract values from
k8s audit events. Most of the implementation is in the form of macros
like ka.user.name, ka.uri, ka.target.name, etc. that just use json
pointers to extact the appropriate value from a k8s audit event. More
advanced fields like ka.uri.param, ka.req.container.image use
indexing to extract individual values out of maps or arrays.
- json_event_filter_factory: used by things like the lua parser api,
output formatter, etc to create the necessary objects and return
them.
- json_event_formatter: given a format string, create the necessary
fields that will be used to create a resolved string when given a
json_event object.
* Add ability to list fields
Similar to sysdig's -l option, add --list (<source>) to list the fields
supported by falco. With no source specified, will print all
fields. Source can be "syscall" for inspector fields e.g. what is
supported by sysdig, or "k8s_audit" to list fields supported only by the
k8s audit support in falco.
* Initial set of k8s audit rules
Add an initial set of k8s audit rules. They're broken into 3 classes of
rules:
- Suspicious activity: this includes things like:
- A disallowed k8s user performing an operation
- A disallowed container being used in a pod.
- A pod created with a privileged pod.
- A pod created with a sensitive mount.
- A pod using host networking
- Creating a NodePort Service
- A configmap containing private credentials
- A request being made by an unauthenticated user.
- Attach/exec to a pod. (We eventually want to also do privileged
pods, but that will require some state management that we don't
currently have).
- Creating a new namespace outside of an allowed set
- Creating a pod in either of the kube-system/kube-public namespaces
- Creating a serviceaccount in either of the kube-system/kube-public
namespaces
- Modifying any role starting with "system:"
- Creating a clusterrolebinding to the cluster-admin role
- Creating a role that wildcards verbs or resources
- Creating a role with writable permissions/pod exec permissions.
- Resource tracking. This includes noting when a deployment, service,
- configmap, cluster role, service account, etc are created or destroyed.
- Audit tracking: This tracks all audit events.
To support these rules, add macros/new indexing functions as needed to
support the required fields and ways to index the results.
* Add ability to read trace files of k8s audit evts
Expand the use of the -e flag to cover both .scap files containing
system calls as well as jsonl files containing k8s audit events:
If a trace file is specified, first try to read it using the
inspector. If that throws an exception, try to read the first line as
json. If both fail, return an error.
Based on the results of the open, the main loop either calls
do_inspect(), looping over system events, or
read_k8s_audit_trace_file(), reading each line as json and passing it to
the engine and outputs.
* Example showing how to enable k8s audit logs.
An example of how to enable k8s audit logging for minikube.
* Add unit tests for k8s audit support
Initial unit test support for k8s audit events. A new multiplex file
falco_k8s_audit_tests.yaml defines the tests. Traces (jsonl files) are
in trace_files/k8s_audit and new rules files are in
test/rules/k8s_audit.
Current test cases include:
- User outside allowed set
- Creating disallowed pod.
- Creating a pod explicitly on the allowed list
- Creating a pod w/ a privileged container (or second container), or a
pod with no privileged container.
- Creating a pod w/ a sensitive mount container (or second container), or a
pod with no sensitive mount.
- Cases for a trace w/o the relevant property + the container being
trusted, and hostnetwork tests.
- Tests that create a Service w/ and w/o a NodePort type.
- Tests for configmaps: tries each disallowed string, ensuring each is
detected, and the other has a configmap with no disallowed string,
ensuring it is not detected.
- The anonymous user creating a namespace.
- Tests for all kactivity rules e.g. those that create/delete
resources as compared to suspicious activity.
- Exec/Attach to Pod
- Creating a namespace outside of an allowed set
- Creating a pod/serviceaccount in kube-system/kube-public namespaces
- Deleting/modifying a system cluster role
- Creating a binding to the cluster-admin role
- Creating a cluster role binding that wildcards verbs or resources
- Creating a cluster role with write/pod exec privileges
* Don't manually install gcc 4.8
gcc 4.8 should already be installed by default on the vm we use for
travis.
* Also add endswith to lua parser
Add endswith as a symbol so it can be parsed in filter expressions.
* Unit test for endswith support
Add a test case for endswith support, based on the filename ending with null.
* Use correct copyright years.
Also include the start year.
* Improve copyright notices.
Use the proper start year instead of just 2018.
Add the right owner Draios dba Sysdig.
Add copyright notices to some files that were missing them.
Replace references to GNU Public License to Apache license in:
- COPYING file
- README
- all source code below falco
- rules files
- rules and code below test directory
- code below falco directory
- entrypoint for docker containers (but not the Dockerfiles)
I didn't generally add copyright notices to all the examples files, as
they aren't core falco. If they did refer to the gpl I changed them to
apache.
* Add ability to skip rules for unknown filters
Add the ability to skip a rule if its condition refers to a filtercheck
that doesn't exist. This allows defining a rules file that contains new
conditions that can still has limited backward compatibility with older
falco versions.
When compiling a filter, return a list of filtercheck names that are
present in the ast (which also includes filterchecks from any
macros). This set of filtercheck names is matched against the set of
filterchecks known to sinsp, expressed as lua patterns, and in the
global table defined_filters. If no match is found, the rule loader
throws an error.
The pattern changes slightly depending on whether the filter has
arguments or not. Two filters (proc.apid/proc.aname) can work with or
without arguments, so both styles of patterns are used.
If the rule has an attribute "skip-if-unknown-filter", the rule will be
skipped instead.
* Unit tests for skipping unknown filter
New unit test for skipping unknown filter. Test cases:
- A rule that refers to an unknown filter results in an error.
- A rule that refers to an unknown filter, but has
"skip-if-unknown-filter: true", can be read, but doesn't match any events.
- A rule that refers to an unknown filter, but has
"skip-if-unknown-filter: false", returns an error.
Also test the case of a filtercheck like evt.arg.xxx working properly
with the embedded patterns as well as proc.aname/apid which work both ways.
* Use better way to skip falco events
Use the new method falco_consider() to determine which events to
skip. This centralizes the logic in a single function. All events will
still be considered if falco was run with -A.
This depends on https://github.com/draios/sysdig/pull/1105.
* Add ability to specify -A flag in tests
test attribute all_events corresponds to the -A flag. Add for some tests
that would normally refer to skipped events.
* Only check whole rule names when matching counts
Tweak the regex so a rule my_great_rule doesn't pick up event counts for
a rule "great_rule: nnn".
* Add ability to skip evttype warnings for rules
A new attribute warn_evttypes, if present, suppresses printing warnings
related to a rule not matching any event type. Useful if you have a rule
where not including an event type is intentional.
* Add test for preserving rule order
Test the fix for https://github.com/draios/falco/issues/354. A rules
file has a event-specific rule first and a catchall rule second. Without
the changes in https://github.com/draios/sysdig/pull/1103, the first
rule does not match the event.
* Properly support syscalls in filter conditions
Syscalls have their own numbers but they weren't really handled within
falco. This meant that there wasn't a way to handle filters with
evt.type=xxx clauses where xxx was a value that didn't have a
corresponding event entry (like "madvise", for examples), or where a
syscall like open could also be done indirectly via syscall(__NR_open,
...).
First, add a new top-level global syscalls that maps from a string like
"madvise" to all the syscall nums for that id, just as we do for event
names/numbers.
In the compiler, when traversing the AST for evt.type=XXX or evt.type in
(XXX, ...) clauses, also try to match XXX against the global syscalls
table, and return any ids in a standalone table.
Also throw an error if an XXX doesn't match any event name or syscall name.
The syscall numbers are passed as an argument to sinsp_evttype_filter so
it can preindex the filters by syscall number.
This depends on https://github.com/draios/sysdig/pull/1100
* Add unit test for syscall support
This does a madvise, which doesn't have a ppm event type, both directly
and indirectly via syscall(__NR_madvise, ...), as well as an open
directly + indirectly. The corresponding rules file matches on madvise
and open.
The test ensures that both opens and both madvises are detected.
* Add ability to read rules files from directories
When the argument to -r <path> or an entry in falco.yaml's rules_file
list is a directory, read all files in the directory and add them to the
rules file list. The files in the directory are sorted alphabetically
before being added to the list.
The installed falco adds directories /etc/falco/rules.available and
/etc/falco/rules.d and moves /etc/falco/application_rules.yaml to
/etc/falco/rules.available. /etc/falco/rules.d is empty, but the idea is
that admins can symlink to /etc/falco/rules.available for applications
they want to enable.
This will make it easier to add application-specific rulesets that
admins can opt-in to.
* Unit test for reading rules from directory
Copy the rules/trace file from the test multiple_rules to a new test
rules_directory. The rules files are in rules/rules_dir/{000,001}*.yaml,
and the test uses a rules_file argument of rules_dir. Ensure that the
same events are detected.
* Allow appending to skipped rules
If a rule has an append attribute but the original rule was skipped (due
to having lower priority than the configured priority), silently skip
the appending rule instead of returning an error.
* Unit test for appending to skipped rules
Unit test verifies fix for appending to skipped rules. One rules file
defines a rule with priority WARNING, a second rules file appends to
that rules file, and the configured priority is ERROR.
Ensures that falco rules without errors.
* Add option to exclude output property in json fmt
New falco.yaml option json_include_output_property controls where the
formatted string "output" is included in the json object when json
output is enabled. By default the string is included.
* Add tests for new json output option
New test sets json_include_output_property to false and then verifies
that the json output does *not* contain the surrounding text "Warning an
open...".
* Let OMS agent for linux write config
Programs are omiagent/omsagent/PerformInventor/in_heartbeat_r* and files
are below /etc/opt/omi and /etc/opt/microsoft/omsagent.
* Handle really long classpath lines for cassandra
Some cassandra cmdlines are so long the classpath truncates the cmdline
before the actual entry class gets named. In those cases also look for
cassandra-specific config options.
* Let postgres binaries read sensitive files
Also add a couple of postgres cluster management programs.
* Add apt-add-reposit(ory) as a debian mgmt program
* Add addl info to debug writing sensitive files
Add parent/grandparent process info.
* Requrire root directory files to contain /
In some cases, a file below root might be detected but the file itself
has no directory component at all. This might be a bug with dropped
events. Make the test more strict by requiring that the file actually
contains a "/".
* Let updmap read sensitive files
Part of texlive (https://www.tug.org/texlive/)
* For selected rules, require proc name to exist
Some rules such as reading sensitive files and writing below etc have
many exceptions that depend on the process name. In very busy
environments, system call events might end up being dropped, which
causes the process name to be missing.
In these cases, we'll let the sensitive file read/write below etc to
occur. That's handled by a macro proc_name_exists, which ensures that
proc.name is not "<NA>" (the placeholder when it doesn't exist).
* Let ucf write generally below /etc
ucf is a general purpose config copying program, so let it generally
write below /etc, as long as it in turn is run by the apt program
"frontend".
* Add new conf writers for couchdb/texmf/slapadd
Each has specific subdirectories below /etc
* Let sed write to addl temp files below /etc
Let sed write to additional temporary files (some directory + "sed")
below /etc. All generally related to package installation scripts.
* Let rabbitmq(ctl) spawn limited shells
Let rabbitmq spawn limited shells that perform read-only tasks like
reading processes/ifaces.
Let rabbitmqctl generally spawn shells.
* Let redis run startup/shutdown scripts
Let redis run specific startup/shutdown scripts that trigger at
start/stop. They generally reside below /etc/redis, but just looking for
the names redis-server.{pre,post}-up in the commandline.
* Let erlexec spawn shells
https://github.com/saleyn/erlexec, "Execute and control OS processes
from Erlang/OTP."
* Handle updated trace files
As a part of these changes, we updated some of the positive trace files
to properly include a process name. These newer trace files have
additional opens, so update the expected event counts to match.
* Let yum-debug-dump write to rpm database
* Additional config writers
Symantec AV for Linux, sosreport, semodule (selinux), all with their
config files.
* Tidy up comments a bit.
* Try protecting node apps again
Try improving coverage of run shell untrusted by looking for shells
below node processes again. Want to see how many FPs this causes before
fully committing to it.
* Let node run directly by docker count as a service
Generally, we don't want to consider all uses of node as a service wrt
spawned shells. But we might be able to consider node run directly by
docker as a "service". So add that to protected_shell_spawner.
* Also add PM2 as a protected shell spawner
This should handle cases where PM2 manages node apps.
* Remove dangling macros/lists
Do a pass over the set of macros/lists, removing most of those that are
no longer referred to by any macro/list. The bulk of the macros/lists
were related to the rule Run Shell Untrusted, which was refactored to
only detect shells run below specific programs. With that change, many
of these exceptions were no longer neeeded.
* Add a "never_true" macro
Add a never_true macro that will never match any event. Useful if you
want to disable a rule/macro/etc.
* Add missing case to write_below_etc
Add the macro veritas_writing_config to write_below_etc, which was
mistakenly not added before.
* Make tracking shells spawned by node optional
The change to generally consider node run directly in a container as a
protected shell spawner was too permissive, causing false
positives. However, there are some deployments that want to track shells
spawned by node as suspect. To address this, create a macro
possibly_node_in_container which defaults to never matching (via the
never_true) macro. In a user rules file, you can override the macro to
remove the never_true clause, reverting to the old behavior.
* Add some dangling macros/lists back
Some macros/lists are still referred to by some widely used user rules
files, so add them back temporarily.
* Refactor shell rules to avoid FPs.
Refactoring the shell related rules to avoid FPs. Instead of considering
all shells suspicious and trying to carve out exceptions for the
legitimate uses of shells, only consider shells spawned below certain
processes suspicious.
The set of processes is a collection of commonly used web servers,
databases, nosql document stores, mail programs, message queues, process
monitors, application servers, etc.
Also, runsv is also considered a top level process that denotes a
service. This allows a way for more flexible servers like ad-hoc nodejs
express apps, etc to denote themselves as a full server process.
* Update event generator to reflect new shell rules
spawn_shell is now a silent action. its replacement is
spawn_shell_under_httpd, which respawns itself as httpd and then runs a
shell.
db_program_spawn_binaries now runs ls instead of a shell so it only
matches db_program_spawn_process.
* Comment out old shell related rules
* Modify nodejs example to work w/ new shell rules
Start the express server using runit's runsv, which allows falco to
consider any shells run by it as suspicious.
* Use the updated argument for mkdir
In https://github.com/draios/sysdig/pull/757 the path argument for mkdir
moved to the second argument. This only became visible in the unit tests
once the trace files were updated to reflect the other shell rule
changes--the trace files had the old format.
* Update unit tests for shell rules changes
Shell in container doesn't exist any longer and its functionality has
been subsumed by run shell untrusted.
* Allow git binaries to run shells
In some cases, these are run below a service runsv so we still need
exceptions for them.
* Let consul agent spawn curl for health checks
* Don't protect tomcat
There's enough evidence of people spawning general commands that we
can't protect it.
* Reorder exceptions, add rabbitmq exception
Move the nginx exception to the main rule instead of the
protected_shell_spawner macro. Also add erl_child_setup (related to
rabbitmq) as an allowed shell spawner.
* Add additional spawn binaries
All off these are either below nginx, httpd, or runsv but should still
be allowed to spawn shells.
* Exclude shells when ancestor is a pkg mgmt binary
Skip shells when any process ancestor (parent, gparent, etc) is a
package management binary. This includes the program needrestart. This
is a deep search but should prevent a lot of other more detailed
exceptions trying to find the specific scripts run as a part of
installations.
* Skip shells related to serf
Serf is a service discovery tool and can in some cases be spawned by
apache/nginx. Also allow shells that are just checking the status of
pids via kill -0.
* Add several exclusions back
Add several exclusions back from the shell in container rule. These are
all allowed shell spawns that happen to be below
nginx/fluentd/apache/etc.
* Remove commented-out rules
This saves space as well as cleanup. I haven't yet removed the
macros/lists used by these rules and not used anywhere else. I'll do
that cleanup in a separate step.
* Also exclude based on command lines
Add back the exclusions based on command lines, using the existing set
of command lines.
* Add addl exclusions for shells
Of note is runsv, which means it can directly run shells (the ./run and
./finish scripts), but the things it runs can not.
* Don't trigger on shells spawning shells
We'll detect the first shell and not any other shells it spawns.
* Allow "runc:" parents to count as a cont entrypnt
In some cases, the initial process for a container can have a parent
"runc:[0:PARENT]", so also allow those cases to count as a container
entrypoint.
* Use container_entrypoint macro
Use the container_entrypoint macro to denote entering a container and
also allow exe to be one of the processes that's the parent of an
entrypoint.
Clean up the handling of priority levels within rules. It used to be a
mix of strings handled in various places. Now, in falco_common.h there's
a consistent type for priority-as-number as well as a list of
priority-as-string values. Priorities are passed around as numbers
instead of strings. It's still permissive about capitalization.
Also add the ability to load rules by severity. New falco
config option "priority=<val>"/-o priority=<val> specifies the minimum
priority level of rules that will be loaded.
Add unit tests for same. The test suppresses INFO notifications for a
rule/trace file combination that would otherwise generate them.
Add the ability to append to rules/macros, like we already do with
lists. For rules/macros, if the object has an append: true key, the
condition value is appended to the condition of an existing rule/macro
with the same name.
Like lists, it's an error to specify append: true without there being an
existing rule/macro.
Also add tests that test the same kind of things we did for lists:
- That append: true really does append
- That append: false overwrites the rule/macro
- That it's an error to append with a prior rule/macro existing.
Add new unit tests to check that list substitution is working as
expected, with test cases for the list substitution occurring at the
beginning, middle, and end of a condition.
Also add tests that verify that overrides on list/macro/rule names
always occur in order.
* Updates from beta customers.
- add anacron as a cron program
* Reorganize package management binaries
Split package_management_binaries into two separate lists rpm_binaries
and deb_binaries. unattended-upgr is common to both worlds so it's still
in package_management_binaries.
Also change Write below rpm database to use rpm_binaries instead of its
own list.
Also add 75-system-updat (truncated) as a shell spawner.
* Add rules for jenkins
Add rules that allow jenkins to spawn shells, both in containers and
directly on the host.
Also handle jenkins slaves that run /tmp/slave.jar.
* Allow npm to run shells.
Not yet allowing node to run shells itself, although we want to add
something to reduce node-related FPs.
* Allow urlgrabber/git-remote to access /etc
urlgrabber and git-remote both try to access the RHEL nss database,
containing shared certificates. I may change this in a more general way
by changing open_read/open_write to only look for successful opens.
* Only look for successful open_read/open_writes
Change the macros open_read/open_write to only trigger on successful
opens (when fd.num > 0). This is a pretty big change to behavior, but
is more intuitive.
This required a small update to the open counts for a couple of unit
tests, but otherwise they still all passed with this change.
* Allow rename_device to write below /dev
Part of udev.
* Allow cloud-init to spawn shells.
Part of https://cloud-init.io/
* Allow python to run a shell that runs sdchecks
sdchecks is a part of the sysdig monitor agent.
* Allow dev creation binaries to write below etc.
Specifically this includes blkid and /etc/blkid/blkid.tab.
* Allow git binaries to spawn shells.
They were already allowed to run shells in a container.
* Add /dev/kmsg as an allowed /dev file
Allows userspace programs to write to kernel log.
* Allow other make programs to spawn shells.
Also allow gmake/cmake to spawn shells and put them in their own list
make_binaries.
* Add better mesos support.
Mesos slaves appear to be in a container due to their cgroup and can run
programs mesos-health-check/mesos-docker-exec to monitor the containers
on the slave, so allow them to run shells.
Add mesos-agent, mesos-logrotate, mesos-fetch as shell spawners both in
and out of containers.
Add gen_resolvconf. (short for gen_resolvconf.py) as a program that can
write to /etc.
Add toybox (used by mesos, part of http://landley.net/toybox/about.html)
as a shell spawner.
* systemd can listen on network ports.
Systemd can listen on network ports to launch daemons on demand, so
allow it to perform network activity.
* Let docker binaries setuid.
Let docker binaries setuid and add docker-entrypoi (truncation
intentional) to the set of docker binaries.
* Change cis-related rules to be less noisy
Change the two cis-related falco rules "File Open by Privileged
Container" and "Sensitive Mount by Container" to be less noisy. We found
in practice that tracking every open still results in too many falco
notifications.
For now, change the rules to only track the initial process start in the
container by looking for vpid=1. This should result in only triggering
when a privileged/sensitive mount container is started. This is slightly
less coverage but is far less noisy.
* Add quay.io/sysdig as trusted containers
These are used for sysdig cloud onpremise deployments.
* Add gitlab-runner-b(uild) as a gitlab binary.
Add gitlab-runner-b (truncated gitlab-runner-build) as a gitlab binary.
* Add ceph as a shell spawner.
Also allow ceph to spawn shells in a container.
* Allow some shells by command line.
For some mesos containers, where the container doesn't have an image and
is just a tarball in a cgroup/namespace, we don't have any image to work
with. In those cases, allow specific command lines.
* Allow user 'nobody' to setuid.
Allow the user nobody to setuid. This depends on the user nobody being
set up in the first place to have no access, but that should be an ok
assumption.
* Additional allowed shell commandlines
* Add additional shells.
* Allow multiple users to become themself.
Add rule somebody_becoming_themself that handles cases of nobody and
www-data trying to setuid to themself. The sysdig filter language
doesn't support template/variable values to allow "user.name=X and
evt.arg.uid=X for a given X", so we have to enumerate the users.
* More known spawn command lines
* Let make binaries be run in containers.
Some CI/CD pipelines build in containers.
* Add additional shell spawning command lines
* Add additional apt program apt-listchanges.
* Add gitlab-ce as shell spawning container.
* Allow PM2 to spawn shells in containers.
Was already in the general list, seen in some customers, so adding to
the in containers list.
* Clean up pass to fix long lines.
Take a pass through the rules making sure each line is < 120 characters.
* Change tests for privileged container rules.
Change unit tests to reflect the new privileged/sensitive mount
container rules that only detect container launch.
The default falco ruleset now has a wider variety of priorities, so
adjust the automated tests to match:
- Instead of creating a generic test yaml entry for every trace file in
traces-{positive,negative,info} with assumptions about detect levels,
add a new falco_traces.yaml.in multiplex file that has specific
information about the detect priorities and rule detect counts for each
trace file.
- If a given trace file doesn't have a corresponding entry in
falco_traces.yaml.in, a generic entry is added with a simple
detect: (True|False) value and level. That way you can get specific
detect levels/counts for existing trace files, but if you forget to
add a trace to falco_traces.yaml.in, you'll still get some coverage.
- falco_tests.yaml.in isn't added to any longer, so rename it to
falco_tests.yaml.
- Avocado is now run twice--once on each yaml file. The final test
passes if both avocado runs pass.
Add automated tests for running falco from a package and container. As a
result, this will also test building the kernel module as well as
runnning falco-probe-loader as a backup.
In travis.yml, switch to the docker-enabled vm and install dkms. This
changed the environment slightly, so change how avocado's python
dependencies are installed. After building falco, copy the .deb package
to docker/local and build a local docker image based on that package.
Add the following new tests:
- docker_package: this uses "docker run" to run the image created in
travis.yml. This includes using dkms to build the kernel module and
load it. In addition, the conf directory is mounted to /host/conf, the
rules directory is mounted to /host/rules, and the traces directory is
mounted to /host/traces.
- docker_package_local_driver: this disables dkms via a volume mount
that maps /dev/null to /usr/sbin/dkms and copies the kernel module by
hand into the container to /root/.sysdig/falco-probe-....ko. As a
result, falco-probe-loader will use the local kernel module instead
of building one itself.
- debian_package: this installs the .deb package and runs the installed
version of falco.
Ideally, there'd also be a test for downloading the driver, but since
the driver depends on the kernel as well as the falco version string,
you can't put a single driver on download.draios.com that will work
long-term.
These tests depend on the following new test attributes:
- package: if present, this points to the docker image/debian package
to install.
- addl_docker_run_args: if present, will be added to the docker run
command.
- copy_local_driver: if present, will copy the built kernel module to
~/.sysdig. ~/.sysdig/* is always cleared out before each test.
- run_duration: maps to falco's -M <secs> flag
- trace_file is now optional.
Also add some misc general test changes:
- Clean up our use of process.run. By default it will fail a test if the
run program returns non-zero, so we don't have to grab the exit
status. In addition, get rid of sudo in the command lines and use the
sudo attribute instead.
- Fix some tests that were writing to files below /tmp/falco_outputs
by creating the directory first. Useful when running avocado directly.
Start packaging (and building when necessary) a falco-specific kernel
module in falco releases. Previously, falco would depend on sysdig and
use its kernel module instead.
The kernel module was already templated to some degree in various
places, so we just had to change the templated name from
sysdig/sysdig-probe to falco/falco-probe.
In containers, run falco-probe-loader instead of
sysdig-probe-loader. This is actually a script in the sysdig repository
which is modified in https://github.com/draios/sysdig/pull/789, and uses
the filename to indicate what kernel module to build and/or load.
For the falco package itself, don't depend on sysdig any longer but instead
depend on dkms and its dependencies, using sysdig as a guide on the set
of required packages.
Additionally, for the package pre-install/post-install scripts start
running falco-probe-loader.
Finally, add a --version argument to falco so it can pass the desired
version string to falco-probe-loader.
- Instead of having a possibly null string pointer as the argument to
enable_* and process_event, have wrapper versions that assume a
default falco ruleset. The default ruleset name is a static member of
the falco_engine class, and the default ruleset id is created/found
in the constructor.
- This makes the whole mechanism simple enough that it doesn't require
seprarate testing, so remove the capability within falco to read a
ruleset from the environment and remove automated tests that specify
a ruleset.
- Make pattern/tags/ruleset arguments to enable_* functions const.
(I'll squash this down before I commit)
Add automated tests that verify the ability to tag sets of rules,
disable them with -T, and run them with -t, works:
- New test option disable_tags adds -T <tag> arguments to the falco
command line, and run_tags adds -t <tag> arguments to the falco command
line.
- A new trace file open-multiple-files.scap opens 13 different files,
and a new rules file has 13 different rules with all combinations of
the tags a, b, c (both forward and backward), a rule with an empty
list of tags, a rule with no tags field, and a rule with a completely
different tag d.
Using the above, add tests for:
- Both disabling all combations of a, b, c using disable_tags as well as
run all combinations of a, b, c, using run_tags.
- Specifying both disabled (-T/-D) and enabled (-t) rules. Not allowed.
- Specifying a ruleset while having tagged rules enabled, rules based
on a name disabled, and no particular rules enabled or disabled.
A new trace file falco-event-generator.scap contains the result of
running the falco event generator in docker, via:
docker run --security-opt seccomp=unconfined sysdig/falco-event-generator:latest /usr/local/bin/event_generator --once
Make sure this trace file detects the exact set of events we expect for
each rule. This required adding a new verification method
check_detections_by_rule that finds the per-rule counts and compares
them to the expected counts, which are included in the test description
under the key "detect_counts".
This is the first time a trace file for a test is actually in one of the
downloaded zip files. This means it will be tested twice (one for simple
detect-or-not, once for actual counts).
Adding this test showed a problem with Run shell in container
rule--since sysdig/falco-event-generator startswith sysdig/falco, it was
being treated as a trusted container. Modify the macro
trusted_containers to not allow falco-event-generator to be trusted.
Add a test that specifically tests truncated outputs. A rule contains an
output field %fd.cport which has no value for an open event. Ensure that
the rule's output has <NA> for the cport and the remainder of the rule's
output is filled in.
New tests that test every possible override:
- Overriding a rule with one that doesn't match
- Overriding a macro to one that doesn't match
- Overriding a top level list to a binary that doesn't match
- Overriding an embedded list to one that doesn't match
In each case, the override results in no longer matching an open by the
program "cat".
New argument --metric, which can be cpu|drops, controls whether to graph
cpu usage or event drop percentage. Titles/axis labels/etc. change
appropriately.
When run via scripts like run_performance_tests.sh, it's useful to
include extra info like the test being run and the specific program
variant to the stats file. So support that via the
environment. Environment keys starting with FALCO_STATS_EXTRA_XXX will
have the XXX and environment value added to the stats file.
It's undocumented as I doubt other programs will need this functionality
and it keeps the docs simpler.
Add the ability to check falco's return code with exit_status and to
generally match stderr with stderr_contains in a test.
Use those to create a test that has an invalid output expression using
%not_a_real_field. It expects falco to exit with 1 and the output to
contain a message about the invalid output.
Make necessary changes to allow run_performance_tests to invoke the
'test_mm' program we use internally.
Also add ability to run with a build directory separate from the source
directory and to specify an alternate rules file.
Finally, set up the kubernetes demo using sudo, a result of recent changes.
- In the regression tests, make the config file configurable in the
multiplex file via 'conf_file'.
- A new multiplex file item 'outputs' containing a list of <filename>:
<regex> tuples. For each item, the test reads the file and matches
each line against the regex. A match must be found for the test to
pass.
- Add 2 new tests that test file output and program output. They write
to files below /tmp/falco_outputs/ and the contents are checked to
ensure that alerts are written.
