Modify the disabled_rules_using_regex test to
disabled_rules_using_substring with an appropriate substring.
Also add a test where rule names have regex chars and allow rule names
to have regex chars when parsing falco's output in tests. These changes
are future-looking in case we want to add back support for rule
enabling/disabling using regexes.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
New test options stdout_is/stderr_is do a direct comparison between
stdout/stderr and the provided value.
Test option validate_rules_file maps to -V arguments, which validate
rules and exits.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
* Supporting files to build/test via jenkins
Changes to build/test via jenkins, which also means running all tests in
a container instead of directly on the host:
- Jenkinsfile controls the stages, build.sh does the build and
run-tests.sh does the regression tests.
- Create a new container falcosecurity/falco-tester that includes the
dependencies required to run the regression tests. This is a different
image than falco-builder because it doesn't need to be centos 6 based,
doesn't install any compiler/etc, and installs the test running
framework we use (avocado). We now use a newer version of avocado,
which resulted in some small changes to how it is run and how yaml
options are parsed.
- Modify run_regression_tests.sh to download trace files to the build
directory and only if not present. Also honor BUILD_TYPE/BUILD_DIR,
which is provided via the docker run cmd.
- The package tests are now moved to a separate falco_tests_package.yaml
file. They will use rpm installs by default instead of debian
packages. Also add the ability to install rpms in addition to debian
packages.
- Automate the process of creating the docker local package by: 1)
Adding CMake rules to copy the Dockerfile, entrypoint to the build
directory and 2) Copy test trace files and rules into the build
directory. This allows running the docker build command from
build/docker/local instead of the source directory.
- Modify the way the container test is run a bit to use the trace
files/rules copied into the container directly instead of host-mounted
trace files.
* Use container builder + tester for travis
We'll probably be using jenkins soon, but this will allow switching back
to travis later if we want.
* Use download.draios.com for binutils packages
That way we won't be dependent on snapshot.debian.org.
* Add option to display times in ISO 8601 UTC
ISO 8601 time is useful when, say, running falco in a container, which
may have a different /etc/localtime than the host system.
A new config option time_format_iso_8601 controls whether log message
and event times are displayed in ISO 8601 in UTC or in local time. The
default is false (display times in local time).
This option is passed to logger init as well as outputs. For outputs it
eventually changes the time format field from %evt.time/%jevt.time to
%evt.time.iso8601/%jevt.time.iso8601.
Adding this field changes the falco engine version so increment it.
This depends on https://github.com/draios/sysdig/pull/1317.
* Unit test for ISO 8601 output
A unit test for ISO 8601 output ensures that both the log and event time
is in ISO 8601 format.
* Use ISO 8601 output by default in containers
Now that we have an option that controls iso 8601 output, use it by
default in containers. We do this by changing the value of
time_format_iso_8601 in falco.yaml in the container.
* Handle errors in strftime/asctime/gmtime
A placeholder "N/A" is used in log messages instead.
* Make stats file interval configurable
New argument --stats_interval=<msec> controls the interval at which
statistics are written to the stats file. The default is 5000 ms (5 sec)
which matches the prior hardcoded interval.
The stats interval is triggered via signals, so an interval below ~250ms
will probably interfere with falco's behavior.
* Add ability to emit general purpose messages
A new method falco_outputs::handle_msg allows emitting generic messages
that have a "rule", message, and output fields, but aren't exactly tied
to any event and aren't passed through an event formatter.
This allows falco to emit "events" based on internal checks like kernel
buffer overflow detection.
* Clean up newline handling for logging
Log messages from falco_logger::log may or may not have trailing
newlines. Handle both by always adding a newline to stderr logs and
always removing any newline from syslog logs.
* Add method to get sequence from subkey
New variant of get_sequence that allows fetching a list of items from a
key + subkey, for example:
key:
subkey:
- list
- items
- here
Both use a shared method get_sequence_from_node().
* Monitor syscall event drops + optional actions
Start actively monitoring the kernel buffer for syscall event drops,
which are visible in scap_stats.n_drops, and add the ability
to take actions when events are dropped. The -v (verbose) and
-s (stats filename) arguments also print out information on dropped
events, but they were only printed/logged without any actions.
In falco config you can specify one or more of the following actions to
take when falco notes system call drops:
- ignore (do nothing)
- log a critical message
- emit an "internal" falco alert. It looks like any other alert with a
time, "rule", message, and output fields but is not related to any
rule in falco_rules.yaml/other rules files.
- exit falco (the idea being that the restart would be monitored
elsewhere).
A new module syscall_event_drop_mgr is called for every event and
collects scap stats every second. If in the prior second there were
drops, perform_actions() handles the actions.
To prevent potential flooding in high drop rate environments, actions
are goverened by a token bucket with a rate of 1 actions per 30 seconds,
with a max burst of 10 seconds. We might tune this later based on
experience in busy environments.
This might be considered a fix for
https://github.com/falcosecurity/falco/issues/545. It doesn't
specifically flag falco rules alerts when there are drops, but does
make it easier to notice when there are drops.
* Add unit test for syscall event drop detection
Add unit tests for syscall event drop detection. First, add an optional
config option that artifically increments the drop count every
second. (This is only used for testing).
Then add test cases for each of the following:
- No dropped events: should not see any log messages or alerts.
- ignore action: should note the drops but not log messages or alert.
- log action: should only see log messages for the dropped events.
- alert action: should only see alerts for the dropped events.
- exit action: should see log message noting the dropped event and exit
with rc=1
A new trace file ping_sendto.scap has 10 seconds worth of events to
allow the periodic tracking of drops to kick in.
* Use correct copyright years.
Also include the start year.
* Improve copyright notices.
Use the proper start year instead of just 2018.
Add the right owner Draios dba Sysdig.
Add copyright notices to some files that were missing them.
Replace references to GNU Public License to Apache license in:
- COPYING file
- README
- all source code below falco
- rules files
- rules and code below test directory
- code below falco directory
- entrypoint for docker containers (but not the Dockerfiles)
I didn't generally add copyright notices to all the examples files, as
they aren't core falco. If they did refer to the gpl I changed them to
apache.
* Use better way to skip falco events
Use the new method falco_consider() to determine which events to
skip. This centralizes the logic in a single function. All events will
still be considered if falco was run with -A.
This depends on https://github.com/draios/sysdig/pull/1105.
* Add ability to specify -A flag in tests
test attribute all_events corresponds to the -A flag. Add for some tests
that would normally refer to skipped events.
* Only check whole rule names when matching counts
Tweak the regex so a rule my_great_rule doesn't pick up event counts for
a rule "great_rule: nnn".
* Add ability to skip evttype warnings for rules
A new attribute warn_evttypes, if present, suppresses printing warnings
related to a rule not matching any event type. Useful if you have a rule
where not including an event type is intentional.
* Add test for preserving rule order
Test the fix for https://github.com/draios/falco/issues/354. A rules
file has a event-specific rule first and a catchall rule second. Without
the changes in https://github.com/draios/sysdig/pull/1103, the first
rule does not match the event.
* Add option to exclude output property in json fmt
New falco.yaml option json_include_output_property controls where the
formatted string "output" is included in the json object when json
output is enabled. By default the string is included.
* Add tests for new json output option
New test sets json_include_output_property to false and then verifies
that the json output does *not* contain the surrounding text "Warning an
open...".
Clean up the handling of priority levels within rules. It used to be a
mix of strings handled in various places. Now, in falco_common.h there's
a consistent type for priority-as-number as well as a list of
priority-as-string values. Priorities are passed around as numbers
instead of strings. It's still permissive about capitalization.
Also add the ability to load rules by severity. New falco
config option "priority=<val>"/-o priority=<val> specifies the minimum
priority level of rules that will be loaded.
Add unit tests for same. The test suppresses INFO notifications for a
rule/trace file combination that would otherwise generate them.
Add automated tests for running falco from a package and container. As a
result, this will also test building the kernel module as well as
runnning falco-probe-loader as a backup.
In travis.yml, switch to the docker-enabled vm and install dkms. This
changed the environment slightly, so change how avocado's python
dependencies are installed. After building falco, copy the .deb package
to docker/local and build a local docker image based on that package.
Add the following new tests:
- docker_package: this uses "docker run" to run the image created in
travis.yml. This includes using dkms to build the kernel module and
load it. In addition, the conf directory is mounted to /host/conf, the
rules directory is mounted to /host/rules, and the traces directory is
mounted to /host/traces.
- docker_package_local_driver: this disables dkms via a volume mount
that maps /dev/null to /usr/sbin/dkms and copies the kernel module by
hand into the container to /root/.sysdig/falco-probe-....ko. As a
result, falco-probe-loader will use the local kernel module instead
of building one itself.
- debian_package: this installs the .deb package and runs the installed
version of falco.
Ideally, there'd also be a test for downloading the driver, but since
the driver depends on the kernel as well as the falco version string,
you can't put a single driver on download.draios.com that will work
long-term.
These tests depend on the following new test attributes:
- package: if present, this points to the docker image/debian package
to install.
- addl_docker_run_args: if present, will be added to the docker run
command.
- copy_local_driver: if present, will copy the built kernel module to
~/.sysdig. ~/.sysdig/* is always cleared out before each test.
- run_duration: maps to falco's -M <secs> flag
- trace_file is now optional.
Also add some misc general test changes:
- Clean up our use of process.run. By default it will fail a test if the
run program returns non-zero, so we don't have to grab the exit
status. In addition, get rid of sudo in the command lines and use the
sudo attribute instead.
- Fix some tests that were writing to files below /tmp/falco_outputs
by creating the directory first. Useful when running avocado directly.
Start packaging (and building when necessary) a falco-specific kernel
module in falco releases. Previously, falco would depend on sysdig and
use its kernel module instead.
The kernel module was already templated to some degree in various
places, so we just had to change the templated name from
sysdig/sysdig-probe to falco/falco-probe.
In containers, run falco-probe-loader instead of
sysdig-probe-loader. This is actually a script in the sysdig repository
which is modified in https://github.com/draios/sysdig/pull/789, and uses
the filename to indicate what kernel module to build and/or load.
For the falco package itself, don't depend on sysdig any longer but instead
depend on dkms and its dependencies, using sysdig as a guide on the set
of required packages.
Additionally, for the package pre-install/post-install scripts start
running falco-probe-loader.
Finally, add a --version argument to falco so it can pass the desired
version string to falco-probe-loader.
- Instead of having a possibly null string pointer as the argument to
enable_* and process_event, have wrapper versions that assume a
default falco ruleset. The default ruleset name is a static member of
the falco_engine class, and the default ruleset id is created/found
in the constructor.
- This makes the whole mechanism simple enough that it doesn't require
seprarate testing, so remove the capability within falco to read a
ruleset from the environment and remove automated tests that specify
a ruleset.
- Make pattern/tags/ruleset arguments to enable_* functions const.
(I'll squash this down before I commit)
Add automated tests that verify the ability to tag sets of rules,
disable them with -T, and run them with -t, works:
- New test option disable_tags adds -T <tag> arguments to the falco
command line, and run_tags adds -t <tag> arguments to the falco command
line.
- A new trace file open-multiple-files.scap opens 13 different files,
and a new rules file has 13 different rules with all combinations of
the tags a, b, c (both forward and backward), a rule with an empty
list of tags, a rule with no tags field, and a rule with a completely
different tag d.
Using the above, add tests for:
- Both disabling all combations of a, b, c using disable_tags as well as
run all combinations of a, b, c, using run_tags.
- Specifying both disabled (-T/-D) and enabled (-t) rules. Not allowed.
- Specifying a ruleset while having tagged rules enabled, rules based
on a name disabled, and no particular rules enabled or disabled.
A new trace file falco-event-generator.scap contains the result of
running the falco event generator in docker, via:
docker run --security-opt seccomp=unconfined sysdig/falco-event-generator:latest /usr/local/bin/event_generator --once
Make sure this trace file detects the exact set of events we expect for
each rule. This required adding a new verification method
check_detections_by_rule that finds the per-rule counts and compares
them to the expected counts, which are included in the test description
under the key "detect_counts".
This is the first time a trace file for a test is actually in one of the
downloaded zip files. This means it will be tested twice (one for simple
detect-or-not, once for actual counts).
Adding this test showed a problem with Run shell in container
rule--since sysdig/falco-event-generator startswith sysdig/falco, it was
being treated as a trusted container. Modify the macro
trusted_containers to not allow falco-event-generator to be trusted.
Add a test that specifically tests truncated outputs. A rule contains an
output field %fd.cport which has no value for an open event. Ensure that
the rule's output has <NA> for the cport and the remainder of the rule's
output is filled in.
Add the ability to check falco's return code with exit_status and to
generally match stderr with stderr_contains in a test.
Use those to create a test that has an invalid output expression using
%not_a_real_field. It expects falco to exit with 1 and the output to
contain a message about the invalid output.
- In the regression tests, make the config file configurable in the
multiplex file via 'conf_file'.
- A new multiplex file item 'outputs' containing a list of <filename>:
<regex> tuples. For each item, the test reads the file and matches
each line against the regex. A match must be found for the test to
pass.
- Add 2 new tests that test file output and program output. They write
to files below /tmp/falco_outputs/ and the contents are checked to
ensure that alerts are written.
Add test that cover reading from multiple sets of rule files and
disabling rules. Specific changes:
- Modify falco to allow multiple -r arguments to read from multiple
files.
- In the test multiplex file, add a disabled_rules attribute,
containing a sequence of rules to disable. Result in -D arguments
when running falco.
- In the test multiplex file, 'rules_file' can be a sequence. It
results in multiple -r arguments when running falco.
- In the test multiplex file, 'detect_level' can be a squence of
multiple severity levels. All levels will be checked for in the
output.
- Move all test rules files to a rules subdirectory and all trace files
to a traces subdirectory.
- Add a small trace file for a simple cat of /dev/null. Used by the
new tests.
- Add the following new tests:
- Reading from multiple files, with the first file being
empty. Ensure that the rules from the second file are properly
loaded.
- Reading from multiple files with the last being empty. Ensures
that the empty file doesn't overwrite anything from the first
file.
- Reading from multiple files with varying severity levels for each
rule. Ensures that both files are properly read.
- Disabling rules from a rules file, both with full rule names
and regexes. Will result in not detecting anything.
Create standalone classes falco_engine/falco_outputs that can be
embedded in other programs. falco_engine is responsible for matching
events against rules, and falco_output is responsible for formatting an
alert string given an event and writing the alert string to all
configured outputs.
falco_engine's main interfaces are:
- load_rules/load_rules_file: Given a path to a rules file or a string
containing a set of rules, load the rules. Also loads needed lua code.
- process_event(): check the event against the set of rules and return
the results of a match, if any.
- describe_rule(): print details on a specific rule or all rules.
- print_stats(): print stats on the rules that matched.
- enable_rule(): enable/disable any rules matching a pattern. New falco
command line option -D allows you to disable one or more rules on the
command line.
falco_output's main interfaces are:
- init(): load needed lua code.
- add_output(): add an output channel for alert notifications.
- handle_event(): given an event that matches one or more rules, format
an alert message and send it to any output channels.
Each of falco_engine/falco_output maintains a separate lua state and
loads separate sets of lua files. The code to create and initialize the
lua state is in a base class falco_common.
falco_engine no longer logs anything. In the case of errors, it throws
exceptions. falco_logger is now only used as a logging mechanism for
falco itself and as an output method for alert messages. (This should
really probably be split, but it's ok for now).
falco_engine contains an sinsp_evttype_filter object containing the set
of eventtype filters. Instead of calling
m_inspector->add_evttype_filter() to add a filter created by the
compiler, call falco_engine::add_evttype_filter() instead. This means
that the inspector runs with a NULL filter and all events are returned
from do_inspect. This depends on
https://github.com/draios/sysdig/pull/633 which has a wrapper around a
set of eventtype filters.
Some additional changes along with creating these classes:
- Some cleanups of unnecessary header files, cmake include_directory()s,
etc to only include necessary includes and only include them in header
files when required.
- Try to avoid 'using namespace std' in header files, or assuming
someone else has done that. Generally add 'using namespace std' to all
source files.
- Instead of using sinsp_exception for all errors, define a
falco_engine_exception class for exceptions coming from the falco
engine and use it instead. For falco program code, switch to general
exceptions under std::exception and catch + display an error for all
exceptions, not just sinsp_exceptions.
- Remove fields.{cpp,h}. This was dead code.
- Start tracking counts of rules by priority string (i.e. what's in the
falco rules file) as compared to priority level (i.e. roughtly
corresponding to a syslog level). This keeps the rule processing and
rule output halves separate. This led to some test changes. The regex
used in the test is now case insensitive to be a bit more flexible.
- Now that https://github.com/draios/sysdig/pull/632 is merged, we can
delete the rules object (and its lua_parser) safely.
- Move loading the initial lua script to the constructor. Otherwise,
calling load_rules() twice re-loads the lua script and throws away any
state like the mapping from rule index to rule.
- Allow an empty rules file.
Finally, fix most memory leaks found by valgrind:
- falco_configuration wasn't deleting the allocated m_config yaml
config.
- several ifstreams were being created simply to test which falco
config file to use.
- In the lua output methods, an event formatter was being created using
falco.formatter() but there was no corresponding free_formatter().
This depends on changes in https://github.com/draios/sysdig/pull/640.
Add tests that verify that the event type identification functionality
is working. Notable changes:
- Modify falco_test.py to additionally check for warnings when loading
any set of rules and verify that the event types for each rule match
expected values. This is controlled by the new multiplex fields
"rules_warning" and "rules_events".
- Instead of starting with an empty falco_tests.yaml from scratch from
the downloaded trace files, use a checked-in version which defines
two tests:
- Loading the checked-in falco_rules.yaml and verify that no rules
have warnings.
- A sample falco_rules_warnings.yaml that has ~30 different
mutations of rule filtering expressions. The test verifies for each
rule whether or not the rule should result in a warning and what the
extracted event types are.
The generated tests from the trace files are appended to this file.
- Add an empty .scap file to use with the above tests.
Pass the travis branch to run_regression_tests.sh. When downloading
trace files, first look for a file traces-XXX-$BRANCH and if found
download it. This allows testing out a set of changes with a trace file
specifically for that branch, that can be moved to the normal file once
the PR is merged.
Also increase the timeout for the spawned falco process from 1 to 3
minutes. In debug mode, the kubernetes demo was taking slightly over 1
minute.
Modify falco_test.py to look for a boolean multiplex attribute
'json_output'. If true, examine the lines of the output and for any line
that begins with '{', parse it as json and ensure it has the 4
attributes we expect.
Modify run_regression_tests to have a utility function
prepare_multiplex_fileset that does the work of looping over files in a
directory, along with detect, level, and json output arguments. The
appropriate multiplex attributes are added for each file.
Use that utility function to test json output for the positive and
informational directories along with non-json output. The negative
directory is only tested once.
Add additional rules related to using pipe installers within a fbash
session:
- Modify write_etc to only trigger if *not* in a fbash session. There's
a new rule write_etc_installer which has the same conditions when in
a fbash session, logging at INFO severity.
- A new rule write_rpm_database warns if any non package management
program tries to write below /var/lib/rpm.
- Add a new warning if any program below a fbash session tries to open
an outbound network connection on ports other than http(s) and dns.
- Add INFO level messages when programs in a fbash session try to run
package management binaries (rpm,yum,etc) or service
management (systemctl,chkconfig,etc) binaries.
In order to test these new INFO level rules, make up a third class of
trace files traces-info.zip containing trace files that should result in
info-level messages.
To differentiate warning and info level detection, add an attribute to
the multiplex file "detect_level", which is "Warning" for the files in
traces-positive and "Info" for the files in traces-info. Modify
falco_test.py to look specifically for a non-zero count for the given
detect_level.
Doing this exposed a bug in the way the level-specific counts were being
recorded--they were keeping counts by level name, not number. Fix that.
Start using the Avocado framework for automated regression
testing. Create a test FalcoTest in falco_test.py which can run on a
collection of trace files. The script test/run_regression_tests.sh is
responsible for pulling zip files containing the positive (falco should
detect) and negative (falco should not detect) trace files, creating a
Avocado multiplex file that defines all the tests (one for each trace
file), running avocado on all the trace files, and showing full logs for
any test that didn't pass.
The old regression script, which simply ran falco, has been removed.
Modify falco's stats output to show the total number of events detected
for use in the tests.
In travis.yml, pull a known stable version of avocado and build it,
including installing any dependencies, as a part of the build process.
