Update tests that validated rules files (typically looking for
errors/warnings) to use the new result struct + json based validation:
- When validating rules files, always use json output.
- In test cases, instead of parsing stderr/stdout, use new test
properties "validate_ok", "validate_errors",
"validate_warnings". These parse the json output and look for
specific tuples of (error code, error message, item type, item name)
in the output.
- There were a few tests that were actually validation tests but using
the -r argument to load rules. Convert them to validation tests. In
one case, split the test into two separate tests--one for
validation, one ensuring that the rule doesn't match anything.
- There were a couple of tests that were duplicates of existing
validation tests, just checking for the error in a different
way. Remove them.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
With the split of falco_init into application:: methods, including a
notion of a success/error/proceed result, there is a
more concrete line between runtime errors (e.g. things that throw
exceptions) and errors returned by methods that run the application.
Some of the plugins tests were expecting errors to be returned by
exceptions (e.g. with a leading "Runtime error: and a trailing "
Exiting.").
Now, the errors are just returned directly in application::run(), so
drop the leading and trailing bits in expected test outputs.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Add new test cases for a rule with an unknown source *and* an
exception, and a macro with an unknown source.
The first results in a rule warning (and no error), and the second
prints an error and skips.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Test infrastructure and sample confs/rules/traces for plugins
automated tests:
New test cases are in falco_tests_plugins.yaml and cover:
- Listing plugins and fields when plugins are loaded.
- Basic cloudtrail + json plugin on a fake cloudtrail json file and a
sample rule that uses both plugins.
- Conflicts between source/extractor plugins
- Incompatible plugin api
- Wrong plugin path
- Checking for warnings when reading rules with unnown sources (e.g. when plugins are not loaded)
Some test-only plugins written in C are in test/plugins and built on
the fly. (They aren't included in packages of course).
The test framework needed some small changes to handle these tests:
- Add a mode to not check detection counts at all (for --list/--list-plugins)
- addl_cmdline_opts to allow specifying --list/--list-plugins
- Using DOTALL when matching stderr/stdout (allows multi-line matches more easily)
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
