Clean up the handling of priority levels within rules. It used to be a
mix of strings handled in various places. Now, in falco_common.h there's
a consistent type for priority-as-number as well as a list of
priority-as-string values. Priorities are passed around as numbers
instead of strings. It's still permissive about capitalization.
Also add the ability to load rules by severity. New falco
config option "priority=<val>"/-o priority=<val> specifies the minimum
priority level of rules that will be loaded.
Add unit tests for same. The test suppresses INFO notifications for a
rule/trace file combination that would otherwise generate them.
Add the ability to append to rules/macros, like we already do with
lists. For rules/macros, if the object has an append: true key, the
condition value is appended to the condition of an existing rule/macro
with the same name.
Like lists, it's an error to specify append: true without there being an
existing rule/macro.
Also add tests that test the same kind of things we did for lists:
- That append: true really does append
- That append: false overwrites the rule/macro
- That it's an error to append with a prior rule/macro existing.
List nodes can now have an 'append' key. If present and true, any values
in this list will be appended to the end of any existing list with the
same name.
It is an error to have a list with 'append' true that has a name that is
not an existing list.
Also validate macros when they are parsed. Macros are also validated as
a part of rules being parsed, but it's possible to have an individual
rules file containing only macros, or a macro not explicitly tied to any
rule. In this case, it's useful to be able to check the macro to see if
it contains dangling macro references.
- in lua, look for a tags attribute to each rule. This is passed up in
add_filter as a tags argument (as a lua table). If not present, an
empty table is used. The tags table is iterated to populate a set
of tags as strings, which is passed to add_filter().
- A new method falco_engine::enable_rule_by_tag is similar to
enable_rule(), but is given a set of tag strings. Any rules containing
one of the tags is enabled/disabled.
- The list of event types has been changed to a set to more accurately
reflect its purpose.
- New argument to falco -T allows disabling all rules matching a given
tag, via enable_rule_by_tag(). It can be provided multiple times.
- New argument to falco -t allows running those rules matching a given
tag. If provided all rules are first disabled. It can be
provided multiple times, but can not be combined with -T or
-D (disable rules by name)
- falco_enging supports the notion of a ruleset. The idea is that you
can choose a set of rules that are enabled/disabled by using
enable_rule()/enable_rule_by_tag() in combination with a
ruleset. Later, in process_event() you include that ruleset and the
rules you had previously enabled will be run.
- rulsets are provided as strings in enable_rule()/enable_rule_by_tag()
and as numbers in process_event()--this avoids the overhead of string
lookups per-event. Ruleset ids are created on the fly as needed. A
utility method find_ruleset_id() looks up the ruleset id for a given
name. The default ruleset is NULL string/0 numeric if not provided.
- Although the ruleset is a useful falco engine feature, it isn't that
important to the falco standalone program, so it's not
documented. However, you can change the ruleset by providing
FALCO_RULESET in the environment.
Prefix output strings with * so they are always permissive in the
engine.
In falco outputs, which adds its own prefix, remove any leading * before
adding the custom prefix.
Allow any list/macro/rule to be overridden by a subsequent file. The
persistent state that lives across invocations of load_rules are the 3
arrays ordered_{list,macro,rule}_names, which have the
lists/macros/rules in the order in which they first appear, and tables
{rules,macros,lists}_by_name, which maps from a name to a yaml object.
With each call to load_rules, the set of loaded rules is reset and the
state of expanded lists, compiled macros, compiled rules, and rule
metadata are recreated from scratch, using the ordered_*_names arrays
and *_by_name tables. That way, any list/macro/rule can be redefined in
a subsequent file with new values.
Validate rule outputs when loading rules by attempting to create a
formatter based on the rule's output field. If there's an error, it will
propagate up through load_rules and cause falco to exit rather than
discover the problem only when trying to format the event and the rule's
output field.
This required moving formats.{cpp,h} into the falco engine directory
from the falco general directory. Note that these functions are loaded
twice in the two lua states used by falco (engine and outputs).
There's also a couple of minor cleanups:
- falco_formats had a private instance variable that was unused, remove
it.
- rename the package for the falco_formats functions to formats instead
of falco so it's more standalone.
- don't throw a c++ exception in falco_formats::formatter. Instead
generate a lua error, which is handled more cleanly.
- free_formatter doesn't return any values, so set the return value of
the function to 0.
container.info handling used to be handled by the the falco_outputs
object. However, this caused problems for applications that only used
the falco engine, doing their own output formatting for matching events.
Fix this by moving output formatting into the falco engine itself. The
part that replaces %container.info/adds extra formatting to the end of a
rule's output now happens while loading the rule.
Related to the changes in https://github.com/draios/agent/pull/267,
improve error messages when trying to load sets of rules with errors:
- Check that yaml parsing of rules_content actually resulted in
something.
- Return an error for rules that have an empty name.
- Return an error for yaml objects that aren't a rule/macro/list.
- When compiling, don't print an error message, simply return one,
including a wrapper "can not compile ..." string.
If a rule has a enabled attribute, and if the value is false, call the
engine's enable_rule() method to disable the rule. Like add_filter,
there's a static method which takes the object as the first argument and
a non-static method that calls the engine.
This fixes#72.
Move the c++ and lua code implementing falco engine/falco common to its
own directory userspace/engine. It's compiled as a static library
libfalco_engine.a, and has its own CMakeLists.txt so it can be included
by other projects.
The engine's CMakeLists.txt has a add_subdirectory for the falco rules
directory, so including the engine also builds the rules.
The variables you need to set to use the engine's CMakeLists.txt are:
- CMAKE_INSTALL_PREFIX: the root directory below which everything is
installed.
- FALCO_ETC_DIR: where to install the rules file.
- FALCO_SHARE_DIR: where to install lua code, relative to the
- install/package root.
- LUAJIT_INCLUDE: where to find header files for lua.
- FALCO_SINSP_LIBRARY: the library containing sinsp code. It will be
- considered a dependency of the engine.
- LPEG_LIB/LYAML_LIB/LIBYAML_LIB: locations for third-party libraries.
- FALCO_COMPONENT: if set, will be included as a part of any install()
commands.
Instead of specifying /usr/share/falco in config_falco_*.h.in, use
CMAKE_INSTALL_PREFIX and FALCO_SHARE_DIR.
The lua code for the engine has also moved, so the two lua source
directories (userspace/engine/lua and userspace/falco/lua) need to be
available separately via falco_common, so make it an argument to
falco_common::init.
As a part of making it easy to include in another project, also clean up
LPEG build/defs. Modify build-lpeg to add a PREFIX argument to allow for
object files/libraries being in an alternate location, and when building
lpeg, put object files in a build/ subdirectory.
