In outputs it could be confusing to see a line:
<filename>: Ok
followed by a set of warnings.
To differentiate this, add a top level status "Ok, with warnings" when
rule loading was successful but had warnings.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
The methods that throw exceptions from stringified results need to
additionally pass a rules_contents_t struct. This also meant that they
need to call the filename + content version of load_rules.
To avoid some duplicate code between the two load_rules_file methods,
move the work of opening the file into a private method
read_file(). It can throw an exception, which is passed through for
the void return method and caught + converted into a load_result error
for the method that returns a load_result.
Also, to avoid duplicate code between the void load_rules and
load_rules_file methods, add a private method interpret_load_result()
which throws an exception if the result has an error and prints
warnings otherwise if verbose is true.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Application changes to support multiple files when stringifying rules
results:
- In both validate_rules_files and load_rules_files, instead of
loading each file individually and then calling load_rules(), add a
separate step that loads all the files at once. The actual rules
content strings are held in a vector. The map from filename to
content (reference) points to entries in that vector.
- Both actions do the same work for this step, so put the
implementation in a shared application template method read_files
that works on iterators. It uses itertors because the load filenames
are a list and the validate filenames are a vector.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
The old version of rules_result assumed that all errors/warnings were
related to a single file. That was generally correct for errors, as
rules parsing always stopped at the first error, so there is only one
relevant file.
However, for warnings that was not the case. When reading multiple
files A and B, you might get a warning from file A *only* after
reading file B. For example, B might redefine a rule in such a way
that you could get unused list/macro warnings from file A.
To properly address this, make some changes to how contexts are
managed:
- Instead of creating snippets at the time the error/warning was
generated, create snippets at the time the error/warning is
converted into a string. This requires passing all rules contents to
as_string()/as_json(), so define a
falco::load_result::rules_contents_t map from filename to rules
content (reference) and pass it in as_string/as_json(). Snippets are
now generated from the rules content matching the filename in the
context.
- When creating warnings/errors, there's no need to pass along the
rules content. This is only used when converting an error into a
string/json.
Also change snippet() to handle potentially very long lines. Instead
of always printing the entire line matching a location, print up to
snippet_width(param, with default 160 chars)/2 characters surrounding
the column from the location.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
The latest released falco always prints full details on errors when
used with -r (read rules)/-V (validate rules). However #2098 changed
this to only print full details when verbose is true.
Fix the regression by always printing errors when loading
rules. Warnings will be printed only with -v.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Update the load_rules_files and validate_rules_files actions to use
the new falco_engine methods that return a rules result struct. The
app action interface is the same, returning ::fatal on error,
ok()/exit() otherwise. The difference is how any warnings/errors are
obtained--from the struct instead of an exception.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Changes to the rule loader to support result objects:
- Instead of throwing falco_exception on internal error, throw a
rule_load_exception instead, which contains distinct
error/message/context information.
- A context object contains a chain of location structs chaining from
the document root to the object where the error occurred. Each
location has a file position (as a YAML::Mark), an item
type (e.g. "rule", "list", "exception"), and an item name (e.g. "Write
Below Etc"). This will allow showing the exact location of an
error (e.g. list item/exception field) while also remembering the item
that contained it.
- All the _info structs now contain a context so errors that occur
after yaml parsing can still point to the original location in the
yaml file.
- rule_loader::result is an implementation of the abstract class
defined in falco_load_result. The implementation keeps track of a
list of errors/warnigns that used to be in the configuration object,
- Clean up compile_ methods to just throw rule_load_exceptions or
return nothing, and ensure that all rule_load_exceptions are caught in
compile(). When caught, errors are added to the result object.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Modify rule reader to use a result struct for errors and warnings:
- Instead of throwing a falco_exception to pass back errors, use a
rule_load_exception, which contains distinct error codes, messages,
and a context that points to the location of the error.
- The static method context_yaml_get_context() has moved to a method
of the rule_loader context object + the result as_string() method.
- As objects are parsed, create relevant context objects as reading
drills down into the contents of a rule/list/exception. This will
enable for specific errors in, say, the middle of an exception/list
while remembering the object that contains it.
- Modify decode_val()/decode_seq() to always return errors as
exceptions. Previously, it was a mix of a bool return + some
exceptions.
- decode_val()/decode_seq() are now more consistent about returning
distinct errors for the property not existing, the property existing
but not being a scalar/sequence value, and not being convertable to
the templated value.
- Combine the two nearly identical decode_seq() functions into a
single one that uses a lambda to perform the final step of adding to
the vector/set.
- There are some item-specific decode_xxx functions for specific
item properties like tags, exceptions fields/comps/values, etc.
that call the above functions.
These changes simplify the calls to decode_seq()/decode_val() as they
don't need to add their own errors when returning false. Also some
calls weren't checking the return value.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Now that we have a result struct and set of warning codes, change the
filter_warning_resolver to use them. This involves populating a set of
warning codes instead of strings.
Also, the methods to format warnings into human-readable strings is
now in the falco_load_result static methods, so move the text there
and remove the methods here.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Add new load_rules methods that return a result object instead of
throwing exceptions on error. The existing load_rules methods call the
new methods internally and continue to throw exceptions on
error/return individual values on success.
The result is returned as a unique_ptr so it can be populated while
loading rules (as a part of the configuration object) and then move()d
to the return value.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Define a falco_load_result abstract class for use in new load_rules
methods. It's abstract so the implementation details in
rule_loader/rule_reader can be hidden from someone who wants to use
the API to load rules and work with a result.
The class defines a set of error codes/warning codes and has static
methods to get a short and long description of each error/warning.
There are virtual methods to access the important parts of a result:
- successful or not
- a string representation of the result, suitable for display to
users. Takes a verbose argument. When verbose is true, the string is
multi-line and has full details, including locations, item names,
etc. When verbose is false, the string is single-line and just
returns error codes.
- a json representation of the result, suitable for automated
parsing/interpretation later.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
The only use of it was to include in --support output, which is
redundant as the support output already includes the full contents of
each rules file.
Additionally, it wasn't even being updated after the switch from lua
rules loading to c++ rules
loading (https://github.com/falcosecurity/falco/pull/1966/ or
surrounding PRs).
This will simplify follow-on changes to add a real "result" to rules
loading methods, as there will be fewer API variants to support.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Signed-off-by: Luca Guerra <luca@guerra.sh>
Co-authored-by: Jason Dellaluce <jasondellaluce@gmail.com>
Co-authored-by: Jason Dellaluce <jasondellaluce@gmail.com>
Perform the validate_rules action before the load_rules action. This
ensures that *only* the rules files named with -V arguments are
validated.
This fixes https://github.com/falcosecurity/falco/issues/2087.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
This updates the engine to comply and work properly with the newly-introduced
interface design.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
The filter_ruleset interface its implementation evt_type_index_ruleset
have been modified as follows:
- Only keep track of ruleset ids and not names. The falco engine will take
care of mapping easy-to-remember ruleset names to ruleset ids.
To emphasize this, use ruleset_id everywhere and not ruleset.
Also, make it non-optional.
- Have explicit separate functions to enable/disable rules, instead of a single enable() method combined with a boolean flag.
This does *not* change the falco_engine interface, which has
similar methods, to avoid breaking API changes.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
Co-authored-by: Mark Stemm <mark.stemm@gmail.com>
This also fixes a couple of bugs. With the current implementation, the multi-ruleset feature is broken with multiple sources.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
This interface will allow us to use different ruleset implementations inside the same engine.
The goal is to define API boundaries that will allow swapping the current evttype-index
ruleset implementation more easily. Key benefits include: smaller component with less responsibilities,
easier substituibility, more testable design, opportunity to adopt different index strategies
depending on the ruleset implementation.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
This means that when starting Falco passing to it a folder for its rules, it will properly manage
changes to any file inside the folders, plus any created/deleted file inside it.
Unified list of rules parsing, instead of having it done twice inside cmdline_options and configuration.
Instead, it is done only once, inside load_rules_files.
Signed-off-by: Federico Di Pierro <nierro92@gmail.com>
When adding an implied "in" comparison to an exception using the
single value form, add it to item, not items.
This fixes#1984.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
This include making a coherent use of const, remove private inheritance, and adding virtual destructors.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
This involves moving the code in falco_init() into individual files
below app_actions/. falco_init() simply calls app.run() now. When
app.run() returns false, print any erorr. When app.run() sets restart
to true, falco_init() is called again.
app.run() is still inside a catch block to catch any uncaught
exception.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Each file below app_actions/ defines some of the methods declared in
falco::app::application.
Any state that needs to be shared betweeen methods, or between the run
and teardown methods, resides in falco::app::application::state(), so
the moved code stays pretty much as-is, other than replacing stack
variables with member variables in app_state.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Changes to the falco::app::application object to support actions:
- All of the code that was in falco_init is now in methods of
application. (A later commit actually moves the code from falco_init
and into the split-up methods, this commit just declares them).
- Methods return an application::run_result object, which is a tuple
of success/errstr/proceed. proceed=false is used to short circuit
calling methods (think --help, --list, --support, etc.)
- application now has a run() method which runs the methods in an order
that honors any implicit dependencies (e.g. you have to init an
inspector before you open it, you have to do almost everything else
before processing events, etc.)
- There are a few methods devoted to teardown, they are always called
after the group of run methods are called.
- State that needs to be saved between methods, or saved between the
run and teardown functions, is in a
falco::app::application::state object.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Some objects used by falco (falco outputs, falco_formats, etc) were
using raw pointer references, which isn't great.
So convert use of raw pointers (originally passed from falco_init or
functions it called) with shared_ptr, as they are now held in
application state.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
The first warnings we support involve the unsafe comparisons with <NA>, which were present
in the legacy regression tests for PSPs.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
At the same time, this also simplifies the unit test cases by using the SCENARIO construct of catch2,
which allows sharing a setup phases between different unit tests, and removes a bunch of repeated LOC in our case.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
Once all rule files have been loaded, and all the rules have been compiled into filters and inserted in the engine rulesets, the loader definitions are maintained in memory without really being used. This commit adds a convenience method to clear the loader state and free-up some memory when engine consumers do not require such information in memory anymore.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
The rule_reader class is responsible of parsing the YAML ruleset text and of using the rule_loader
to store the new definition in the internal state. This is a first step towards separating the YAML
reading logic from the rule parsing one. Potentially, this will allow us to read rulesets from another
YAML library or from something different than YAML files too.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
The rule_loader is now simply responsible of collecting list/macro/rule definitions and then compiling them as falco_rules. The ruleset file reading code will be moved to another class
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
This is a porting of what we had inside the Lua codebase. This now handles the single responsibility
of gathering stats about rule-event matching, and of formatting them to print them to the user.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
The function implementation was removed, however it was still defined in the .h header. Moreover,
this will now be required in order to replace its lua equivalent.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
falco_engine::process_event gets called for every inspector event.
Profiling showed that std::map::find takes about 10% of
falco_engine::process_event, and that can easily improved by accessing
the source by index.
Signed-off-by: Angelo Puglisi <angelopuglisi86@gmail.com>
Both the parser.lua and compiler.lua modules are not necessary anymore, because all the logic related
to filter parsing and compilation is handled inside libsinsp now. Accordingly, they have been removed from
the lua-to-cpp.sh scripts. README.md and parse-smoke.sh have been removed since they are not needed anymore:
lpeg is not used by the project, and the smoke tests are implemented in libsisnsp unit test suite.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
The lua_filter_helper class is a simple Lua wrapper that can be used in the Lua rule loader to
parse/compile rule filters, and manipulate them to resolve/replace list and macro references.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
The Lua parser grabbed from libs chisels is not used anymore, as the compilation logic happen inside the new
filter parser of libsinsp.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
This is a first step towards porting the rule filter building logic that is currently implemented in Lua.
filter_macro_resolver uses the newly introduced AST constructs from libsinsp, and
allow manipulating filter ASTs to resolve/replace macro references. This is meant to be used
at boot time by the rule loader (which we still want to maintain implemented in Lua for now).
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
The Lua PEG parser is not longer needed, since we now use the new filter parser implemented
in libsinsp.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
Also skip macros with unknown sources. This matters primarily for
macros related to plugins that have a distinct event source.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
If a rule has an unknown source, *and* has exceptions, loading the
rule will result in an error and not skipping the rule. This is
because exceptions are also validated for unknown fields, and that
occurs before the current check for unknown sources.
The fix is to move the check for unknown sources as soon as the rules
object is read.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Update falco's main falco_init() to use a falco::app::application and
falco::app::cmdline_opts object instead of storing all its command
line state in stack variables.
The bulk of the removed code is in usage() (not needed as cxxopt's
help() is self-documenting.) and getopt_long() which is replaced by
app.init(argc, argv).
For the most part, this is simply replacing references to local
variables (e.g. "all_events") to the bound variable inside the
cmdline_opts object (e.g. app.copts().all_events).
There are a few cases where more complex logic was used (output
formats, initializing k8s/mesos with string pointers), and those
changes are still in falco_init().
For the most part, the monolithic parts of falco_init that involve
reading config files, creating the inspector, loading rules, etc are
still present. Those will be addressed in later changes.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
For the most part, replacing getopt() with cxxopts + falco application
had no effect on falco engine/config interfaces. However, there were a
few places where it was wasier to change the interface than add
middleware code that transformed from, for example, vectors to lists.
This commit has those changes.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Fill in an initial falco::app::cmdline_options class using cxxopts
library to hold options:
- falco::app::cmdline_options contains a cxxopts::Options object to
parse options and a cxxopts::ParseResult to hold the result.
- The only meaningful public method is parse() which parses argc/argv
and returns true/false + error.
- The parsed options are all public instance variables of the object
and generally use the same names of the corresponding variables in
the old falco_init(). These variables are all bound to the
corresponding command line option and are updated in parse().
- In a few cases, the command line option does not directly map to a
bound variable (e.g. -b to set buffer format, -p/-pk/-pc to set
extra formatting options, etc.) In these cases the option values are
read after parsing and update the public instance variable.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Add a notion of a falco application object. Eventually this will
replace the bulk of falco_init and contain methods to:
- Parse/validate command line options
- Parse/validate falco config
- Initialize prerequsites (inspector, falco engine, webserver, etc)
- Load plugins
- Load/validate rules
- Command/subcommand execution (e.g. --list/--list-fields, or
nothing specified to run "main" loop)
For now, it is only responsible for command line options handling,
which is stubbed out.
Currently, the only public methods are init() to initialize everything
and copts() to access command line options.
Command line options are held in a different class
falco::app::cmdline_opts. application::copts() returns a reference to
that object, which allows access to parsed command line options bound
to various public instance variables.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
This workaround an issue in libs, targeting Falco 0.31.0.
Signed-off-by: Federico Di Pierro <nierro92@gmail.com>
Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Jason Dellaluce <jasondellaluce@gmail.com>
When listing fields with -N (names only), also skip fields with the
EPF_TABLE_ONLY flag. (Skipping fields without -N is handled in libs,
in the as_string() method).
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
This fixes the parser introduced in https://github.com/falcosecurity/falco/pull/1792.
Now, nested fields such as `arr[1].subval` are supported, whereas the parser used
to recognize the `.` as an unexpected character.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
Instead of having .lua files external to the program responsible for
loading rules, embed the contents of those files into the executable
and load them as strings instead of as files:
Add a cmake custom command below userspace/engine/lua that calls a
bash script lua-to-cpp.sh to generate falco_engine_lua_files.{cpp,hh}
that are compiled into the falco engine library.
The script creates a .cpp file that has const char * symbols for each
file, as well as lists of files that should be loaded when the falco
engine is loaded. There are actually two lists:
- lua_module_strings: these are loaded and also added to the lua
runtime package.preload table, so they are available when lua code
require()s them.
- lua_code_strings: these are loaded *and* evaluated, so the functions
in them are availble to be called from C++.
This simplifies some of the falco_common methods, as there's no need
to keep track of a "main" lua file to load or paths from which the lua
loader should find files for modules, and there's no need to keep
track of an "alternate" lua directory that occurs for debug builds.
Also, there's no need to include any .lua files in the installed
packages, as they're built into the falco binary.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
This will distinguish it from rule_loader.lua, which is *not* a module
but lua code with functions that can be called directly.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Update json_event_filter_factory::get_fields() to add the new
info (shortdesc, data_type, tags) to field descriptions.
This allows for richer outputs when printing info on the fields.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
With the new implementation of list_fields(), the order of fields
changed slightly. So update the checksum.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Instead of having a falco-specific function to print field info, use
the built-in filter_fieldclass_info::as_string() instead. This is a
better implementation (displays addl info, has better wrapping, wider
output) and having a single implementation allows for consistent
outputs between falco and other potential programs that could use the libs.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Since now, the maximum depth supported to access config fields is two.
This adds support for accessing fields of arbitrary nesting depth.
A formal grammar has been explicited for the regular language representing
the field keys. The accessor methods have been updated accordingly.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
This is a change of direction from the current design, that imposes loading
the configuration from file only, and in the object constructor. Instead,
yaml_configuration objects can now be reused ad can load the YAML config
from either file or string. This also makes it easier to unit test this class.
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
Signed-off-by: Sai Arigeli <saiharisharigeli@gmail.com>
Return warnings after validation of rule exceptions
Signed-off-by: Sai Arigeli <saiharisharigeli@gmail.com>
Update FALCO_ENGINE_VERSION
Signed-off-by: Sai Arigeli <saiharisharigeli@gmail.com>
This allows defining rules that simply enable/disable already defined rules, like the following:
- rule: A rule enabled by default
enabled: false
- rule: A rule disabled by default
enabled: true
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
When MUSL_OPTIMIZED_BUILD is specified, falco is statically linked under
musl, and can't dlopen() files: see
https://inbox.vuxu.org/musl/20200423162406.GV11469@brightrain.aerifal.cx/T/
So skip listing/loading/testing plugins when MUSL_OPTIMIZED_BUILD is specified.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Update the falco binary to add support for plugins.
- Keep track of an "event source", which is initially "syscall" but
changes to the input plugin's source if an source plugin ends up being
loaded.
- New argument --list-plugins will return info on any loaded plugins,
using sinsp_plugin::plugin_infos.
- Create filter/formatter factories for plugins. This ensures that
filterchecks for syscalls are not used for plugins and vice versa.
- Use sinsp_plugin::register_plugin() to load each plugin found in
config. The first source plugin found (if any) calls
engine->add_source withthe source plugin's event source.
- If a second source plugin is found, exit with an error.
- Extractor plugins must be compatible with the event source (usually
the plugin event source, but could be "syscall"). If not, exit with
an error.
- Multiple Extractor plugins are allowed, but they can not have
overlapping compatible event sources. This is mostly to avoid
confusion, but we might change this later.
- After loading plugins, use engine is_plugin_compatible to ensure
that the plugin is compatible with any required_plugin_version blocks
in falco rules.
- Normally falco would log warnings if too many SCAP_TIMEOUT results
were received. These are more expected when using plugins, so only
log these warnings when using syscalls.
Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
The generic events support already handled most of this, with a
dedicated formatter factory for plugin sources. Just one missing
header include and change the logic slightly for json parsing.
Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Rules loading changes for plugins:
- parse required_engine_versions from yaml and pass up to rules
loader as a lua table as an additional return value from load_rules().
- c++ rules loader converts to map: plugin -> list of required plugin
versions
- support is_source_valid callback from lua, calls engine method. If
a source is not valid, skip any rules for that source and add a warning.
Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Mostly plugins are just handled as a new filter/formatter factory with
a new source based on the loaded input plugin, but there are a few
changes at the engine level:
- is_source_valid returns whether a filter/formatter factory exists
for a given source. Will be used by rules loaded to skip rules for
an unknown source.
- the falco engine now holds the required_plugin_version predicates
found in rules files and a method is_plugin_compatible returns whether
a plugin semver is compatible with the predicates in the rules
- Update the falco engine version and fields checksum for plugins
Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Add a cmake module "plugins" that does the following:
- Downloads/installs the plugins artifacts from a known tag
- Copies the resulting cloudtrail/json shared libraries to
CMAKE_CURRENT_BINARY_DIR/plugins
- Installs them to FALCO_SHARE_DIR/plugins
The default config will define the plugins but they will be disabled
by default.
Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
It took a while, but we remembered to finish moving the token_bucket
from falco engine to libs. There were 2 copies for a while.
This brings over one change to libs--to have an optional timer
function.
Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Update config code/default falco.yaml to add support for plugins:
- Update config parsing methods to support reading plugin config
objects in a list from yaml.
- The default config defines the cloudtrail/json plugins but does not
give them any actual config for init config/open
params (cloudtrail), or init config (json).
- load_plugins is empty so neither plugin is actually loaded by default.
Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>-
This makes the output of --list a bit more precise to only include
filter fields and not output fields.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
This step used to be done in the lua rule loading code, but now we can
get it directly from the filters, so do it in falco instead.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Update the lua side of rule loading to reflect other changes:
- install_filter renamed to create_filter_obj, and takes just a
lua_parser object created via falco_rules.create_lua_parser() and
uses a single lua callback "filter" instead of separate ones for
syscall/k8s_audit. It can return an error, including about
undefined fields
- is_defined_filter, which used to be local and based on the result of
sinsp_rule_utils.check_for_ignored_syscalls_events, is now a
lua_callback falco_rules.is_defined_field().
- Don't need to pass down sinsp_lua_parser/json_lua_parser now,
creating filters is handled via lua callbacks.
- Checking for ignored syscalls/events is now done in falco itself,
after loading rules.
- add_xxx_filter replaced by add_filter + source.
- Use is_format_valid instead of formats.formatter/formats.free_formatter.
- We don't need the functions in sinsp_rule_utils any longer, so
remove the file and don't import it.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Add a function is_defined_field(source, fldname) that returns whether
a field with name fldname exists for the given event source. This uses
the filter factory to create a filtercheck, and returns true if an
object was created.
This prevents having to push down the entire set of defined fields
before calling load_rules().
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Use the new falco engine interface with support for generic events
instead of event-specific process_xxx_event methods.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Update rules loader to be more general purpose by using factories and
the general purpose engine:
- A lua callback create_lua_parser creates a lua_parser with a filter
object of the right type. The lua parser can then iterate the AST
and populate the filter object.
- Like the falco engine, the rules loader is configured with a list of
factories, and add_filter is now general purpose, taking a source.
Given the fix in https://github.com/falcosecurity/libs/pull/72, there
isn't any need to pass down the entire set of sinsp event
types/syscalls and validate that all filter event types are
valid. That job is now handled by the sinsp filter parsing
code. add_filter now returns the number of event types used by the new
filter, and if that number is excessive the lua code will return a
warning.
Format handling is mostly not handled by the rules loader any more. As
a convienence, there's a new lua callback is_format_valid which takes
a source and output string and uses the right formatter factory to
create a formatter. As long as that doesn't throw an exception, the
format is valid.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Instead of having hard-coded support for syscall/k8s_audit events, use
the notions of filter factories/formatter factories to provide generic
support for events having a given source:
- Within the engine, maps m_filter_factories / m_rulesets /
m_format_factories map from a given source to something that can
create filters, hold filters, and create formatters for a given
source. The hard-coded sinsp_factory/json_factory objects are removed.
- The specific add_xxx_filter/process_xxx_event are general purpose
and take an event source.
- A new method create_formatter() takes a source/output format and
provides a shared_ptr to a formatter than can resolve format
strings. This is used by the falco outputs code.
- In falco main, create the syscall/k8s_audit filter and formatter
factories and pass them to the engine. Later, we might make this
configurable/selective.
With all of the above changes, the falco engine doesn't need a direct
inspector any longer, so remove it.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Make json_event_formatter a generic event formatter by inheriting from
gen_event_formatter and implementing its methods.
Most of the actual work is still done by resolve_format (previously
resolve_tokens, to avoid confusion with sinsp formatter, as it behaves
slightly differently).
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Take advantage of the changes in
https://github.com/falcosecurity/libs/pull/75 to have a
general-purpose way to list fields for a given event source.
in the engine, list_fields() now takes a source, iterates over filter
factories, and calls get_fields() for each factory, printing the results.
list_source_fields now calls the engine regardless of source.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Modify falco_formats to only be responsible for resolving a rule's
output string or coming up with a map of field name->field values from
a given output string.
It relies on the changes in
https://github.com/falcosecurity/libs/pull/77 to use generic
formatters for a given source.
Remove lua bindings to create a formatter/free a formatter. Those were
unused as of the changes in
https://github.com/falcosecurity/falco/pull/1451, so finally remove
them now.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Move the code that splits a json object into a list of k8s audit/json
events out of falco engine and into json_evt.
This, along with other changes, allows the falco engine to be more
general purpose and not directly tied to the notion of syscall vs k8s
audit events.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Modify rulesets to not keep track of the event types for a given set
filter. Instead, using the changes in
https://github.com/falcosecurity/libs/pull/74 event types are returned
directly by the filter.
Within each ruleset, there's a vector that maps from event number to
set of filters that are related to that event number. There's also a
general set of filters for all event types.
run() both indexes into the per-event vector as well as iterate over
the all event types set.
Also, used shared_ptr instead of direct pointers, which matches the
updated interface used by lua_parser. This simplifies the bookkeeping
a bit (no more delete when removing rulesets).
Given these changes, there's no need for a separate
falco_sinsp_ruleset class any longer, so remove it.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Also, print out the time of the last processed event in the output
fields of the notification.
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Falco uses a shared buffer between the kernel and userspace to receive
the events (eg., system call information) in userspace.
Anyways, the underlying libraries can also timeout for various reasons.
For example, there could have been issues while reading an event.
Or the particular event needs to be skipped.
Normally, it's very unlikely that Falco does not receive events consecutively.
Falco is able to detect such uncommon situation.
Here you can configure the maximum number of consecutive timeouts without an event
after which you want Falco to alert.
By default this value is set to 1000 consecutive timeouts without an event at all.
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
The rationale is that in case Falco obtains a consistent number of
consecutive timeouts (in a row) without a valid event, something is
going wrong.
This because, normally, the libs send timeouts to Falco (also) to signal events to discard.
In such cases, which are the majority of cases, `ev` exists and is not
`null`.
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
This fix has two major points in it:
- when `std::stoll` is used in parse_as_int64 handle all the exceptions it
can throw (https://en.cppreference.com/w/cpp/string/basic_string/stol)
- when `process_k8s_audit_event` an eventual exception in it does not
stop the webserver process. This is done by doing a catch all handle
outside it and by logging an error message to the caller as well as in
stderr
Signed-off-by: Lorenzo Fontana <lo@linux.com>
In some cases, it might be useful to know what falco engine version a
given falco binary supports. We already have a --support option that
prints info about the system, config, rules files, etc.
Add a engine_info object, with an engine_version property containing the
falco engine version in falco_engine_version.h. In the output, it looks
like this:
...
"engine_info": {
"engine_version": 8
},
...
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
We want users to continue using rules without having to use exceptions.
Exceptions are an additional feature for more advanced use-cases, having
a warning in there will mean that everyone now adds an empty exception
to avoid the warning.
Co-Authored-By: Leonardo Grasso <me@leonardograsso.com>
Signed-off-by: Lorenzo Fontana <lo@linux.com>
While testing, I found a case when creating a pod where:
1) the first container had no securityContext value
2) the second container had a security context with privileged=true
and this did not match the default rule Create Privileged Pod, when it
should match.
The rule Create Privileged Pod uses the field
ka.req.pod.containers.privileged, which in turn uses
json_event_filter_check::def_extract(). def_extract() iterates
over a set of json_pointers, potentially expanding arrays as they are
returned. Many k8s audit fields use this extract function.
For ka.req.pod.containers.privileged, the first json_pointer is
/requestObject/spec/containers to find the list of containers, and the
second is /securityContext/privileged to extract the privileged property
out of the securityContext object. What's returned is an array of
true/false noting if each container is privileged.
The problem is that def_extract() aborts when iterating over arrays if
extracting a pointer from an array can't be done.
In this case, the first pointer extracts the array of containers, and
then when iterating over the array of containers, the security context
pointer doesn't extract, causing the whole filter field to abort and
return ::no_value.
The fix is to not abort when iterating over arrays, but use ::no_value
for that array item's value instead. This allows def_extract() to
extract the privileged value out of the second container.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Falco won't properly parse a rule like this:
---
- rule: Some Rule
desc: Some Desc
condition: evt.type=execve and container.image.repository = 271931939120.dkr
output: Some output
priority: INFO
---
This is the error when validating the rules:
Tue Mar 30 12:00:40 2021: Validating rules file(s):
Tue Mar 30 12:00:40 2021: /home/mstemm/test.yaml
1 errors:
Compilation error when compiling "evt.type=execve and container.image.repository = 271931939120.dkr": 63: syntax error, unexpected 'dkr', expecting 'or', 'and'
The parsing of the string on the right hand side stops at the period
before the dkr. The dkr then doesn't match the grammar, resulting in the
error.
Looking at the parser implementation more closely, the problem is in the
definition of "Number":
---
- Number = C(V "Hex" + V "Float" + V "Int") / function(n)
return tonumber(n)
end,
---
Note that it stops after the number, but does not have any requirement
about what follows.
This changes the definition of number to require that what follows the
number is not an identifier character. With this change, values that are
only numbers are parsed as numbers, and values that start with numbers
don't match the Number definition and are parsed as BareStrings instead.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
When returning a rule_result struct, also include a set of field names
used by all exceptions for this rule. This may make building exception
values a bit easier.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
If a list:
- list: foo
items: [a, b, c]
Was referenced in another list:
- list: bar
items: [foo, d, e, f]
The first list would not be marked as used, when it should.
This avoids mistaken messages like "list xxx not refered to by any rule/macro/list"
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Support exceptions properties on rules as described in
https://github.com/falcosecurity/falco/pull/1376.
- When parsing rules, add an empty exceptions table if not specified.
- If exceptions are specified, they must contain names and lists of
fields, and optionally can contain lists of comps and lists of lists of
values.
- If comps are not specified, = is used.
- If a rule has exceptions and append:true, add values to the original rule's
exception values with the matching name.
- It's a warning but not an error to have exception values with a name
not matching any fields.
After loading all rules, build the exception condition string based on
any exceptions:
- If an exception has a single value for the "fields" property, values are
combined into a single set to build a condition string like "field
cmp (val1, val2, ...)".
- Otherwise, iterate through each rule's exception
values, finding the matching field names (field1, field2, ...) and
comp operators (cmp1, cmp2, ...), then
iterating over the list of field values (val1a, val1b, ...), (val2a,
val2b, ...), building up a string of the form:
and not ((field1 cmp1 val1a and field2 cmp2 val1b and ...) or
(field1 cmp1 val2a and field2 cmp2 val2b and ...)...
)"
- If a value is not already quoted and contains a space, quote it in the
string.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
When parsing a rules file, if a top level object is not one of the known
types rule, macro, list, required_engine_version, instead of failing
parsing, add a warning instead.
This adds some forwards-compatibility to rules files.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Add the notion of warnings when loading rules, which are printed if
verbose is true:
- load_rules now returns a tuple (success, required engine version,
error array, warnings array) instead of (true, required engine
version) or (false, error string)
- build_error/build_error_with_context now returns an array instead of
string value.
- warnings are combined across calls to load_rules_doc
- Current warnings include:
- a rule that contains an unknown filter
- a macro not referred to by any rule
- a list not referred to by any rule/macro/list
Any errors/warnings are concatenated into the exception if success was
false. Any errors/warnings will be printed if verbose is true.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Fixes#1269
Add two new fields in the version service for falco's engine version and
the checksum of all of the fields it understands.
This will require rebuilding/re-releasing all the clients.
Signed-off-by: Spencer Krum <nibz@spencerkrum.com>
This is needed because Luajit does not support many architectures
such as aarch64 and ppcle64.
Note: some operating systems, such as Alpine, already use moonjit as a dropin
replacement for luajit.
Signed-off-by: Lorenzo Fontana <fontanalorenz@gmail.com>
Previously, formatters were freed by LUA code when re-opening outputs.
Since now, outputs are not controlling anymore the falco_formats class (see #1412), we just free formatters only if were already initialized.
That is needed when the engine restarts (see #1446).
By doing so, we also ensure that correct inspector instance is set to the formatter cache.
Signed-off-by: Leonardo Grasso <me@leonardograsso.com>
This will be used by the static build to load lua files from
alternate directories that are not tied to the compile flags
Signed-off-by: Radu Andries <radu.andries@sysdig.com>
Do it only when not running with userspace instrumentation enabled and
the syscall input source is enabled (!disable_syscall)
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
means "auto")
The 0 ("auto") value sets the threadiness to the number of online cores
automatically.
Co-authored-by: Lorenzo Fontana <lo@linux.com>
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Currently, when calling enable_rule, the provided rule name pattern is a
substring match, that is if the rules file has a rule "My fantastic
rule", and you call engine->enable_rule("fantastic", true), the rule
will be enabled.
This can cause problems if one rule name is a complete subset of another
rule name e.g. rules "My rule" and "My rule is great", and calling
engine->enable_rule("My rule", true).
To allow for this case, add an alternate method enable_rule_exact() in
both default ruleset and ruleset variants. In this case, the rule name
must be an exact match.
In the underlying ruleset code, add a "match_exact" option to
falco_ruleset::enable() that denotes whether the substring is an exact
or substring match.
This doesn't change the default behavior of falco in any way, as the
existing calls still use enable_rule().
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
BAN_ALTERNATIVE is same as BAN but the message also provides an alternative
function that the user could use instead of the banned function.
Fixes#1035
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
These include:
* vsprintf()
* sprintf()
* strcat()
* strncat()
* strncpy()
* swprintf()
* vswprintf()
This also changes `userspace/falco/logger.cpp` to remove a `sprintf`
statement. The statement did not affect the codebase in any form so
it was simply removed rather than being substituted.
Fixes#1035
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
This defines certain functions as invalid tokens, i.e., when
compiled, the compiler throws an error.
Currently only `strcpy` is included as a banned function.
Fixes#788
Signed-off-by: Vaibhav <vrongmeal@gmail.com>
Callers aren't expected to catch execeptions and instead rely on the
bool return value to indicate whether or not the parsing was successful.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Currently, the json object POSTed to the /k8s_audit endpoint is assumed
to be an obect, with a "type" of either "Event" or "EventList". When the
K8s API Server POSTs events, it aggregates them into an EventList,
ensuring that there is always a single object.
However, we're going to add some intermediate tools that tail log files
and send them to the endpoint, and the easiest way to send a batch of
events is to pass them as a json array instead of a single object.
To properly handle this, modify parse_k8s_audit_event_json to also
handle a json array. For arrays, it iterates over the objects, calling
parse_k8s_audit_json recursively. This only iterates an initial top
level array to avoid excessive recursion/attacks involving degenerate
json objects with excessively nested arrays.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
In all extraction functions, always catch json type errors alongside
json out of range errors. Both cases result in not extracting any value
from the event.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
First of a handful of PRs to start clarifying the independence of Falco
I don't see any breaking changes here, just cosmetic changes.
Signed-off-by: Kris Nova <kris@nivenly.com>
The call to rule_loader.load_rules only returns 2 values, so only pop
two values from the stack. This fixes#906.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
As a part of the changes in
https://github.com/falcosecurity/falco/pull/826/, we added several
breaking changes to rules files like renaming/removing some filter
fields. This isn't ideal for customers who are using their own rules
files.
We shouldn't break older rules files in this way, so add some minimal
backwards compatibility which adds back the fields that were
removed *and* actually used in k8s_audit_rules.yaml. They have the same
functionality as before. One exception is
ka.req.binding.subject.has_name, which was only used in a single output
field for debugging and shouldn't have been in the rules file in the
first place. This always returns the string "N/A".
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Support the notion of a message for all fields in a single class, and
making sure it's wrapped as well as the other fields.
This is used to display a single message about how indexing working for
ka.* filter fields and what IDX_ALLOWED/IDX_NUMERIC/IDX_KEY means,
rather than repeating the same text over and over in every field.
The wrapping is handled by a function falco::utils::wrap_text.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Refactor how JSON event/k8s audit events extract values in two important
ways:
1. An event can now extract multiple values.
2. The extracted value is a class json_event_value instead of a simple
string.
The driver for 1. was that some filtercheck fields like
"ka.req.container.privileged" actually should extract multiple values,
as a pod can have multiple containers and it doesn't make sense to
summarize that down to a single value.
The driver for 2. is that by having an object represent a single
extracted value, you can also hold things like numbers e.g. ports, uids,
gids, etc. and ranges e.g. [0:3]. With an object, you can override
operators ==, <, etc. to do comparisons between the numbers and ranges,
or even set membership tests between extracted numbers and sets of
ranges.
This is really handy for a lot of new fields implemented as a part of
PSP support, where you end up having to check for overlaps between the
paths, images, ports, uids, etc in a K8s Audit Event and the acceptable
values, ranges, path prefixes enumerated in a PSP.
Implementing these changes also involve an overhaul of how aliases are
implemented. Instead of having an optional "formatting" function, where
arguments to the formatting function were expressed as text within the
index, define optional extraction and indexing functions. If an
extraction function is defined, it's responsible for taking the full
json object and calling add_extracted_value() to add values. There's a
default extraction function that uses a list of json_pointers with
automatic iteration over array values returned by a json pointer.
There's still a notion of filter fields supporting indexes--that's
simply handled within the default extraction or custom extraction
function. And for most fields, there won't be a need to write a custom
extraction function simply to implement indexing.
Within a json_event_filter_check object, instead of having a single
extracted value as a string, hold a vector of extracted json_event_value
objects (vector because order matters) and a set of json_event_value
objects (for set comparisons) as m_evalues. Values on the right hand
side of the expression are held as a set m_values.
json_event_filter_check::compare now supports IN/INTERSECTS as set
comparisons. It also supports PMATCH using path_prefix_search objects,
which simplifies checks like ka.req.pod.volumes.hostpath--now they can
be expressed as "ka.req.pod.volumes.hostpath intersects (/proc,
/var/run/docker.sock, /, /etc, /root)" instead of
"ka.req.volume.hostpath[/proc]=true or
ka.req.volume.hostpath[/root]=true or ...".
Define ~10 new filtercheck fields that extract pod properties like
hostIpc, readOnlyRootFilesystem, etc. that are relevant for PSP validation.
As a part of these changes, also clarify the names of filter fields
related to pods to always have a .pod in the name. Furthermore, fields
dealing with containers in a pod always have a .pod.containers prefix in
the name.
Finally, change the comparisons for existing k8s audit rules to use
"intersects" and/or "in" when appropriate instead of a single equality
comparison.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Related to the changes in https://github.com/draios/sysdig/pull/1501,
add support for an "intersects" operator that verifies if any of the
values in the rhs of an expression are found in the set of extracted
values.
For example:
(a,b,c) in (a,b) is false, but (a,b,c) intersects (a,b) is true.
The code that implements CO_INTERSECTS is in a different commit.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
I wasn't able to compile the dev branch with gcc 5.4 (e.g. not using the
builder), getting this error:
```
.../falco/userspace/falco/grpc_server.cpp:40:109: error: specialization of ‘template<class Request, class Response> void falco::grpc::request_stream_context<Request, Response>::start(falco::grpc::server*)’ in different namespace [-fpermissive]
void falco::grpc::request_stream_context<falco::output::request, falco::output::response>::start(server* srv)
^
In file included from .../falco/userspace/falco/grpc_server.cpp:26:0:
.../falco/userspace/falco/grpc_server.h:102:7: error: from definition of ‘template<class Request, class Response> void falco::grpc::request_stream_context<Request, Response>::start(falco::grpc::server*)’ [-fpermissive]
void start(server* srv);
```
It looks like gcc 5.4 doesn't handle a declaration with namespace blocks
but a definition with namespaces in the
function. https://gcc.gnu.org/bugzilla/show_bug.cgi?id=56480 has more
detail.
A workaround is to add `namespace falco {` and `namespace grpc {` around
the declarations.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
If this work as intended PR will automatically get the area labels depending on the files he modified.
In case the user wants it can still apply other areas manually, by slash command, or editing the PR template during the opening of the PR.
Signed-off-by: Leonardo Di Donato <leodidonato@gmail.com>
Properly parse multi-document yaml files e.g. blocks separated by
---. This is easily handled by lyaml itself--you just need to pass the
option all = true to yaml.load, and each document will be provided as a table.
This does break the table iteration a bit, so some more refactoring:
- Create a load_state table that holds context like the current
- document index, the required_engine_version, etc.
- Pull out the parts that parse a single document to load_rules_doc(),
which is given the table for a single document + load_state.
- Simplify get_orig_yaml_obj to just provide a single row index and
- return all rows from that point to the next blank line or line
starting with '-'
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Make additional improvements to display relevant context when validating
files. This handles cases where a macro/rule overwrites a prior rule.
- Instead of saving the index into the array of lines for each rule,
save the rule yaml itself, as a property 'context' for each object.
- When appending rules, the context of the base macro/rule and the
context of the appended rule/macro are concatenated.
- New functions get_orig_yaml_obj, build_error, and
build_error_with_context handle building the error string.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Fix a couple of small bugs when verifying macro/rule objects:
1) Yaml can have document separators "---", and those were mistakenly
being considered array items.
2) When reading macros and rules and using array position to find the
right document offset, the overall object order should be
used (e.g. this is the 5th object from the file) and not the array
position (e.g. this is the 3rd rule from the file).
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Given the compiler we currently use, you can't actually enable/disable
regexes in falco_engine::enable_rule using a regex pattern. The regex
either will fail to compile or will compile but not actually match
strings. This is noted on the c++11 compatibility notes for gcc 4.8.2:
https://gcc.gnu.org/onlinedocs/gcc-4.8.2/libstdc++/manual/manual/status.html#status.iso.2011.
The only use of using enable_rule was treating the regex pattern as a
substring match anyway, so we can change the engine to treat the pattern
as a substring.
So change the method/supporting sub-classes to note that the argument is
a substring match, and change falco itself to refer to substrings
instead of patterns.
This fixes https://github.com/falcosecurity/falco/issues/742.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Instead of relying on lua errors to pass back parse errors, pass back an
explicit true + required engine version or false + error message.
Also clean up the error message to display info + context on the
error. When the error related to yaml parsing, use the row number passed
back in lyaml's error string to print the specific line with the error.
When parsing rules/macros/lists, print the object being parsed alongside
the error.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
When parsing rules files with -V (validate), print info on the result of
loading the rules file to stdout. That way a caller can capture stdout
to pass along any rules parsing error.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
To speed up list expansion, instead of using regexes to replace a list
name with its contents, do string searches followed by examining the
preceding/following characters for the proper delimiter.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
We shouldn't need to clean up strings via a cleanup function and don't
need to do it via a bunch of string.gsub() functions.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Instead of iterating over the entire list of filters and doing pattern
matches against each defined filter, perform table lookups.
For filters that take arguments e.g. proc.aname[3] or evt.arg.xxx, split
the filtercheck string on bracket/dot and check the values against a
table.
There are now two tables of defined filters: defined_arg_filters and
defined_noarg_filters. Each filter is put into a table depending on
whether the filter takes an argument or not.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Json-related filtercheck fields supported indexing with brackets, but
when looking at the field descriptions you couldn't tell if a field
allowed an index, required an index, or did not allow an index.
This information was available, but it was a part of the protected
aliases map within the class.
Move this to the public field information so it can be used outside the
class.
Also add m_ prefixes for member names, now that the struct isn't
trivial.
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
Add more accurate tracking of the number of falco rules loaded per
ruleset, which are made available via the engine method
::num_rules_for_ruleset().
In the ruleset objects, keep track if a filter wrapper is actually
added/removed and if so increment/decrement the count.
For a while, falco has set the inspector drop mode to 1, which should
discard several classes of events that weren't necessary to use most
falco rules.
However, it was mistakenly being called before the inspector was opened,
which meant it wasn't actually doing anything.
Fix this by setting the dropping mode after the inspector open.
On some spot testing on a moderately loaded environment, this results in
a 30-40% drop in the number of system calls processed per second, and
should result in a nice boost in performance.
* Update engine fields checksum for fd.dev.*
New fields fd.dev.*, so updating the fields checksum.
* Print a message why the trace file can't be read.
At debug level only, but better than nothing.
* Adjust tests to match new container_started macro
Now that the container_started macro works either on the container event
or the first process being spawned in a container, we need to adjust the
counts for some rules to handle both cases.
* Add option to display times in ISO 8601 UTC
ISO 8601 time is useful when, say, running falco in a container, which
may have a different /etc/localtime than the host system.
A new config option time_format_iso_8601 controls whether log message
and event times are displayed in ISO 8601 in UTC or in local time. The
default is false (display times in local time).
This option is passed to logger init as well as outputs. For outputs it
eventually changes the time format field from %evt.time/%jevt.time to
%evt.time.iso8601/%jevt.time.iso8601.
Adding this field changes the falco engine version so increment it.
This depends on https://github.com/draios/sysdig/pull/1317.
* Unit test for ISO 8601 output
A unit test for ISO 8601 output ensures that both the log and event time
is in ISO 8601 format.
* Use ISO 8601 output by default in containers
Now that we have an option that controls iso 8601 output, use it by
default in containers. We do this by changing the value of
time_format_iso_8601 in falco.yaml in the container.
* Handle errors in strftime/asctime/gmtime
A placeholder "N/A" is used in log messages instead.
When creating syscall event drop alerts, instead of including just the
total and dropped event count, include all possible causes of drops as
well as whether bpf is enabled.
* Make stats file interval configurable
New argument --stats_interval=<msec> controls the interval at which
statistics are written to the stats file. The default is 5000 ms (5 sec)
which matches the prior hardcoded interval.
The stats interval is triggered via signals, so an interval below ~250ms
will probably interfere with falco's behavior.
* Add ability to emit general purpose messages
A new method falco_outputs::handle_msg allows emitting generic messages
that have a "rule", message, and output fields, but aren't exactly tied
to any event and aren't passed through an event formatter.
This allows falco to emit "events" based on internal checks like kernel
buffer overflow detection.
* Clean up newline handling for logging
Log messages from falco_logger::log may or may not have trailing
newlines. Handle both by always adding a newline to stderr logs and
always removing any newline from syslog logs.
* Add method to get sequence from subkey
New variant of get_sequence that allows fetching a list of items from a
key + subkey, for example:
key:
subkey:
- list
- items
- here
Both use a shared method get_sequence_from_node().
* Monitor syscall event drops + optional actions
Start actively monitoring the kernel buffer for syscall event drops,
which are visible in scap_stats.n_drops, and add the ability
to take actions when events are dropped. The -v (verbose) and
-s (stats filename) arguments also print out information on dropped
events, but they were only printed/logged without any actions.
In falco config you can specify one or more of the following actions to
take when falco notes system call drops:
- ignore (do nothing)
- log a critical message
- emit an "internal" falco alert. It looks like any other alert with a
time, "rule", message, and output fields but is not related to any
rule in falco_rules.yaml/other rules files.
- exit falco (the idea being that the restart would be monitored
elsewhere).
A new module syscall_event_drop_mgr is called for every event and
collects scap stats every second. If in the prior second there were
drops, perform_actions() handles the actions.
To prevent potential flooding in high drop rate environments, actions
are goverened by a token bucket with a rate of 1 actions per 30 seconds,
with a max burst of 10 seconds. We might tune this later based on
experience in busy environments.
This might be considered a fix for
https://github.com/falcosecurity/falco/issues/545. It doesn't
specifically flag falco rules alerts when there are drops, but does
make it easier to notice when there are drops.
* Add unit test for syscall event drop detection
Add unit tests for syscall event drop detection. First, add an optional
config option that artifically increments the drop count every
second. (This is only used for testing).
Then add test cases for each of the following:
- No dropped events: should not see any log messages or alerts.
- ignore action: should note the drops but not log messages or alert.
- log action: should only see log messages for the dropped events.
- alert action: should only see alerts for the dropped events.
- exit action: should see log message noting the dropped event and exit
with rc=1
A new trace file ping_sendto.scap has 10 seconds worth of events to
allow the periodic tracking of drops to kick in.
* Add support for container metaevent to detect container spawning
Create a new macro "container_started" to check both the old and
the new check.
Also, only look for execve exit events with vpid=1.
* Use TBB_INCLUDE_DIR for consistency w sysdig,agent
Previously it was a mix of TBB_INCLUDE and TBB_INCLUDE_DIR.
* Build using matching sysdig branch, if exists
* Expose required_engine_version when loading rules
When loading a rules file, have alternate methods that return the
required_engine_version. The existing methods remain unchanged and just
call the new methods with a dummy placeholder.
* Add --support argument to print support bundle
Add an argument --support that can be used as a single way to collect
necessary support information, including the falco version, config,
commandline, and all rules files.
There might be a big of extra structure to the rules files, as they
actually support an array of "variants", but we're thinking ahead to
cases where there might be a comprehensive library of rules files and
choices, so we're adding the extra structure.
* Add ability to print field names only
Add ability to print field names only instead of all information about
fields (description, etc) using -N cmdline option.
This will be used to add some versioning support steps that check for a
changed set of fields.
* Add an engine version that changes w/ filter flds
Add a method falco_engine::engine_version() that returns the current
engine version (e.g. set of supported fields, rules objects, operators,
etc.). It's defined in falco_engine_version.h, starts at 2 and should be
updated whenever a breaking change is made.
The most common reason for an engine change will be an update to the set
of filter fields. To make this easy to diagnose, add a build time check
that compares the sha256 output of "falco --list -N" against a value
that's embedded in falco_engine_version.h. A mismatch fails the build.
* Check engine version when loading rules
A rules file can now have a field "required_engine_version N". If
present, the number is compared to the falco engine version. If the
falco engine version is less, an error is thrown.
* Unit tests for engine versioning
Add a required version: 2 to one trace file to check the positive case
and add a new test that verifies that a too-new rules file won't be loaded.
* Rename falco test docker image
Rename sysdig/falco to falcosecurity/falco in unit tests.
* Don't pin falco_rules.yaml to an engine version
Currently, falco_rules.yaml is compatible with versions <= 0.13.1 other
than the required_engine_version object itself, so keep that line
commented out so users can use this rules file with older falco
versions.
We'll uncomment it with the first incompatible falco engine change.
* Allow SSL for k8s audit endpoint
Allow enabling SSL for the Kubernetes audit log web server. This
required adding two new configuration options: webserver.ssl_enabled and
webserver.ssl_certificate. To enable SSL add the below to the webserver
section of the falco.yaml config:
webserver:
enabled: true
listen_port: 8765s
k8s_audit_endpoint: /k8s_audit
ssl_enabled: true
ssl_certificate: /etc/falco/falco.pem
Note that the port number has an s appended to indicate SSL
for the port which is how civetweb expects SSL ports be denoted. We
could change this to dynamically add the s if ssl_enabled: true.
The ssl_certificate is a combination SSL Certificate and corresponding
key contained in a single file. You can generate a key/cert as follows:
$ openssl req -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem
$ cat certificate.pem key.pem > falco.pem
$ sudo cp falco.pem /etc/falco/falco.pem
fix ssl option handling
* Add notes on how to create ssl certificate
Add notes on how to create the ssl certificate to the config comments.
In the common case, falco doesn't generate much output, so it's
desirable to not buffer it in case you're tail -fing some logs.
So change the default for buffered outputs to false.
As per https://github.com/draios/sysdig/pull/1275, the gen_event class
mandate the implementation of two new methods.
This change aims to simplify the implementation of a generic event
processing infrastructure, that could handle both sinsp and json
events.
The -Wextra compile-time option will enable additional diagnostic
warnigns. The -Werror option will cause the compiler to treat warnings
as errors. This change adds a build time option,
BUILD_WARNINGS_AS_ERRORS, to conditionally enable those flags. Note
that depending on the compiler you're using, if you enable this option,
compilation may fail (some compiler version have additional warnings
that have not yet been resolved).
Testing with these options in place identified a destructor that was
throwing an exception. C++11 doesn't allow destructors to throw
exceptions, so those throw's would have resulted in calls to
terminate(). I replace them with an error log and a call to assert().
It's possible to call event_tags_for_ruleset/evttypes_for_ruleset for a
ruleset that hasn't been loaded. In this case, it's possible to go past
the end of the m_rulesets array.
After fixing that, it's also possible to go past the end of the
event_tags array in event_tags_for_ruleset().
So in both cases, check the index against the array size before
indexing.
* Add new json/webserver libs, embedded webserver
Add two new external libraries:
- nlohmann-json is a better json library that has stronger use of c++
features like type deduction, better conversion from stl structures,
etc. We'll use it to hold generic json objects instead of jsoncpp.
- civetweb is an embeddable webserver that will allow us to accept
posted json data.
New files webserver.{cpp,h} start an embedded webserver that listens for
POSTS on a configurable url and passes the json data to the falco
engine.
New falco config items are under webserver:
- enabled: true|false. Whether to start the embedded webserver or not.
- listen_port. Port that webserver listens on
- k8s_audit_endpoint: uri on which to accept POSTed k8s audit events.
(This commit doesn't compile entirely on its own, but we're grouping
these related changes into one commit for clarity).
* Don't use relative paths to find lua code
You can look directly below PROJECT_SOURCE_DIR.
* Reorganize compiler lua code
The lua compiler code is generic enough to work on more than just
sinsp-based rules, so move the parts of the compiler related to event
types and filterchecks out into a standalone lua file
sinsp_rule_utils.lua.
The checks for event types/filterchecks are now done from rule_loader,
and are dependent on a "source" attribute of the rule being
"sinsp". We'll be adding additional types of events next that come from
sources other than system calls.
* Manage separate syscall/k8s audit rulesets
Add the ability to manage separate sets of rules (syscall and
k8s_audit). Stop using the sinsp_evttype_filter object from the sysdig
repo, replacing it with falco_ruleset/falco_sinsp_ruleset from
ruleset.{cpp,h}. It has the same methods to add rules, associate them
with rulesets, and (for syscall) quickly find the relevant rules for a
given syscall/event type.
At the falco engine level, there are new parallel interfaces for both
types of rules (syscall and k8s_audit) to:
- add a rule: add_k8s_audit_filter/add_sinsp_filter
- match an event against rules, possibly returning a result:
process_sinsp_event/process_k8s_audit_event
At the rule loading level, the mechanics of creating filterchecks
objects is handled two factories (sinsp_filter_factory and
json_event_filter_factory), both of which are held by the engine.
* Handle multiple rule types when parsing rules
Modify the steps of parsing a rule's filter expression to handle
multiple types of rules. Notable changes:
- In the rule loader/ast traversal, pass a filter api object down,
which is passed back up in the lua parser api calls like nest(),
bool_op(), rel_expr(), etc.
- The filter api object is either the sinsp factory or k8s audit
factory, depending on the rule type.
- When the rule is complete, the complete filter is passed to the
engine using either add_sinsp_filter()/add_k8s_audit_filter().
* Add multiple output formatting types
Add support for multiple output formatters. Notable changes:
- The falco engine is passed along to falco_formats to gain access to
the engine's factories.
- When creating a formatter, the source of the rule is passed along
with the format string, which controls which kind of output formatter
is created.
Also clean up exception handling a bit so all lua callbacks catch all
exceptions and convert them into lua errors.
* Add support for json, k8s audit filter fields
With some corresponding changes in sysdig, you can now create general
purpose filter fields and events, which can be tied together with
nesting, expressions, and relational operators. The classes here
represent an instance of these fields devoted to generic json objects as
well as k8s audit events. Notable changes:
- json_event: holds a json object, used by all of the below
- json_event_filter_check: Has the ability to extract values out of a
json_event object and has the ability to define macros that associate
a field like "group.field" with a json pointer expression that
extracts a single property's value out of the json object. The basic
field definition also allows creating an index
e.g. group.field[index], where a std::function is responsible for
performing the indexing. This class has virtual void methods so it
must be overridden.
- jevt_filter_check: subclass of json_event_filter_check and defines
the following fields:
- jevt.time/jevt.rawtime: extracts the time from the underlying json object.
- jevt.value[<json pointer>]: general purpose way to extract any
json value out of the underlying object. <json pointer> is a json
pointer expression
- jevt.obj: Return the entire object, stringified.
- k8s_audit_filter_check: implements fields that extract values from
k8s audit events. Most of the implementation is in the form of macros
like ka.user.name, ka.uri, ka.target.name, etc. that just use json
pointers to extact the appropriate value from a k8s audit event. More
advanced fields like ka.uri.param, ka.req.container.image use
indexing to extract individual values out of maps or arrays.
- json_event_filter_factory: used by things like the lua parser api,
output formatter, etc to create the necessary objects and return
them.
- json_event_formatter: given a format string, create the necessary
fields that will be used to create a resolved string when given a
json_event object.
* Add ability to list fields
Similar to sysdig's -l option, add --list (<source>) to list the fields
supported by falco. With no source specified, will print all
fields. Source can be "syscall" for inspector fields e.g. what is
supported by sysdig, or "k8s_audit" to list fields supported only by the
k8s audit support in falco.
* Initial set of k8s audit rules
Add an initial set of k8s audit rules. They're broken into 3 classes of
rules:
- Suspicious activity: this includes things like:
- A disallowed k8s user performing an operation
- A disallowed container being used in a pod.
- A pod created with a privileged pod.
- A pod created with a sensitive mount.
- A pod using host networking
- Creating a NodePort Service
- A configmap containing private credentials
- A request being made by an unauthenticated user.
- Attach/exec to a pod. (We eventually want to also do privileged
pods, but that will require some state management that we don't
currently have).
- Creating a new namespace outside of an allowed set
- Creating a pod in either of the kube-system/kube-public namespaces
- Creating a serviceaccount in either of the kube-system/kube-public
namespaces
- Modifying any role starting with "system:"
- Creating a clusterrolebinding to the cluster-admin role
- Creating a role that wildcards verbs or resources
- Creating a role with writable permissions/pod exec permissions.
- Resource tracking. This includes noting when a deployment, service,
- configmap, cluster role, service account, etc are created or destroyed.
- Audit tracking: This tracks all audit events.
To support these rules, add macros/new indexing functions as needed to
support the required fields and ways to index the results.
* Add ability to read trace files of k8s audit evts
Expand the use of the -e flag to cover both .scap files containing
system calls as well as jsonl files containing k8s audit events:
If a trace file is specified, first try to read it using the
inspector. If that throws an exception, try to read the first line as
json. If both fail, return an error.
Based on the results of the open, the main loop either calls
do_inspect(), looping over system events, or
read_k8s_audit_trace_file(), reading each line as json and passing it to
the engine and outputs.
* Example showing how to enable k8s audit logs.
An example of how to enable k8s audit logging for minikube.
* Add unit tests for k8s audit support
Initial unit test support for k8s audit events. A new multiplex file
falco_k8s_audit_tests.yaml defines the tests. Traces (jsonl files) are
in trace_files/k8s_audit and new rules files are in
test/rules/k8s_audit.
Current test cases include:
- User outside allowed set
- Creating disallowed pod.
- Creating a pod explicitly on the allowed list
- Creating a pod w/ a privileged container (or second container), or a
pod with no privileged container.
- Creating a pod w/ a sensitive mount container (or second container), or a
pod with no sensitive mount.
- Cases for a trace w/o the relevant property + the container being
trusted, and hostnetwork tests.
- Tests that create a Service w/ and w/o a NodePort type.
- Tests for configmaps: tries each disallowed string, ensuring each is
detected, and the other has a configmap with no disallowed string,
ensuring it is not detected.
- The anonymous user creating a namespace.
- Tests for all kactivity rules e.g. those that create/delete
resources as compared to suspicious activity.
- Exec/Attach to Pod
- Creating a namespace outside of an allowed set
- Creating a pod/serviceaccount in kube-system/kube-public namespaces
- Deleting/modifying a system cluster role
- Creating a binding to the cluster-admin role
- Creating a cluster role binding that wildcards verbs or resources
- Creating a cluster role with write/pod exec privileges
* Don't manually install gcc 4.8
gcc 4.8 should already be installed by default on the vm we use for
travis.
Add a signal handler for SIGHUP that sets a global variable g_restart.
All the real execution of falco was already centralized in a standalone
function falco_init(), so simply exit on g_restart=true and call
falco_init() in a loop that restarts if g_restart is set to true.
Take care to not daemonize more than once and to reset the getopt index
to 1 on restart.
This fixes https://github.com/falcosecurity/falco/issues/432.
* Also add endswith to lua parser
Add endswith as a symbol so it can be parsed in filter expressions.
* Unit test for endswith support
Add a test case for endswith support, based on the filename ending with null.
* Use correct copyright years.
Also include the start year.
* Improve copyright notices.
Use the proper start year instead of just 2018.
Add the right owner Draios dba Sysdig.
Add copyright notices to some files that were missing them.
Replace references to GNU Public License to Apache license in:
- COPYING file
- README
- all source code below falco
- rules files
- rules and code below test directory
- code below falco directory
- entrypoint for docker containers (but not the Dockerfiles)
I didn't generally add copyright notices to all the examples files, as
they aren't core falco. If they did refer to the gpl I changed them to
apache.
* Proactively enable rules instead of only disabling
Previously, rules were enabled by default. Some performance improvements
in https://github.com/draios/sysdig/pull/1126 broke this, requiring that
each rule is explicitly enabled or disabled for a given ruleset.
So if enabled is true, explicitly enable the rule for the default ruleset.
* Get rid of shadowed res variable.
It was used both for the inspector loop and the falco result.
* Add ability to skip rules for unknown filters
Add the ability to skip a rule if its condition refers to a filtercheck
that doesn't exist. This allows defining a rules file that contains new
conditions that can still has limited backward compatibility with older
falco versions.
When compiling a filter, return a list of filtercheck names that are
present in the ast (which also includes filterchecks from any
macros). This set of filtercheck names is matched against the set of
filterchecks known to sinsp, expressed as lua patterns, and in the
global table defined_filters. If no match is found, the rule loader
throws an error.
The pattern changes slightly depending on whether the filter has
arguments or not. Two filters (proc.apid/proc.aname) can work with or
without arguments, so both styles of patterns are used.
If the rule has an attribute "skip-if-unknown-filter", the rule will be
skipped instead.
* Unit tests for skipping unknown filter
New unit test for skipping unknown filter. Test cases:
- A rule that refers to an unknown filter results in an error.
- A rule that refers to an unknown filter, but has
"skip-if-unknown-filter: true", can be read, but doesn't match any events.
- A rule that refers to an unknown filter, but has
"skip-if-unknown-filter: false", returns an error.
Also test the case of a filtercheck like evt.arg.xxx working properly
with the embedded patterns as well as proc.aname/apid which work both ways.
* Use better way to skip falco events
Use the new method falco_consider() to determine which events to
skip. This centralizes the logic in a single function. All events will
still be considered if falco was run with -A.
This depends on https://github.com/draios/sysdig/pull/1105.
* Add ability to specify -A flag in tests
test attribute all_events corresponds to the -A flag. Add for some tests
that would normally refer to skipped events.
* Only check whole rule names when matching counts
Tweak the regex so a rule my_great_rule doesn't pick up event counts for
a rule "great_rule: nnn".
* Add ability to skip evttype warnings for rules
A new attribute warn_evttypes, if present, suppresses printing warnings
related to a rule not matching any event type. Useful if you have a rule
where not including an event type is intentional.
* Add test for preserving rule order
Test the fix for https://github.com/draios/falco/issues/354. A rules
file has a event-specific rule first and a catchall rule second. Without
the changes in https://github.com/draios/sysdig/pull/1103, the first
rule does not match the event.
* Properly support syscalls in filter conditions
Syscalls have their own numbers but they weren't really handled within
falco. This meant that there wasn't a way to handle filters with
evt.type=xxx clauses where xxx was a value that didn't have a
corresponding event entry (like "madvise", for examples), or where a
syscall like open could also be done indirectly via syscall(__NR_open,
...).
First, add a new top-level global syscalls that maps from a string like
"madvise" to all the syscall nums for that id, just as we do for event
names/numbers.
In the compiler, when traversing the AST for evt.type=XXX or evt.type in
(XXX, ...) clauses, also try to match XXX against the global syscalls
table, and return any ids in a standalone table.
Also throw an error if an XXX doesn't match any event name or syscall name.
The syscall numbers are passed as an argument to sinsp_evttype_filter so
it can preindex the filters by syscall number.
This depends on https://github.com/draios/sysdig/pull/1100
* Add unit test for syscall support
This does a madvise, which doesn't have a ppm event type, both directly
and indirectly via syscall(__NR_madvise, ...), as well as an open
directly + indirectly. The corresponding rules file matches on madvise
and open.
The test ensures that both opens and both madvises are detected.
To further reduce falco's cpu usage, start setting the inspector in
"autodrop" mode with a sampling ratio of 1. When autodrop mode is
enabled, a second class of events (those having EF_ALWAYS_DROP in the
syscall table, or those syscalls that do not have specific handling in
the syscall table) are also excluded.
* Add ability to read rules files from directories
When the argument to -r <path> or an entry in falco.yaml's rules_file
list is a directory, read all files in the directory and add them to the
rules file list. The files in the directory are sorted alphabetically
before being added to the list.
The installed falco adds directories /etc/falco/rules.available and
/etc/falco/rules.d and moves /etc/falco/application_rules.yaml to
/etc/falco/rules.available. /etc/falco/rules.d is empty, but the idea is
that admins can symlink to /etc/falco/rules.available for applications
they want to enable.
This will make it easier to add application-specific rulesets that
admins can opt-in to.
* Unit test for reading rules from directory
Copy the rules/trace file from the test multiple_rules to a new test
rules_directory. The rules files are in rules/rules_dir/{000,001}*.yaml,
and the test uses a rules_file argument of rules_dir. Ensure that the
same events are detected.
* Reopen file/program outputs on SIGUSR1
When signaled with SIGUSR1, close and reopen file and program based
outputs. This is useful when combined with logrotate to rotate logs.
* Example logrotate config
Example logrotate config that relies on SIGUSR1 to rotate logs.
* Ensure options exist for all outputs
Options may not be provided for some outputs (like stdout), so create an
empty set of options in that case.
* Allow appending to skipped rules
If a rule has an append attribute but the original rule was skipped (due
to having lower priority than the configured priority), silently skip
the appending rule instead of returning an error.
* Unit test for appending to skipped rules
Unit test verifies fix for appending to skipped rules. One rules file
defines a rule with priority WARNING, a second rules file appends to
that rules file, and the configured priority is ERROR.
Ensures that falco rules without errors.
* Add option to exclude output property in json fmt
New falco.yaml option json_include_output_property controls where the
formatted string "output" is included in the json object when json
output is enabled. By default the string is included.
* Add tests for new json output option
New test sets json_include_output_property to false and then verifies
that the json output does *not* contain the surrounding text "Warning an
open...".
* Add the ability to validate multiple rules files
Allow multiple -V arguments just as we do with multiple -r arguments.
* With verbose output, print dangling macros/lists
Start tracking whether or not a given macro/list is actually used when
compiling the set of rules. Every macro/list has an attribute used,
which defaults to false and is set to true whenever it is referred to in
a macro/rule/list.
When run with -v, any macro/list that still has used=false results in a
warning message.
Also, it turns out the fix for
https://github.com/draios/falco/issues/197 wasn't being applied to
macros. Fix that.
The rules CMakeLists.txt, which controls the installation of the falco
rules files, was in the engine CMakeLists.txt, which meant that programs
that included the engine would also include rules files.
This may not always be desired, so move the rules CMakeLists.txt to the
main falco CMakeLists.txt instead.
A new falco.yaml option buffered_outputs, also controlled by
-U/--unbuffered, sets unbuffered outputs for the output methods. This is
especially useful with keep_alive files/programs where you want the
output right away.
Also add cleanup methods for the output channels that ensure output to
the file/program is flushed and closed.
Add the ability to keep file/program outputs open (i.e. writing to the
same open file/program for multiple notifications). A new option to the
file/program output "keep_alive", if true, keeps the file/program pipe
open across events.
This makes the need for unbuffered output aka
https://github.com/draios/falco/issues/211 more pressing. Will add that next.
When json output is set, add a sub-object called output_fields to the
json output that contains the individual templated fields from the
output string. Makes it easier to parse those fields.
This fixes https://github.com/draios/falco/issues/261.
These changes allow for a local rules file that will be preserved across
upgrades and allows the main rules file to be overwritten across upgrades.
- Move all config/rules files below /etc/falco/
- Add a "local rules" file /etc/falco/falco_rules.local.yaml. The intent
is that it contains modifications/deltas to the main rules file
/etc/falco/falco_rules.yaml. The main falco_rules.yaml should be
treated as immutable.
- All config files are flagged so they are not overwritten on upgrade.
- Change the handling of the config item "rules_file" in falco.yaml to
allow a list of files. By default, this list contains:
[/etc/falco/falco_rules.yaml, /etc/falco/falco_rules.local.yaml].
Also change rpm/debian packaging to ensure that the above files are
preserved across upgrades:
- Use relative paths for share/bin dirs. This ensures that when packaged
as rpms they won't be flagged as config files.
- Add CMAKE_INSTALL_PREFIX to FALCO_ENGINE_LUA_DIR now that it's relative.
- In debian packaging, flag
/etc/falco/{falco.yaml,falco_rules.yaml,falco_rules.local.yaml} as
conffiles. That way they are preserved across upgrades if modified.
- In rpm packaging when using cmake, any files installed with an
absolute path are automatically flagged as %config. The only files
directly installed are now the config files, so that addresses the problem.
Add CMAKE_INSTALL_PREFIX to lua dir.
Clean up the handling of priority levels within rules. It used to be a
mix of strings handled in various places. Now, in falco_common.h there's
a consistent type for priority-as-number as well as a list of
priority-as-string values. Priorities are passed around as numbers
instead of strings. It's still permissive about capitalization.
Also add the ability to load rules by severity. New falco
config option "priority=<val>"/-o priority=<val> specifies the minimum
priority level of rules that will be loaded.
Add unit tests for same. The test suppresses INFO notifications for a
rule/trace file combination that would otherwise generate them.
Add the ability to append to rules/macros, like we already do with
lists. For rules/macros, if the object has an append: true key, the
condition value is appended to the condition of an existing rule/macro
with the same name.
Like lists, it's an error to specify append: true without there being an
existing rule/macro.
Also add tests that test the same kind of things we did for lists:
- That append: true really does append
- That append: false overwrites the rule/macro
- That it's an error to append with a prior rule/macro existing.
List nodes can now have an 'append' key. If present and true, any values
in this list will be appended to the end of any existing list with the
same name.
It is an error to have a list with 'append' true that has a name that is
not an existing list.
When performing list substitution, only replace a list name when it is
surrounded by whitespace or expected punctuation characters. Lua
patterns don't have a notion of this-or-that patterns e.g. (^|abc), so
we have 3 versions of the substitution depending on whether he list name
occurs in the beginning, middle, or end of a string.
This fixes#197.
Also validate macros when they are parsed. Macros are also validated as
a part of rules being parsed, but it's possible to have an individual
rules file containing only macros, or a macro not explicitly tied to any
rule. In this case, it's useful to be able to check the macro to see if
it contains dangling macro references.
When parsing condition expressions, if the type of an ast node is
String (aka quoted string), don't trim whitespace from the value. This
ensures that conditions that want to match exact strings e.g. command
lines with leading/trailing spaces will work properly.
This fixes#253.
