Commit Graph

49 Commits

Author SHA1 Message Date
sai-arigeli
23706da75e Allow append of new exceptions to rules
Signed-off-by: Sai Arigeli <saiharisharigeli@gmail.com>

Return warnings after validation of rule exceptions

Signed-off-by: Sai Arigeli <saiharisharigeli@gmail.com>

Update FALCO_ENGINE_VERSION

Signed-off-by: Sai Arigeli <saiharisharigeli@gmail.com>
2021-11-18 09:11:20 +01:00
Jason Dellaluce
eec2f5062f update(userspace/engine): allow overwriting rules with enabled flag only
This allows defining rules that simply enable/disable already defined rules, like the following:
- rule: A rule enabled by default
  enabled: false
- rule: A rule disabled by default
  enabled: true

Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
2021-11-16 16:37:45 +01:00
Mark Stemm
e7d41f8166 Rules loading changes for plugins
Rules loading changes for plugins:

 - parse required_engine_versions from yaml and pass up to rules
   loader as a lua table as an additional return value from load_rules().
 - c++ rules loader converts to map: plugin -> list of required plugin
   versions
 - support is_source_valid callback from lua, calls engine method. If
   a source is not valid, skip any rules for that source and add a warning.

Co-authored-by: Leonardo Grasso <me@leonardograsso.com>
Co-authored-by: Loris Degioanni <loris@sysdig.com>
Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2021-11-12 18:27:59 +01:00
Mark Stemm
230c22b674 Update lua rule loading to reflect other changes
Update the lua side of rule loading to reflect other changes:

- install_filter renamed to create_filter_obj, and takes just a
  lua_parser object created via falco_rules.create_lua_parser() and
  uses a single lua callback "filter" instead of separate ones for
  syscall/k8s_audit. It can return an error, including about
  undefined fields

- is_defined_filter, which used to be local and based on the result of
  sinsp_rule_utils.check_for_ignored_syscalls_events, is now a
  lua_callback falco_rules.is_defined_field().

- Don't need to pass down sinsp_lua_parser/json_lua_parser now,
  creating filters is handled via lua callbacks.

- Checking for ignored syscalls/events is now done in falco itself,
  after loading rules.

- add_xxx_filter replaced by add_filter + source.

- Use is_format_valid instead of formats.formatter/formats.free_formatter.

- We don't need the functions in sinsp_rule_utils any longer, so
  remove the file and don't import it.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2021-10-12 17:59:38 +02:00
Jason Dellaluce
ca66b84e5a new(grpc): Add tags to outputs service
Signed-off-by: Jason Dellaluce <jasondellaluce@gmail.com>
2021-09-21 18:59:10 +02:00
Lorenzo Fontana
bd562a1ed9 update(userspace/engine): remove warnings for missing exceptions
We want users to continue using rules without having to use exceptions.
Exceptions are an additional feature for more advanced use-cases, having
a warning in there will mean that everyone now adds an empty exception
to avoid the warning.

Co-Authored-By: Leonardo Grasso <me@leonardograsso.com>
Signed-off-by: Lorenzo Fontana <lo@linux.com>
2021-04-09 18:17:11 +02:00
Mark Stemm
8c4040b610 Also include all exception fields in rule_result
When returning a rule_result struct, also include a set of field names
used by all exceptions for this rule. This may make building exception
values a bit easier.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2021-01-19 10:37:55 +01:00
Mark Stemm
e6b0d2697f Use the right not equals operator.
Fix typo, "!" should be "!=".

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2021-01-19 10:37:55 +01:00
Mark Stemm
91bfa379ce Properly note lists in other lists as used
If a list:

- list: foo
  items: [a, b, c]

Was referenced in another list:

- list: bar
  items: [foo, d, e, f]

The first list would not be marked as used, when it should.

This avoids mistaken messages like "list xxx not refered to by any rule/macro/list"

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2021-01-19 10:37:55 +01:00
Mark Stemm
a582599778 Support exceptions properties on rules
Support exceptions properties on rules as described in
https://github.com/falcosecurity/falco/pull/1376.

- When parsing rules, add an empty exceptions table if not specified.
- If exceptions are specified, they must contain names and lists of
  fields, and optionally can contain lists of comps and lists of lists of
  values.
- If comps are not specified, = is used.
- If a rule has exceptions and append:true, add values to the original rule's
  exception values with the matching name.
- It's a warning but not an error to have exception values with a name
  not matching any fields.

After loading all rules, build the exception condition string based on
any exceptions:

- If an exception has a single value for the "fields" property, values are
  combined into a single set to build a condition string like "field
  cmp (val1, val2, ...)".
- Otherwise, iterate through each rule's exception
  values, finding the matching field names (field1, field2, ...) and
  comp operators (cmp1, cmp2, ...), then
  iterating over the list of field values (val1a, val1b, ...), (val2a,
  val2b, ...), building up a string of the form:
    and not ((field1 cmp1 val1a and field2 cmp2 val1b and ...) or
              (field1 cmp1 val2a and field2 cmp2 val2b and ...)...
	     )"
- If a value is not already quoted and contains a space, quote it in the
  string.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2021-01-19 10:37:55 +01:00
Mark Stemm
4f192e89fa Allow unknown top level objs as warnings
When parsing a rules file, if a top level object is not one of the known
types rule, macro, list, required_engine_version, instead of failing
parsing, add a warning instead.

This adds some forwards-compatibility to rules files.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2021-01-19 10:37:55 +01:00
Mark Stemm
07abb89f36 Pass back warnings when loading rules
Add the notion of warnings when loading rules, which are printed if
verbose is true:

 - load_rules now returns a tuple (success, required engine version,
   error array, warnings array) instead of (true, required engine
   version) or (false, error string)
 - build_error/build_error_with_context now returns an array instead of
   string value.
 - warnings are combined across calls to load_rules_doc
 - Current warnings include:
   - a rule that contains an unknown filter
   - a macro not referred to by any rule
   - a list not referred to by any rule/macro/list

Any errors/warnings are concatenated into the exception if success was
false. Any errors/warnings will be printed if verbose is true.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2021-01-19 10:37:55 +01:00
Mark Stemm
b6fec781b7 Add support for parsing "intersects" operator
Related to the changes in https://github.com/draios/sysdig/pull/1501,
add support for an "intersects" operator that verifies if any of the
values in the rhs of an expression are found in the set of extracted
values.

For example:

  (a,b,c) in (a,b) is false, but (a,b,c) intersects (a,b) is true.

The code that implements CO_INTERSECTS is in a different commit.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2019-10-15 19:45:31 +02:00
Lorenzo Fontana
c76518c681 update: license headers
Co-Authored-By: Leonardo Di Donato <leodidonato@gmail.com>
Signed-off-by: Lorenzo Fontana <lo@linux.com>
2019-10-08 16:02:26 +02:00
Mark Stemm
76f64f5d79 Properly parse multi-document yaml files
Properly parse multi-document yaml files e.g. blocks separated by
---. This is easily handled by lyaml itself--you just need to pass the
option all = true to yaml.load, and each document will be provided as a table.

This does break the table iteration a bit, so some more refactoring:

 - Create a load_state table that holds context like the current
 - document index, the required_engine_version, etc.
 - Pull out the parts that parse a single document to load_rules_doc(),
   which is given the table for a single document + load_state.
 - Simplify get_orig_yaml_obj to just provide a single row index and
 - return all rows from that point to the next blank line or line
   starting with '-'

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2019-08-02 11:01:59 -07:00
Mark Stemm
1f0065e4b1 Further improvements when displaying contexts
Make additional improvements to display relevant context when validating
files. This handles cases where a macro/rule overwrites a prior rule.

 - Instead of saving the index into the array of lines for each rule,
   save the rule yaml itself, as a property 'context' for each object.

 - When appending rules, the context of the base macro/rule and the
   context of the appended rule/macro are concatenated.

 - New functions get_orig_yaml_obj, build_error, and
   build_error_with_context handle building the error string.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2019-07-31 11:22:38 +02:00
Mark Stemm
46b1a3c841 Fix bugs when verifying macro/rule objects.
Fix a couple of small bugs when verifying macro/rule objects:

1) Yaml can have document separators "---", and those were mistakenly
being considered array items.

2) When reading macros and rules and using array position to find the
right document offset, the overall object order should be
used (e.g. this is the 5th object from the file) and not the array
position (e.g. this is the 3rd rule from the file).

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2019-07-30 15:56:04 +02:00
Mark Stemm
40e3e21391 Allow all lowercase priorities
Just being tolerant given that the comparison used to be entirely
case-insensitive.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2019-07-24 13:05:17 +02:00
Mark Stemm
1711ed0a2e Pass back explicit errors in load_rules()
Instead of relying on lua errors to pass back parse errors, pass back an
explicit true + required engine version or false + error message.

Also clean up the error message to display info + context on the
error. When the error related to yaml parsing, use the row number passed
back in lyaml's error string to print the specific line with the error.

When parsing rules/macros/lists, print the object being parsed alongside
the error.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2019-07-11 11:24:22 -07:00
Mark Stemm
52a44c171c Look up priorities using a table
This is faster than iteration + string case changes.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2019-07-05 15:29:26 -07:00
Mark Stemm
047f12d0f6 More efficient searches for defined filters
Instead of iterating over the entire list of filters and doing pattern
matches against each defined filter, perform table lookups.

For filters that take arguments e.g. proc.aname[3] or evt.arg.xxx, split
the filtercheck string on bracket/dot and check the values against a
table.

There are now two tables of defined filters: defined_arg_filters and
defined_noarg_filters. Each filter is put into a table depending on
whether the filter takes an argument or not.

Signed-off-by: Mark Stemm <mark.stemm@gmail.com>
2019-07-05 15:29:26 -07:00
Mark Stemm
5e9bbd139c Add support bundle (#517)
* Expose required_engine_version when loading rules

When loading a rules file, have alternate methods that return the
required_engine_version. The existing methods remain unchanged and just
call the new methods with a dummy placeholder.

* Add --support argument to print support bundle

Add an argument --support that can be used as a single way to collect
necessary support information, including the falco version, config,
commandline, and all rules files.

There might be a big of extra structure to the rules files, as they
actually support an array of "variants", but we're thinking ahead to
cases where there might be a comprehensive library of rules files and
choices, so we're adding the extra structure.
2019-02-06 16:36:33 -08:00
Mark Stemm
513cf2ed8b Rules versioning (#492)
* Add ability to print field names only

Add ability to print field names only instead of all information about
fields (description, etc) using -N cmdline option.

This will be used to add some versioning support steps that check for a
changed set of fields.

* Add an engine version that changes w/ filter flds

Add a method falco_engine::engine_version() that returns the current
engine version (e.g. set of supported fields, rules objects, operators,
etc.). It's defined in falco_engine_version.h, starts at 2 and should be
updated whenever a breaking change is made.

The most common reason for an engine change will be an update to the set
of filter fields. To make this easy to diagnose, add a build time check
that compares the sha256 output of "falco --list -N" against a value
that's embedded in falco_engine_version.h. A mismatch fails the build.

* Check engine version when loading rules

A rules file can now have a field "required_engine_version N". If
present, the number is compared to the falco engine version. If the
falco engine version is less, an error is thrown.

* Unit tests for engine versioning

Add a required version: 2 to one trace file to check the positive case
and add a new test that verifies that a too-new rules file won't be loaded.

* Rename falco test docker image

Rename sysdig/falco to falcosecurity/falco in unit tests.

* Don't pin falco_rules.yaml to an engine version

Currently, falco_rules.yaml is compatible with versions <= 0.13.1 other
than the required_engine_version object itself, so keep that line
commented out so users can use this rules file with older falco
versions.

We'll uncomment it with the first incompatible falco engine change.
2019-01-29 12:43:15 -08:00
Mark Stemm
5e5742f87d Only have -pc/-pk apply to syscall rules (#495)
Currently, -pc/-pk results in extra fields added to the output field of
all rules. They should only be added for syscall rules.
2019-01-14 10:00:43 -08:00
Mark Stemm
1f28f85bdf K8s audit evts (#450)
* Add new json/webserver libs, embedded webserver

Add two new external libraries:

 - nlohmann-json is a better json library that has stronger use of c++
   features like type deduction, better conversion from stl structures,
   etc. We'll use it to hold generic json objects instead of jsoncpp.

 - civetweb is an embeddable webserver that will allow us to accept
   posted json data.

New files webserver.{cpp,h} start an embedded webserver that listens for
POSTS on a configurable url and passes the json data to the falco
engine.

New falco config items are under webserver:
  - enabled: true|false. Whether to start the embedded webserver or not.
  - listen_port. Port that webserver listens on
  - k8s_audit_endpoint: uri on which to accept POSTed k8s audit events.

(This commit doesn't compile entirely on its own, but we're grouping
these related changes into one commit for clarity).

* Don't use relative paths to find lua code

You can look directly below PROJECT_SOURCE_DIR.

* Reorganize compiler lua code

The lua compiler code is generic enough to work on more than just
sinsp-based rules, so move the parts of the compiler related to event
types and filterchecks out into a standalone lua file
sinsp_rule_utils.lua.

The checks for event types/filterchecks are now done from rule_loader,
and are dependent on a "source" attribute of the rule being
"sinsp". We'll be adding additional types of events next that come from
sources other than system calls.

* Manage separate syscall/k8s audit rulesets

Add the ability to manage separate sets of rules (syscall and
k8s_audit). Stop using the sinsp_evttype_filter object from the sysdig
repo, replacing it with falco_ruleset/falco_sinsp_ruleset from
ruleset.{cpp,h}. It has the same methods to add rules, associate them
with rulesets, and (for syscall) quickly find the relevant rules for a
given syscall/event type.

At the falco engine level, there are new parallel interfaces for both
types of rules (syscall and k8s_audit) to:
  - add a rule: add_k8s_audit_filter/add_sinsp_filter
  - match an event against rules, possibly returning a result:
    process_sinsp_event/process_k8s_audit_event

At the rule loading level, the mechanics of creating filterchecks
objects is handled two factories (sinsp_filter_factory and
json_event_filter_factory), both of which are held by the engine.

* Handle multiple rule types when parsing rules

Modify the steps of parsing a rule's filter expression to handle
multiple types of rules. Notable changes:

 - In the rule loader/ast traversal, pass a filter api object down,
   which is passed back up in the lua parser api calls like nest(),
   bool_op(), rel_expr(), etc.
 - The filter api object is either the sinsp factory or k8s audit
   factory, depending on the rule type.
 - When the rule is complete, the complete filter is passed to the
   engine using either add_sinsp_filter()/add_k8s_audit_filter().

* Add multiple output formatting types

Add support for multiple output formatters. Notable changes:

 - The falco engine is passed along to falco_formats to gain access to
   the engine's factories.
 - When creating a formatter, the source of the rule is passed along
   with the format string, which controls which kind of output formatter
   is created.

Also clean up exception handling a bit so all lua callbacks catch all
exceptions and convert them into lua errors.

* Add support for json, k8s audit filter fields

With some corresponding changes in sysdig, you can now create general
purpose filter fields and events, which can be tied together with
nesting, expressions, and relational operators. The classes here
represent an instance of these fields devoted to generic json objects as
well as k8s audit events. Notable changes:

 - json_event: holds a json object, used by all of the below

 - json_event_filter_check: Has the ability to extract values out of a
   json_event object and has the ability to define macros that associate
   a field like "group.field" with a json pointer expression that
   extracts a single property's value out of the json object. The basic
   field definition also allows creating an index
   e.g. group.field[index], where a std::function is responsible for
   performing the indexing. This class has virtual void methods so it
   must be overridden.

 - jevt_filter_check: subclass of json_event_filter_check and defines
   the following fields:
     - jevt.time/jevt.rawtime: extracts the time from the underlying json object.
     - jevt.value[<json pointer>]: general purpose way to extract any
       json value out of the underlying object. <json pointer> is a json
       pointer expression
     - jevt.obj: Return the entire object, stringified.

 - k8s_audit_filter_check: implements fields that extract values from
   k8s audit events. Most of the implementation is in the form of macros
   like ka.user.name, ka.uri, ka.target.name, etc. that just use json
   pointers to extact the appropriate value from a k8s audit event. More
   advanced fields like ka.uri.param, ka.req.container.image use
   indexing to extract individual values out of maps or arrays.

 - json_event_filter_factory: used by things like the lua parser api,
   output formatter, etc to create the necessary objects and return
   them.

  - json_event_formatter: given a format string, create the necessary
    fields that will be used to create a resolved string when given a
    json_event object.

* Add ability to list fields

Similar to sysdig's -l option, add --list (<source>) to list the fields
supported by falco. With no source specified, will print all
fields. Source can be "syscall" for inspector fields e.g. what is
supported by sysdig, or "k8s_audit" to list fields supported only by the
k8s audit support in falco.

* Initial set of k8s audit rules

Add an initial set of k8s audit rules. They're broken into 3 classes of
rules:

 - Suspicious activity: this includes things like:
    - A disallowed k8s user performing an operation
    - A disallowed container being used in a pod.
    - A pod created with a privileged pod.
    - A pod created with a sensitive mount.
    - A pod using host networking
    - Creating a NodePort Service
    - A configmap containing private credentials
    - A request being made by an unauthenticated user.
    - Attach/exec to a pod. (We eventually want to also do privileged
      pods, but that will require some state management that we don't
      currently have).
    - Creating a new namespace outside of an allowed set
    - Creating a pod in either of the kube-system/kube-public namespaces
    - Creating a serviceaccount in either of the kube-system/kube-public
      namespaces
    - Modifying any role starting with "system:"
    - Creating a clusterrolebinding to the cluster-admin role
    - Creating a role that wildcards verbs or resources
    - Creating a role with writable permissions/pod exec permissions.
 - Resource tracking. This includes noting when a deployment, service,
    - configmap, cluster role, service account, etc are created or destroyed.
 - Audit tracking: This tracks all audit events.

To support these rules, add macros/new indexing functions as needed to
support the required fields and ways to index the results.

* Add ability to read trace files of k8s audit evts

Expand the use of the -e flag to cover both .scap files containing
system calls as well as jsonl files containing k8s audit events:

If a trace file is specified, first try to read it using the
inspector. If that throws an exception, try to read the first line as
json. If both fail, return an error.

Based on the results of the open, the main loop either calls
do_inspect(), looping over system events, or
read_k8s_audit_trace_file(), reading each line as json and passing it to
the engine and outputs.

* Example showing how to enable k8s audit logs.

An example of how to enable k8s audit logging for minikube.

* Add unit tests for k8s audit support

Initial unit test support for k8s audit events. A new multiplex file
falco_k8s_audit_tests.yaml defines the tests. Traces (jsonl files) are
in trace_files/k8s_audit and new rules files are in
test/rules/k8s_audit.

Current test cases include:

- User outside allowed set
- Creating disallowed pod.
- Creating a pod explicitly on the allowed list
- Creating a pod w/ a privileged container (or second container), or a
  pod with no privileged container.
- Creating a pod w/ a sensitive mount container (or second container), or a
  pod with no sensitive mount.
- Cases for a trace w/o the relevant property + the container being
  trusted, and hostnetwork tests.
- Tests that create a Service w/ and w/o a NodePort type.
- Tests for configmaps: tries each disallowed string, ensuring each is
  detected, and the other has a configmap with no disallowed string,
  ensuring it is not detected.
- The anonymous user creating a namespace.
- Tests for all kactivity rules e.g. those that create/delete
  resources as compared to suspicious activity.
- Exec/Attach to Pod
- Creating a namespace outside of an allowed set
- Creating a pod/serviceaccount in kube-system/kube-public namespaces
- Deleting/modifying a system cluster role
- Creating a binding to the cluster-admin role
- Creating a cluster role binding that wildcards verbs or resources
- Creating a cluster role with write/pod exec privileges

* Don't manually install gcc 4.8

gcc 4.8 should already be installed by default on the vm we use for
travis.
2018-11-09 10:15:39 -08:00
Mark Stemm
6445cdb950 Better copyright notices (#426)
* Use correct copyright years.

Also include the start year.

* Improve copyright notices.

Use the proper start year instead of just 2018.

Add the right owner Draios dba Sysdig.

Add copyright notices to some files that were missing them.
2018-09-26 19:49:19 -07:00
Mark Stemm
2352b96d6b Change license to Apache 2.0 (#419)
Replace references to GNU Public License to Apache license in:

 - COPYING file
 - README
 - all source code below falco
 - rules files
 - rules and code below test directory
 - code below falco directory
 - entrypoint for docker containers (but not the Dockerfiles)

I didn't generally add copyright notices to all the examples files, as
they aren't core falco. If they did refer to the gpl I changed them to
apache.
2018-09-20 11:47:10 -07:00
Mark Stemm
70f768d9ea Enable all rules (#379)
* Proactively enable rules instead of only disabling

Previously, rules were enabled by default. Some performance improvements
in https://github.com/draios/sysdig/pull/1126 broke this, requiring that
each rule is explicitly enabled or disabled for a given ruleset.

So if enabled is true, explicitly enable the rule for the default ruleset.

* Get rid of shadowed res variable.

It was used both for the inspector loop and the falco result.
2018-06-07 17:16:30 -07:00
Mark Stemm
512a36dfe1 Conditional rules (#364)
* Add ability to skip rules for unknown filters

Add the ability to skip a rule if its condition refers to a filtercheck
that doesn't exist. This allows defining a rules file that contains new
conditions that can still has limited backward compatibility with older
falco versions.

When compiling a filter, return a list of filtercheck names that are
present in the ast (which also includes filterchecks from any
macros). This set of filtercheck names is matched against the set of
filterchecks known to sinsp, expressed as lua patterns, and in the
global table defined_filters. If no match is found, the rule loader
throws an error.

The pattern changes slightly depending on whether the filter has
arguments or not. Two filters (proc.apid/proc.aname) can work with or
without arguments, so both styles of patterns are used.

If the rule has an attribute "skip-if-unknown-filter", the rule will be
skipped instead.

* Unit tests for skipping unknown filter

New unit test for skipping unknown filter. Test cases:

 - A rule that refers to an unknown filter results in an error.
 - A rule that refers to an unknown filter, but has
   "skip-if-unknown-filter: true", can be read, but doesn't match any events.
 - A rule that refers to an unknown filter, but has
   "skip-if-unknown-filter: false", returns an error.

Also test the case of a filtercheck like evt.arg.xxx working properly
with the embedded patterns as well as proc.aname/apid which work both ways.
2018-05-03 14:24:32 -07:00
Mark Stemm
e922a849a9 Add tests catchall order (#355)
* Only check whole rule names when matching counts

Tweak the regex so a rule my_great_rule doesn't pick up event counts for
a rule "great_rule: nnn".

* Add ability to skip evttype warnings for rules

A new attribute warn_evttypes, if present, suppresses printing warnings
related to a rule not matching any event type. Useful if you have a rule
where not including an event type is intentional.

* Add test for preserving rule order

Test the fix for https://github.com/draios/falco/issues/354. A rules
file has a event-specific rule first and a catchall rule second. Without
the changes in https://github.com/draios/sysdig/pull/1103, the first
rule does not match the event.
2018-04-19 09:31:20 -07:00
Mark Stemm
ac190ca457 Properly support syscalls in filter conditions (#352)
* Properly support syscalls in filter conditions

Syscalls have their own numbers but they weren't really handled within
falco.  This meant that there wasn't a way to handle filters with
evt.type=xxx clauses where xxx was a value that didn't have a
corresponding event entry (like "madvise", for examples), or where a
syscall like open could also be done indirectly via syscall(__NR_open,
...).

First, add a new top-level global syscalls that maps from a string like
"madvise" to all the syscall nums for that id, just as we do for event
names/numbers.

In the compiler, when traversing the AST for evt.type=XXX or evt.type in
(XXX, ...) clauses, also try to match XXX against the global syscalls
table, and return any ids in a standalone table.

Also throw an error if an XXX doesn't match any event name or syscall name.

The syscall numbers are passed as an argument to sinsp_evttype_filter so
it can preindex the filters by syscall number.

This depends on https://github.com/draios/sysdig/pull/1100

* Add unit test for syscall support

This does a madvise, which doesn't have a ppm event type, both directly
and indirectly via syscall(__NR_madvise, ...), as well as an open
directly + indirectly. The corresponding rules file matches on madvise
and open.

The test ensures that both opens and both madvises are detected.
2018-04-17 17:14:45 -07:00
Mark Stemm
a5daf8b058 Allow append skipped rules (#346)
* Allow appending to skipped rules

If a rule has an append attribute but the original rule was skipped (due
to having lower priority than the configured priority), silently skip
the appending rule instead of returning an error.

* Unit test for appending to skipped rules

Unit test verifies fix for appending to skipped rules. One rules file
defines a rule with priority WARNING, a second rules file appends to
that rules file, and the configured priority is ERROR.

Ensures that falco rules without errors.
2018-04-05 10:28:45 -07:00
Mark Stemm
38eb5b8741 Add more validations (#329)
* Add the ability to validate multiple rules files

Allow multiple -V arguments just as we do with multiple -r arguments.

* With verbose output, print dangling macros/lists

Start tracking whether or not a given macro/list is actually used when
compiling the set of rules. Every macro/list has an attribute used,
which defaults to false and is set to true whenever it is referred to in
a macro/rule/list.

When run with -v, any macro/list that still has used=false results in a
warning message.

Also, it turns out the fix for
https://github.com/draios/falco/issues/197 wasn't being applied to
macros. Fix that.
2018-02-26 16:59:18 -05:00
Mark Stemm
aa073586f1 Add ability to filter events by priority/cleanups
Clean up the handling of priority levels within rules. It used to be a
mix of strings handled in various places. Now, in falco_common.h there's
a consistent type for priority-as-number as well as a list of
priority-as-string values. Priorities are passed around as numbers
instead of strings. It's still permissive about capitalization.

Also add the ability to load rules by severity. New falco
config option "priority=<val>"/-o priority=<val> specifies the minimum
priority level of rules that will be loaded.

Add unit tests for same. The test suppresses INFO notifications for a
rule/trace file combination that would otherwise generate them.
2017-10-05 18:07:54 -07:00
Mark Stemm
a38f7f181b Add ability to append to rules/macros
Add the ability to append to rules/macros, like we already do with
lists. For rules/macros, if the object has an append: true key, the
condition value is appended to the condition of an existing rule/macro
with the same name.

Like lists, it's an error to specify append: true without there being an
existing rule/macro.

Also add tests that test the same kind of things we did for lists:
 - That append: true really does append
 - That append: false overwrites the rule/macro
 - That it's an error to append with a prior rule/macro existing.
2017-09-22 17:08:00 -07:00
Mark Stemm
240a8ffffa Merge pull request #264 from draios/mergable-lists
Mergable lists
2017-08-10 11:08:36 -07:00
Mark Stemm
2c189d6a60 Add ability to append to lists.
List nodes can now have an 'append' key. If present and true, any values
in this list will be appended to the end of any existing list with the
same name.

It is an error to have a list with 'append' true that has a name that is
not an existing list.
2017-08-09 18:07:34 -07:00
Mark Stemm
ebed9f8dfd Remove trailing newlines from output
If in yaml, the output field is folded-style aka:

output: <
   some multi-line
   output here

The unfolded string will have a trailing newline. Remove it.
2017-08-09 17:53:53 -07:00
Mark Stemm
9e7ce4d36f Also validate macros at parse time.
Also validate macros when they are parsed. Macros are also validated as
a part of rules being parsed, but it's possible to have an individual
rules file containing only macros, or a macro not explicitly tied to any
rule. In this case, it's useful to be able to check the macro to see if
it contains dangling macro references.
2017-06-27 16:44:42 -07:00
Mark Stemm
a0a6914b6a Add support for tagging rules.
- in lua, look for a tags attribute to each rule. This is passed up in
  add_filter as a tags argument (as a lua table). If not present, an
  empty table is used. The tags table is iterated to populate a set
  of tags as strings, which is passed to add_filter().
- A new method falco_engine::enable_rule_by_tag is similar to
  enable_rule(), but is given a set of tag strings. Any rules containing
  one of the tags is enabled/disabled.
- The list of event types has been changed to a set to more accurately
  reflect its purpose.
- New argument to falco -T allows disabling all rules matching a given
  tag, via enable_rule_by_tag(). It can be provided multiple times.
- New argument to falco -t allows running those rules matching a given
  tag. If provided all rules are first disabled. It can be
  provided multiple times, but can not be combined with -T or
  -D (disable rules by name)
- falco_enging supports the notion of a ruleset. The idea is that you
  can choose a set of rules that are enabled/disabled by using
  enable_rule()/enable_rule_by_tag() in combination with a
  ruleset. Later, in process_event() you include that ruleset and the
  rules you had previously enabled will be run.
- rulsets are provided as strings in enable_rule()/enable_rule_by_tag()
  and as numbers in process_event()--this avoids the overhead of string
  lookups per-event. Ruleset ids are created on the fly as needed. A
  utility method find_ruleset_id() looks up the ruleset id for a given
  name. The default ruleset is NULL string/0 numeric if not provided.
- Although the ruleset is a useful falco engine feature, it isn't that
  important to the falco standalone program, so it's not
  documented. However, you can change the ruleset by providing
  FALCO_RULESET in the environment.
2017-02-08 11:08:36 -08:00
Mark Stemm
362a6b7b9a Prefix outputs with * within the engine.
Prefix output strings with * so they are always permissive in the
engine.

In falco outputs, which adds its own prefix, remove any leading * before
adding the custom prefix.
2017-01-03 12:58:01 -08:00
Mark Stemm
7c419b6d6b Allow any macro/list/rule to be overridden
Allow any list/macro/rule to be overridden by a subsequent file. The
persistent state that lives across invocations of load_rules are the 3
arrays ordered_{list,macro,rule}_names, which have the
lists/macros/rules in the order in which they first appear, and tables
{rules,macros,lists}_by_name, which maps from a name to a yaml object.

With each call to load_rules, the set of loaded rules is reset and the
state of expanded lists, compiled macros, compiled rules, and rule
metadata are recreated from scratch, using the ordered_*_names arrays
and *_by_name tables. That way, any list/macro/rule can be redefined in
a subsequent file with new values.
2016-12-29 13:32:55 -08:00
Mark Stemm
37388c56ff Validate rule outputs when loading rules.
Validate rule outputs when loading rules by attempting to create a
formatter based on the rule's output field. If there's an error, it will
propagate up through load_rules and cause falco to exit rather than
discover the problem only when trying to format the event and the rule's
output field.

This required moving formats.{cpp,h} into the falco engine directory
from the falco general directory. Note that these functions are loaded
twice in the two lua states used by falco (engine and outputs).

There's also a couple of minor cleanups:

 - falco_formats had a private instance variable that was unused, remove
   it.
 - rename the package for the falco_formats functions to formats instead
   of falco so it's more standalone.
 - don't throw a c++ exception in falco_formats::formatter. Instead
   generate a lua error, which is handled more cleanly.
 - free_formatter doesn't return any values, so set the return value of
   the function to 0.
2016-12-22 12:55:36 -08:00
Mark Stemm
0d46fcf819 Move container.info handling to falco engine.
container.info handling used to be handled by the the falco_outputs
object. However, this caused problems for applications that only used
the falco engine, doing their own output formatting for matching events.

Fix this by moving output formatting into the falco engine itself. The
part that replaces %container.info/adds extra formatting to the end of a
rule's output now happens while loading the rule.
2016-12-22 12:55:36 -08:00
Mark Stemm
14c9d05f9f Improve error messages when loading rules.
Related to the changes in https://github.com/draios/agent/pull/267,
improve error messages when trying to load sets of rules with errors:

 - Check that yaml parsing of rules_content actually resulted in
   something.
 - Return an error for rules that have an empty name.
 - Return an error for yaml objects that aren't a rule/macro/list.
 - When compiling, don't print an error message, simply return one,
   including a wrapper "can not compile ..." string.
2016-12-22 12:55:36 -08:00
Mark Stemm
3e1117d746 Add license comments to all source code.
Add comment blocks to all source code w/ our gpl copyright notice.
2016-10-24 15:56:45 -07:00
Mark Stemm
ae0ba57306 Add the new pmatch operator.
Make changes to the lua-specific rule parser/compiler to handle the
pmatch operator.
2016-10-24 15:56:45 -07:00
Mark Stemm
f68fba103e Support enabled flag for rules.
If a rule has a enabled attribute, and if the value is false, call the
engine's enable_rule() method to disable the rule. Like add_filter,
there's a static method which takes the object as the first argument and
a non-static method that calls the engine.

This fixes #72.
2016-10-24 15:56:45 -07:00
Mark Stemm
f547dc97ab Move falco engine to its own library.
Move the c++ and lua code implementing falco engine/falco common to its
own directory userspace/engine. It's compiled as a static library
libfalco_engine.a, and has its own CMakeLists.txt so it can be included
by other projects.

The engine's CMakeLists.txt has a add_subdirectory for the falco rules
directory, so including the engine also builds the rules.

The variables you need to set to use the engine's CMakeLists.txt are:

- CMAKE_INSTALL_PREFIX: the root directory below which everything is
  installed.
- FALCO_ETC_DIR: where to install the rules file.
- FALCO_SHARE_DIR: where to install lua code, relative to the
- install/package root.
- LUAJIT_INCLUDE: where to find header files for lua.
- FALCO_SINSP_LIBRARY: the library containing sinsp code. It will be
- considered a dependency of the engine.
- LPEG_LIB/LYAML_LIB/LIBYAML_LIB: locations for third-party libraries.
- FALCO_COMPONENT: if set, will be included as a part of any install()
  commands.

Instead of specifying /usr/share/falco in config_falco_*.h.in, use
CMAKE_INSTALL_PREFIX and FALCO_SHARE_DIR.

The lua code for the engine has also moved, so the two lua source
directories (userspace/engine/lua and userspace/falco/lua) need to be
available separately via falco_common, so make it an argument to
falco_common::init.

As a part of making it easy to include in another project, also clean up
LPEG build/defs. Modify build-lpeg to add a PREFIX argument to allow for
object files/libraries being in an alternate location, and when building
lpeg, put object files in a build/ subdirectory.
2016-10-24 15:56:45 -07:00