diff --git a/docs/docs/20-usage/20-pipeline-syntax.md b/docs/docs/20-usage/20-pipeline-syntax.md index 2fe1ba54b..6f342f522 100644 --- a/docs/docs/20-usage/20-pipeline-syntax.md +++ b/docs/docs/20-usage/20-pipeline-syntax.md @@ -21,88 +21,6 @@ pipeline: In the above example we define two pipeline steps, `frontend` and `backend`. The names of these steps are completely arbitrary. -## Global Pipeline Conditionals - -Woodpecker gives the ability to skip whole pipelines (not just steps) based on certain conditions. - -### `branches` - -Woodpecker can skip commits based on the target branch. If the branch matches the `branches:` block the pipeline is executed, otherwise it is skipped. - -Example skipping a commit when the target branch is not master: - -```diff -pipeline: - build: - image: golang - commands: - - go build - - go test - -+branches: master -``` - -Example matching multiple target branches: - -```diff -pipeline: - build: - image: golang - commands: - - go build - - go test - -+branches: [ master, develop ] -``` - -Example uses glob matching: - -```diff -pipeline: - build: - image: golang - commands: - - go build - - go test - -+branches: [ master, feature/* ] -``` - -Example includes branches: - -```diff -pipeline: - build: - image: golang - commands: - - go build - - go test - -+branches: -+ include: [ master, feature/* ] -``` - -Example excludes branches: - -```diff -pipeline: - build: - image: golang - commands: - - go build - - go test - -+branches: -+ exclude: [ develop, feature/* ] -``` - -The branch matching is done using [doublestar](https://github.com/bmatcuk/doublestar/#usage), note that a pattern starting with `*` should be put between quotes and a literal `/` needs to be escaped. A few examples: - -- `*\\/*` to match patterns with exactly 1 `/` -- `*\\/**` to match patters with at least 1 `/` -- `*` to match patterns without `/` -- `**` to match everything - ### Skip Commits Woodpecker gives the ability to skip individual commits by adding `[CI SKIP]` to the commit message. Note this is case-insensitive. @@ -334,6 +252,13 @@ when: - branch: prefix/* ``` +The branch matching is done using [doublestar](https://github.com/bmatcuk/doublestar/#usage), note that a pattern starting with `*` should be put between quotes and a literal `/` needs to be escaped. A few examples: + +- `*\\/*` to match patterns with exactly 1 `/` +- `*\\/**` to match patters with at least 1 `/` +- `*` to match patterns without `/` +- `**` to match everything + Execute a step using custom include and exclude logic: ```yaml @@ -469,7 +394,7 @@ when: :::info Path conditions are applied only to **push** and **pull_request** events. It is currently **only available** for GitHub, GitLab. -Gitea only support **push** at the moment ([go-gitea/gitea#18228](https://github.com/go-gitea/gitea/pull/18228)). +Gitea only supports **push** at the moment ([go-gitea/gitea#18228](https://github.com/go-gitea/gitea/pull/18228)). ::: Execute a step only on a pipeline with certain files being changed: @@ -731,6 +656,155 @@ pipeline: ... ``` +## `when` - Global pipeline conditions + +Woodpecker gives the ability to skip whole pipelines (not just steps #when---conditional-execution-1) based on certain conditions by a `when` block. If all conditions in the `when` block evaluate to true the pipeline is executed, otherwise it is skipped, but treated as successful and other pipelines depending on it will still continue. + +### `repo` + +Example conditional execution by repository: + +```diff + pipeline: + slack: + image: plugins/slack + settings: + channel: dev ++ when: ++ repo: test/test +``` + +### `branch` + +:::note +Branch conditions are not applied to tags. +::: + +Example conditional execution by branch: + +```diff +pipeline: + slack: + image: plugins/slack + settings: + channel: dev ++ when: ++ branch: master +``` + +> The step now triggers on master, but also if the target branch of a pull request is `master`. Add an event condition to limit it further to pushes on master only. + +Execute a step if the branch is `master` or `develop`: + +```diff +when: + branch: [master, develop] +``` + +Execute a step if the branch starts with `prefix/*`: + +```diff +when: + branch: prefix/* +``` + +Execute a step using custom include and exclude logic: + +```diff +when: + branch: + include: [ master, release/* ] + exclude: [ release/1.0.0, release/1.1.* ] +``` + +### `event` + +Execute a step if the build event is a `tag`: + +```diff +when: + event: tag +``` + +Execute a step if the pipeline event is a `push` to a specified branch: + +```diff +when: + event: push ++ branch: main +``` + +Execute a step for all non-pull request events: + +```diff +when: + event: [push, tag, deployment] +``` + +Execute a step for all build events: + +```diff +when: + event: [push, pull_request, tag, deployment] +``` + +### `tag` + +This filter only applies to tag events. +Use glob expression to execute a step if the tag name starts with `v`: + +```diff +when: + event: tag + tag: v* +``` + +### `environment` + +Execute a step for deployment events matching the target deployment environment: + +```diff +when: + environment: production + event: deployment +``` + +### `instance` + +Execute a step only on a certain Woodpecker instance matching the specified hostname: + +```diff +when: + instance: stage.woodpecker.company.com +``` + +### `path` + +:::info +Path conditions are applied only to **push** and **pull_request** events. +It is currently **only available** for GitHub, GitLab. +Gitea only supports **push** at the moment ([go-gitea/gitea#18228](https://github.com/go-gitea/gitea/pull/18228)). +::: + +Execute a step only on a pipeline with certain files being changed: + +```diff +when: + path: "src/*" +``` + +You can use [glob patterns](https://github.com/bmatcuk/doublestar#patterns) to match the changed files and specify if the step should run if a file matching that pattern has been changed `include` or if some files have **not** been changed `exclude`. + +```diff +when: + path: + include: [ '.woodpecker/*.yml', '*.ini' ] + exclude: [ '*.md', 'docs/**' ] + ignore_message: "[ALL]" +``` + +**Hint:** Passing a defined ignore-message like `[ALL]` inside the commit message will ignore all path conditions. + ## `depends_on` Woodpecker supports to define multiple pipelines for a repository. Those pipelines will run independent from each other. To depend them on each other you can use the [`depends_on`](https://woodpecker-ci.org/docs/usage/multi-pipeline#flow-control) keyword. diff --git a/pipeline/frontend/yaml/compiler/compiler.go b/pipeline/frontend/yaml/compiler/compiler.go index 0ff6f31d4..0885c8128 100644 --- a/pipeline/frontend/yaml/compiler/compiler.go +++ b/pipeline/frontend/yaml/compiler/compiler.go @@ -85,6 +85,12 @@ func New(opts ...Option) *Compiler { func (c *Compiler) Compile(conf *yaml.Config) *backend.Config { config := new(backend.Config) + if !conf.When.Match(c.metadata, true) { + // This pipeline does not match the configured filter so return an empty config and stop further compilation. + // An empty pipeline will just be skipped completely. + return config + } + // create a default volume config.Volumes = append(config.Volumes, &backend.Volume{ Name: fmt.Sprintf("%s_default", c.prefix), @@ -149,7 +155,7 @@ func (c *Compiler) Compile(conf *yaml.Config) *backend.Config { config.Stages = append(config.Stages, stage) } else if !c.local && !conf.SkipClone { for i, container := range conf.Clone.Containers { - if !container.When.Match(c.metadata) { + if !container.When.Match(c.metadata, false) { continue } stage := new(backend.Stage) @@ -176,7 +182,7 @@ func (c *Compiler) Compile(conf *yaml.Config) *backend.Config { stage.Alias = nameServices for i, container := range conf.Services.Containers { - if !container.When.Match(c.metadata) { + if !container.When.Match(c.metadata, false) { continue } @@ -196,7 +202,7 @@ func (c *Compiler) Compile(conf *yaml.Config) *backend.Config { continue } - if !container.When.Match(c.metadata) { + if !container.When.Match(c.metadata, false) { continue } diff --git a/pipeline/frontend/yaml/config.go b/pipeline/frontend/yaml/config.go index 9296828da..025302e77 100644 --- a/pipeline/frontend/yaml/config.go +++ b/pipeline/frontend/yaml/config.go @@ -10,9 +10,9 @@ import ( type ( // Config defines a pipeline configuration. Config struct { + When constraint.When `yaml:"when,omitempty"` Cache types.Stringorslice Platform string - Branches constraint.List Workspace Workspace Clone Containers Pipeline Containers @@ -23,6 +23,8 @@ type ( DependsOn []string `yaml:"depends_on,omitempty"` RunsOn []string `yaml:"runs_on,omitempty"` SkipClone bool `yaml:"skip_clone"` + // Deprecated use When.Branch + Branches constraint.List } // Workspace defines a pipeline workspace. diff --git a/pipeline/frontend/yaml/config_test.go b/pipeline/frontend/yaml/config_test.go index 14deae65e..79fce715c 100644 --- a/pipeline/frontend/yaml/config_test.go +++ b/pipeline/frontend/yaml/config_test.go @@ -5,6 +5,7 @@ import ( "github.com/franela/goblin" + "github.com/woodpecker-ci/woodpecker/pipeline/frontend" "github.com/woodpecker-ci/woodpecker/pipeline/frontend/yaml/types" ) @@ -19,6 +20,8 @@ func TestParse(t *testing.T) { g.Fail(err) } + g.Assert(out.When.Constraints[0].Event.Match("tester")).Equal(true) + g.Assert(out.Workspace.Base).Equal("/go") g.Assert(out.Workspace.Path).Equal("src/github.com/octocat/hello-world") g.Assert(out.Volumes.Volumes[0].Name).Equal("custom") @@ -61,17 +64,65 @@ func TestParse(t *testing.T) { } g.Assert(out.Pipeline.Containers[0].Name).Equal("notify_fail") g.Assert(out.Pipeline.Containers[0].Image).Equal("plugins/slack") + g.Assert(out.Pipeline.Containers[1].Name).Equal("notify_success") + g.Assert(out.Pipeline.Containers[1].Image).Equal("plugins/slack") + g.Assert(len(out.Pipeline.Containers[0].When.Constraints)).Equal(0) g.Assert(out.Pipeline.Containers[1].Name).Equal("notify_success") g.Assert(out.Pipeline.Containers[1].Image).Equal("plugins/slack") g.Assert(out.Pipeline.Containers[1].When.Constraints[0].Event.Include).Equal([]string{"success"}) }) + + matchConfig, err := ParseString(sampleYaml) + if err != nil { + g.Fail(err) + } + + g.It("Should match event tester", func() { + g.Assert(matchConfig.When.Match(frontend.Metadata{ + Curr: frontend.Build{ + Event: "tester", + }, + }, false)).Equal(true) + }) + + g.It("Should match event tester2", func() { + g.Assert(matchConfig.When.Match(frontend.Metadata{ + Curr: frontend.Build{ + Event: "tester2", + }, + }, false)).Equal(true) + }) + + g.It("Should match branch tester", func() { + g.Assert(matchConfig.When.Match(frontend.Metadata{ + Curr: frontend.Build{ + Commit: frontend.Commit{ + Branch: "tester", + }, + }, + }, true)).Equal(true) + }) + + g.It("Should not match event push", func() { + g.Assert(matchConfig.When.Match(frontend.Metadata{ + Curr: frontend.Build{ + Event: "push", + }, + }, false)).Equal(false) + }) }) }) } var sampleYaml = ` image: hello-world +when: + - event: + - tester + - tester2 + - branch: + - tester build: context: . dockerfile: Dockerfile diff --git a/pipeline/frontend/yaml/constraint/constraint.go b/pipeline/frontend/yaml/constraint/constraint.go index 757945c43..6a3325633 100644 --- a/pipeline/frontend/yaml/constraint/constraint.go +++ b/pipeline/frontend/yaml/constraint/constraint.go @@ -58,9 +58,9 @@ func (when *When) IsEmpty() bool { } // Returns true if at least one of the internal constraints is true. -func (when *When) Match(metadata frontend.Metadata) bool { +func (when *When) Match(metadata frontend.Metadata, global bool) bool { for _, c := range when.Constraints { - if c.Match(metadata) { + if c.Match(metadata, global) { return true } } @@ -68,7 +68,7 @@ func (when *When) Match(metadata frontend.Metadata) bool { if when.IsEmpty() { // test against default Constraints empty := &Constraint{} - return empty.Match(metadata) + return empty.Match(metadata, global) } return false } @@ -126,24 +126,21 @@ func (when *When) UnmarshalYAML(value *yaml.Node) error { // Match returns true if all constraints match the given input. If a single // constraint fails a false value is returned. -func (c *Constraint) Match(metadata frontend.Metadata) bool { - // if event filter is not set, set default - if c.Event.IsEmpty() { - c.Event.Include = []string{ - frontend.EventPush, - frontend.EventPull, - frontend.EventTag, - frontend.EventDeploy, - } +func (c *Constraint) Match(metadata frontend.Metadata, global bool) bool { + match := true + if !global { + c.SetDefaultEventFilter() + + // apply step only filters + match = c.Matrix.Match(metadata.Job.Matrix) } - match := c.Platform.Match(metadata.Sys.Platform) && + match = match && c.Platform.Match(metadata.Sys.Platform) && c.Environment.Match(metadata.Curr.Target) && c.Event.Match(metadata.Curr.Event) && c.Repo.Match(metadata.Repo.Name) && c.Ref.Match(metadata.Curr.Commit.Ref) && - c.Instance.Match(metadata.Sys.Host) && - c.Matrix.Match(metadata.Job.Matrix) + c.Instance.Match(metadata.Sys.Host) // changed files filter apply only for pull-request and push events if metadata.Curr.Event == frontend.EventPull || metadata.Curr.Event == frontend.EventPush { @@ -161,6 +158,18 @@ func (c *Constraint) Match(metadata frontend.Metadata) bool { return match } +// SetDefaultEventFilter set default e event filter if not event filter is already set +func (c *Constraint) SetDefaultEventFilter() { + if c.Event.IsEmpty() { + c.Event.Include = []string{ + frontend.EventPush, + frontend.EventPull, + frontend.EventTag, + frontend.EventDeploy, + } + } +} + // IsEmpty return true if a constraint has no conditions func (c List) IsEmpty() bool { return len(c.Include) == 0 && len(c.Exclude) == 0 diff --git a/pipeline/frontend/yaml/constraint/constraint_test.go b/pipeline/frontend/yaml/constraint/constraint_test.go index e822638a9..ee4e20779 100644 --- a/pipeline/frontend/yaml/constraint/constraint_test.go +++ b/pipeline/frontend/yaml/constraint/constraint_test.go @@ -489,7 +489,7 @@ func TestConstraints(t *testing.T) { for _, test := range testdata { t.Run(test.desc, func(t *testing.T) { c := parseConstraints(t, test.conf) - got, want := c.Match(test.with), test.want + got, want := c.Match(test.with, false), test.want if got != want { t.Errorf("Expect %+v matches %q is %v", test.with, test.conf, want) } diff --git a/pipeline/schema/.woodpecker/test-pipeline-when.yml b/pipeline/schema/.woodpecker/test-pipeline-when.yml new file mode 100644 index 000000000..168e88f53 --- /dev/null +++ b/pipeline/schema/.woodpecker/test-pipeline-when.yml @@ -0,0 +1,18 @@ +when: + - branch: [master, deploy] + event: push + path: + - "folder/**" + - "**/*.c" + - tag: "v**" + event: tag + - event: cron + cron: + include: + - hello + +pipeline: + echo: + image: alpine + commands: + - echo "test" diff --git a/pipeline/schema/schema.json b/pipeline/schema/schema.json index 94a54c2fa..fe0d94eee 100644 --- a/pipeline/schema/schema.json +++ b/pipeline/schema/schema.json @@ -98,6 +98,100 @@ { "type": "array", "items": { "$ref": "#/definitions/step" }, "minLength": 1 } ] }, + "pipeline_when": { + "description": "Whole pipelines can be skipped based on conditions. Read more: TODO", + "oneOf": [ + { + "type": "array", + "minLength": 1, + "items": { "$ref": "#/definitions/pipeline_when_condition" } + }, + { + "$ref": "#/definitions/pipeline_when_condition" + } + ] + }, + "pipeline_when_condition": { + "type": "object", + "additionalProperties": false, + "properties": { + "repo": { + "description": "Execute a step only on a specific repository. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#repo", + "$ref": "#/definitions/constraint_list" + }, + "branch": { + "description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#branch", + "$ref": "#/definitions/constraint_list" + }, + "event": { + "description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#event", + "default": [], + "oneOf": [ + { + "type": "array", + "minLength": 1, + "items": { "$ref": "#/definitions/event_enum" } + }, + { + "$ref": "#/definitions/event_enum" + } + ] + }, + "tag": { + "description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#tag", + "type": "string" + }, + "cron": { + "description": "filter cron by title. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#cron", + "$ref": "#/definitions/constraint_list" + }, + "platform": { + "description": "Execute a step only on a specific platform. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#platform", + "$ref": "#/definitions/constraint_list" + }, + "environment": { + "description": "Execute a step only for a specific environment. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#environment", + "$ref": "#/definitions/constraint_list" + }, + "instance": { + "description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#instance", + "$ref": "#/definitions/constraint_list" + }, + "path": { + "description": "Execute a step only on commit with certain files added/removed/modified. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#path", + "oneOf": [ + { "type": "string" }, + { + "type": "array", + "items": { + "type": "string" + } + }, + { + "type": "object", + "properties": { + "include": { + "type": "array", + "items": { + "type": "string" + } + }, + "exclude": { + "type": "array", + "items": { + "type": "string" + } + }, + "ignore_message": { + "type": "string" + } + }, + "additionalProperties": false + } + ] + } + } + }, "step": { "description": "Every step of your pipeline executes arbitrary commands inside a specified docker container. Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#steps", "type": "object", @@ -165,6 +259,7 @@ }, "event": { "description": "Read more: https://woodpecker-ci.org/docs/usage/pipeline-syntax#event", + "default": ["push", "pull_request", "tag", "deployment"], "oneOf": [ { "type": "array", @@ -253,7 +348,6 @@ } }, "event_enum": { - "default": ["push", "pull_request", "tag", "deployment"], "enum": ["push", "pull_request", "tag", "deployment", "cron"] }, "constraint_list": { diff --git a/server/pipeline/create.go b/server/pipeline/create.go index 6ac20a104..8cfee0994 100644 --- a/server/pipeline/create.go +++ b/server/pipeline/create.go @@ -63,7 +63,7 @@ func Create(ctx context.Context, _store store.Store, repo *model.Repo, build *mo configFetcher := shared.NewConfigFetcher(server.Config.Services.Remote, server.Config.Services.ConfigService, repoUser, repo, build) remoteYamlConfigs, configFetchErr = configFetcher.Fetch(ctx) if configFetchErr == nil { - filtered, parseErr = branchFiltered(build, remoteYamlConfigs) + filtered, parseErr = checkIfFiltered(build, remoteYamlConfigs) if parseErr == nil { if filtered { err := ErrFiltered{Msg: "branch does not match restrictions defined in yaml"} diff --git a/server/pipeline/filter.go b/server/pipeline/filter.go index 6d6cbffe5..7491886c3 100644 --- a/server/pipeline/filter.go +++ b/server/pipeline/filter.go @@ -19,6 +19,7 @@ package pipeline import ( "github.com/rs/zerolog/log" + "github.com/woodpecker-ci/woodpecker/pipeline/frontend" "github.com/woodpecker-ci/woodpecker/pipeline/frontend/yaml" "github.com/woodpecker-ci/woodpecker/server/model" "github.com/woodpecker-ci/woodpecker/server/remote" @@ -49,11 +50,17 @@ func zeroSteps(build *model.Build, remoteYamlConfigs []*remote.FileMeta) bool { } // TODO: parse yaml once and not for each filter function -func branchFiltered(build *model.Build, remoteYamlConfigs []*remote.FileMeta) (bool, error) { +// Check if at least one pipeline step will be execute otherwise we will just ignore this webhook +func checkIfFiltered(build *model.Build, remoteYamlConfigs []*remote.FileMeta) (bool, error) { log.Trace().Msgf("hook.branchFiltered(): build branch: '%s' build event: '%s' config count: %d", build.Branch, build.Event, len(remoteYamlConfigs)) - if build.Event == model.EventTag || build.Event == model.EventDeploy { - return false, nil + matchMetadata := frontend.Metadata{ + Curr: frontend.Build{ + Event: string(build.Event), + Commit: frontend.Commit{ + Branch: build.Branch, + }, + }, } for _, remoteYamlConfig := range remoteYamlConfigs { @@ -64,10 +71,20 @@ func branchFiltered(build *model.Build, remoteYamlConfigs []*remote.FileMeta) (b } log.Trace().Msgf("config '%s': %#v", remoteYamlConfig.Name, parsedPipelineConfig) - if parsedPipelineConfig.Branches.Match(build.Branch) { - return false, nil + // ignore if the pipeline was filtered by matched constraints + if !parsedPipelineConfig.When.Match(matchMetadata, true) { + continue } + + // ignore if the pipeline was filtered by the branch (legacy) + if !parsedPipelineConfig.Branches.Match(build.Branch) { + continue + } + + // at least one config yielded in a valid run. + return false, nil } + // no configs yielded a valid run. return true, nil } diff --git a/server/shared/procBuilder.go b/server/shared/procBuilder.go index f021b625c..e5dc9b8a4 100644 --- a/server/shared/procBuilder.go +++ b/server/shared/procBuilder.go @@ -23,6 +23,7 @@ import ( "strings" "github.com/drone/envsubst" + "github.com/rs/zerolog/log" backend "github.com/woodpecker-ci/woodpecker/pipeline/backend/types" "github.com/woodpecker-ci/woodpecker/pipeline/frontend" @@ -117,7 +118,19 @@ func (b *ProcBuilder) Build() ([]*BuildItem, error) { return nil, &yaml.PipelineParseError{Err: err} } + // checking if filtered. + if !parsed.When.Match(metadata, true) { + log.Debug().Str("pipeline", proc.Name).Msg( + "Marked as skipped, dose not match metadata", + ) + proc.State = model.StatusSkipped + } + + // TODO: deprecated branches filter => remove after some time if !parsed.Branches.Match(b.Curr.Branch) && (b.Curr.Event != model.EventDeploy && b.Curr.Event != model.EventTag) { + log.Debug().Str("pipeline", proc.Name).Msg( + "Marked as skipped, dose not match branch", + ) proc.State = model.StatusSkipped } diff --git a/server/shared/procBuilder_test.go b/server/shared/procBuilder_test.go index 2b92281ca..56c8c3872 100644 --- a/server/shared/procBuilder_test.go +++ b/server/shared/procBuilder_test.go @@ -326,6 +326,52 @@ pipeline: } } +func TestRootWhenFilter(t *testing.T) { + t.Parallel() + + b := ProcBuilder{ + Repo: &model.Repo{}, + Curr: &model.Build{Event: "tester"}, + Last: &model.Build{}, + Netrc: &model.Netrc{}, + Secs: []*model.Secret{}, + Regs: []*model.Registry{}, + Link: "", + Yamls: []*remote.FileMeta{ + {Data: []byte(` +when: + event: + - tester +pipeline: + xxx: + image: scratch +`)}, + {Data: []byte(` +when: + event: + - push +pipeline: + xxx: + image: scratch +`)}, + {Data: []byte(` +pipeline: + build: + image: scratch +`)}, + }, + } + + buildItems, err := b.Build() + if err != nil { + t.Fatal(err) + } + + if len(buildItems) != 2 { + t.Fatal("Should have generated 2 buildItems") + } +} + func TestZeroSteps(t *testing.T) { t.Parallel()