github/woodpecker
1
0
Fork 0
mirror of https://github.com/woodpecker-ci/woodpecker.git synced 2025-09-18 01:00:14 +00:00
Code Issues Packages Projects Releases Wiki Activity

Versioned Docs for v2.8.0 Release (#4469)

Browse Source
This commit is contained in:
6543
2024-11-28 19:50:14 +01:00
committed by GitHub
parent d4c0178251
commit 9fedc88a9c
99 changed files with 907 additions and 436 deletions
Download Patch File Download Diff File Expand all files Collapse all files

153
docs/versioned_docs/version-2.8/92-development/01-getting-started.md Normal file
View File

@@ -0,0 +1,153 @@
# Getting started
You can develop on your local computer by following the [steps below](#preparation-for-local-development) or you can start with a fully prepared online setup using [Gitpod](https://github.com/gitpod-io/gitpod) and [Gitea](https://github.com/go-gitea/gitea).
## Gitpod
If you want to start development or updating docs as easy as possible, you can use our preconfigured setup for Woodpecker using [Gitpod](https://github.com/gitpod-io/gitpod). Gitpod starts a complete development setup in the cloud containing:
- An IDE in the browser or bridged to your local VS-Code or Jetbrains
- A preconfigured [Gitea](https://github.com/go-gitea/gitea) instance as forge
- A preconfigured Woodpecker server
- A single preconfigured Woodpecker agent node
- Our docs preview server
Start Woodpecker in Gitpod by clicking on the following badge. You can log in with `woodpecker` and `password`.
[![Open in Gitpod](https://gitpod.io/button/open-in-gitpod.svg)](https://gitpod.io/#https://github.com/woodpecker-ci/woodpecker)
## Preparation for local development
### Install Go
Install Golang (>=1.20) as described by [this guide](https://go.dev/doc/install).
### Install make
> GNU Make is a tool which controls the generation of executables and other non-source files of a program from the program's source files (<https://www.gnu.org/software/make/>).
Install make on:
- Ubuntu: `apt install make` - [Docs](https://wiki.ubuntuusers.de/Makefile/)
- [Windows](https://stackoverflow.com/a/32127632/8461267)
- Mac OS: `brew install make`
### Install Node.js & `pnpm`
Install [Node.js (>=14)](https://nodejs.org/en/download/) if you want to build Woodpecker's UI or documentation.
For dependency installation (`node_modules`) of UI and documentation of Woodpecker the package manager pnpm is used.
[This guide](https://pnpm.io/installation) describes the installation of `pnpm`.
### Install `pre-commit` (optional)
Woodpecker uses [`pre-commit`](https://pre-commit.com/) to allow you to easily autofix your code.
To apply it during local development, take a look at [`pre-commit`s documentation](https://pre-commit.com/#usage).
### Create a `.env` file with your development configuration
Similar to the environment variables you can set for your production setup of Woodpecker, you can create a `.env` file in the root of the Woodpecker project and add any needed config to it.
A common config for debugging would look like this:
```ini
WOODPECKER_OPEN=true
WOODPECKER_ADMIN=your-username
# if you want to test webhooks with an online forge like GitHub this address needs to be accessible from public server
WOODPECKER_HOST=http://your-dev-address.com
# github (sample for a forge config - see /docs/administration/forge/overview for other forges)
WOODPECKER_GITHUB=true
WOODPECKER_GITHUB_CLIENT=<redacted>
WOODPECKER_GITHUB_SECRET=<redacted>
# agent
WOODPECKER_SERVER=localhost:9000
WOODPECKER_AGENT_SECRET=a-long-and-secure-password-used-for-the-local-development-system
WOODPECKER_MAX_WORKFLOWS=1
# enable if you want to develop the UI
# WOODPECKER_DEV_WWW_PROXY=http://localhost:8010
# used so you can login without using a public address
WOODPECKER_DEV_OAUTH_HOST=http://localhost:8000
# disable health-checks while debugging (normally not needed while developing)
WOODPECKER_HEALTHCHECK=false
# WOODPECKER_LOG_LEVEL=debug
# WOODPECKER_LOG_LEVEL=trace
```
### Setup OAuth
Create an OAuth app for your forge as described in the [forges documentation](../30-administration/11-forges/11-overview.md). If you set `WOODPECKER_DEV_OAUTH_HOST=http://localhost:8000` you can use that address with the path as explained for the specific forge to login without the need for a public address. For example for GitHub you would use `http://localhost:8000/authorize` as authorization callback URL.
## Developing with VS Code
You can use different methods for debugging the Woodpecker applications. One of the currently recommended ways to debug and test the Woodpecker application is using [VS-Code](https://code.visualstudio.com/) or [VS-Codium](https://vscodium.com/) (Open-Source binaries of VS-Code) as most maintainers are using it and Woodpecker already includes the needed debug configurations for it.
To launch all needed services for local development, you can use "Woodpecker CI" debugging configuration that will launch UI, server and agent in debugging mode. Then open `http://localhost:8000` to access it.
As a starting guide for programming Go with VS Code, you can use this video guide:
[![Getting started with Go in VS Code](https://img.youtube.com/vi/1MXIGYrMk80/0.jpg)](https://www.youtube.com/watch?v=1MXIGYrMk80)
### Debugging Woodpecker
The Woodpecker source code already includes launch configurations for the Woodpecker server and agent. To start debugging you can click on the debug icon in the navigation bar of VS-Code (ctrl-shift-d). On that page you will see the existing launch jobs at the top. Simply select the agent or server and click on the play button. You can set breakpoints in the source files to stop at specific points.
![Woodpecker debugging with VS Code](./vscode-debug.png)
## Testing & linting code
To test or lint parts of Woodpecker, you can run one of the following commands:
```bash
# test server code
make test-server
# test agent code
make test-agent
# test cli code
make test-cli
# test datastore / database related code like migrations of the server
make test-server-datastore
# lint go code
make lint
# lint UI code
make lint-frontend
# test UI code
make test-frontend
```
If you want to test a specific Go file, you can also use:
```bash
go test -race -timeout 30s go.woodpecker-ci.org/woodpecker/v2/<path-to-the-package-or-file-to-test>
```
Or you can open the test-file inside [VS-Code](#developing-with-vs-code) and run or debug the test by clicking on the inline commands:
![Run test via VS-Code](./vscode-run-test.png)
## Run applications from terminal
If you want to run a Woodpecker applications from your terminal, you can use one of the following commands from the base of the Woodpecker project. They will execute Woodpecker in a similar way as described in [debugging Woodpecker](#debugging-woodpecker) without the ability to really debug it in your editor.
```bash title="start server"
go run ./cmd/server
```
```bash title="start agent"
go run ./cmd/agent
```
```bash title="execute cli command"
go run ./cmd/cli [command]
```

26
docs/versioned_docs/version-2.8/92-development/02-core-ideas.md Normal file
View File

@@ -0,0 +1,26 @@
# Core ideas
- A configuration (e.g. of a pipeline) should never be [turing complete](https://en.wikipedia.org/wiki/Turing_completeness) (We have agents to exec things 🙂).
- If possible, follow the [KISS principle](https://en.wikipedia.org/wiki/KISS_principle).
- What is used most often should be default.
- Keep different topics separated, so you can write plugins, port new ideas ... more easily, see [Architecture](./05-architecture.md).
## Addons and extensions
If you are wondering whether your contribution will be accepted to be merged in the Woodpecker core, or whether it's better to write an
[addon forge](../30-administration/11-forges/100-addon.md), [extension](../30-administration/40-advanced/100-external-configuration-api.md) or an
[external custom backend](../30-administration/22-backends/50-custom-backends.md), please check these points:
- Is your change very specific to your setup and unlikely to be used by anyone else?
- Does your change violate the [guidelines](#guidelines)?
Both should be false when you open a pull request to get your change into the core repository.
### Guidelines
#### Forges
A new forge must support these features:
- OAuth2
- Webhooks

39
docs/versioned_docs/version-2.8/92-development/03-ui.md Normal file
View File

@@ -0,0 +1,39 @@
# UI Development
To develop the UI you need to install [Node.js and pnpm](./01-getting-started.md#install-nodejs--pnpm). In addition it is recommended to use VS-Code with the recommended plugin selection to get features like auto-formatting, linting and typechecking. The UI is written with [Vue 3](https://v3.vuejs.org/) as Single-Page-Application accessing the Woodpecker REST api.
## Setup
The UI code is placed in `web/`. Change to that folder in your terminal with `cd web/` and install all dependencies by running `pnpm install`. For production builds the generated UI code is integrated into the Woodpecker server by using [go-embed](https://pkg.go.dev/embed).
Testing UI changes would require us to rebuild the UI after each adjustment to the code by running `pnpm build` and restarting the Woodpecker server. To avoid this you can make use of the dev-proxy integrated into the Woodpecker server. This integrated dev-proxy will forward all none api request to a separate http-server which will only serve the UI files.
![UI Proxy architecture](./ui-proxy.svg)
Start the UI server locally with [hot-reloading](https://stackoverflow.com/a/41429055/8461267) by running: `pnpm start`. To enable the forwarding of requests to the UI server you have to enable the dev-proxy inside the Woodpecker server by adding `WOODPECKER_DEV_WWW_PROXY=http://localhost:8010` to your `.env` file.
After starting the Woodpecker server as explained in the [debugging](./01-getting-started.md#debugging-woodpecker) section, you should now be able to access the UI under [http://localhost:8000](http://localhost:8000).
## Tools and frameworks
The following list contains some tools and frameworks used by the Woodpecker UI. For some points we added some guidelines / hints to help you developing.
- [Vue 3](https://v3.vuejs.org/)
- use `setup` and composition api
- place (re-usable) components in `web/src/components/`
- views should have a route in `web/src/router.ts` and are located in `web/src/views/`
- [Windicss](https://windicss.org/) (similar to Tailwind)
- use Windicss classes where possible
- if needed extend the Windicss config to use new classes
- [Vite](https://vitejs.dev/) (similar to Webpack)
- [Typescript](https://www.typescriptlang.org/)
- avoid using `any` and `unknown` (the linter will prevent you from doing so anyways :wink:)
- [eslint](https://eslint.org/)
- [Volar & vue-tsc](https://github.com/johnsoncodehk/volar/) for type-checking in .vue file
- use the take-over mode of Volar as described by [this guide](https://github.com/johnsoncodehk/volar/discussions/471)
## Messages and Translations
Woodpecker uses [Vue I18n](https://vue-i18n.intlify.dev/) as translation library. New translations have to be added to `web/src/assets/locales/en.json`. The English source file will be automatically imported into [Weblate](https://translate.woodpecker-ci.org/) (the translation system used by Woodpecker) where all other languages will be translated by the community based on the English source.
You must not provide translations except English in PRs, otherwise weblate could put git into conflicts (when someone has translated in that language file and changes are not into main branch yet)
For more information about translations see [Translations](./08-translations.md).

20
docs/versioned_docs/version-2.8/92-development/04-docs.md Normal file
View File

@@ -0,0 +1,20 @@
# Documentation
The documentation is using docusaurus as framework. You can learn more about it from its [official documentation](https://docusaurus.io/docs/).
If you only want to change some text it probably is enough if you just search for the corresponding [Markdown](https://www.markdownguide.org/basic-syntax/) file inside the `docs/docs/` folder and adjust it. If you want to change larger parts and test the rendered documentation you can run docusaurus locally. Similarly to the UI you need to install [Node.js and pnpm](./01-getting-started.md#install-nodejs--pnpm). After that you can run and build docusaurus locally by using the following commands:
```bash
cd docs/
pnpm install
# build plugins used by the docs
pnpm build:woodpecker-plugins
# start docs with hot-reloading, so you can change the docs and directly see the changes in the browser without reloading it manually
pnpm start
# or build the docs to deploy it to some static page hosting
pnpm build
```

48
docs/versioned_docs/version-2.8/92-development/05-architecture.md Normal file
View File

@@ -0,0 +1,48 @@
# Architecture
## Package architecture
![Woodpecker architecture](./woodpecker-architecture.png)
## System architecture
### main package hierarchy
| package | meaning | imports |
| ------------------ | -------------------------------------------------------------- | ------------------------------------- |
| `cmd/**` | parse command-line args & environment to stat server/cli/agent | all other |
| `agent/**` | code only agent (remote worker) will need | `pipeline`, `shared` |
| `cli/**` | code only cli tool does need | `pipeline`, `shared`, `woodpecker-go` |
| `server/**` | code only server will need | `pipeline`, `shared` |
| `shared/**` | code shared for all three main tools (go help utils) | only std and external libs |
| `woodpecker-go/**` | go client for server rest api | std |
### Server
| package | meaning | imports |
| -------------------- | ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `server/api/**` | handle web requests from `server/router` | `pipeline`, `../badges`, `../ccmenue`, `../logging`, `../model`, `../pubsub`, `../queue`, `../forge`, `../shared`, `../store`, `shared`, (TODO: mv `server/router/middleware/session`) |
| `server/badges/**` | generate svg badges for pipelines | `../model` |
| `server/ccmenu/**` | generate xml ccmenu for pipelines | `../model` |
| `server/grpc/**` | gRPC server agents can connect to | `pipeline/rpc/**`, `../logging`, `../model`, `../pubsub`, `../queue`, `../forge`, `../pipeline`, `../store` |
| `server/logging/**` | logging lib for gPRC server to stream logs while running | std |
| `server/model/**` | structs for store (db) and api (json) | std |
| `server/plugins/**` | plugins for server | `../model`, `../forge` |
| `server/pipeline/**` | orchestrate pipelines | `pipeline`, `../model`, `../pubsub`, `../queue`, `../forge`, `../store`, `../plugins` |
| `server/pubsub/**` | pubsub lib for server to push changes to the WebUI | std |
| `server/queue/**` | queue lib for server where agents pull new pipelines from via gRPC | `server/model` |
| `server/forge/**` | forge lib for server to connect and handle forge specific stuff | `shared`, `server/model` |
| `server/router/**` | handle requests to REST API (and all middleware) and serve UI and WebUI config | `shared`, `../api`, `../model`, `../forge`, `../store`, `../web` |
| `server/store/**` | handle database | `server/model` |
| `server/shared/**` | TODO: move and split [#974](https://github.com/woodpecker-ci/woodpecker/issues/974) | |
| `server/web/**` | server SPA | |
- `../` = `server/`
### Agent
TODO
### CLI
TODO

7
docs/versioned_docs/version-2.8/92-development/06-conventions.md Normal file
View File

@@ -0,0 +1,7 @@
# Conventions
## Database naming
Database tables are named plural, columns don't have any prefix.
Example: Table name `agent`, columns `id`, `name`.

23
docs/versioned_docs/version-2.8/92-development/07-guides.md Normal file
View File

@@ -0,0 +1,23 @@
# Guides
## ORM
Woodpecker uses [Xorm](https://xorm.io/) as ORM for the database connection.
## Add a new migration
Woodpecker uses migrations to change the database schema if a database model has been changed. Add the new migration task into `server/store/datastore/migration/`.
:::info
Adding new properties to models will be handled automatically by the underlying [ORM](#orm) based on the [struct field tags](https://stackoverflow.com/questions/10858787/what-are-the-uses-for-tags-in-go) of the model. If you add a completely new model, you have to add it to the `allBeans` variable at `server/store/datastore/migration/migration.go` to get a new table created.
:::
:::warning
You should not use `sess.Begin()`, `sess.Commit()` or `sess.Close()` inside a migration. Session / transaction handling will be done by the underlying migration manager.
:::
To automatically execute the migration after the start of the server, the new migration needs to be added to the end of `migrationTasks` in `server/store/datastore/migration/migration.go`. After a successful execution of that transaction the server will automatically add the migration to a list, so it won't be executed again on the next start.
## Constants of official images
All official default images, are saved in [shared/constant/constant.go](https://github.com/woodpecker-ci/woodpecker/blob/main/shared/constant/constant.go) and must be pinned by an exact tag.

9
docs/versioned_docs/version-2.8/92-development/08-translations.md Normal file
View File

@@ -0,0 +1,9 @@
# Translations
To translate the web UI into your language, we have [our own Weblate instance](https://translate.woodpecker-ci.org/). Please register there and translate Woodpecker into your language. **We won't accept PRs changing any language except English.**
<a href="https://translate.woodpecker-ci.org/engage/woodpecker-ci/">
<img src="https://translate.woodpecker-ci.org/widgets/woodpecker-ci/-/ui/multi-blue.svg" alt="Translation status" />
</a>
Woodpecker uses [Vue I18n](https://vue-i18n.intlify.dev/) as translation library.

63
docs/versioned_docs/version-2.8/92-development/09-swagger.md Normal file
View File

@@ -0,0 +1,63 @@
# Swagger, API Spec and Code Generation
Woodpecker uses [gin-swagger](https://github.com/swaggo/gin-swagger) middleware to automatically
generate Swagger v2 API specifications and a nice looking Web UI from the source code.
Also, the generated spec will be transformed into Markdown, using [go-swagger](https://github.com/go-swagger/go-swagger)
and then being using on the community's website documentation.
It's paramount important to keep the gin handler function's godoc documentation up-to-date,
to always have accurate API documentation.
Whenever you change, add or enhance an API endpoint, please update the godocs.
You don't require any extra tools on your machine, all Swagger tooling is automatically fetched by standard Go tools.
## Gin-Handler API documentation guideline
Here's a typical example of how annotations for Swagger documentation look like...
```go title="server/api/user.go"
// @Summary Get a user
// @Description Returns a user with the specified login name. Requires admin rights.
// @Router /users/{login} [get]
// @Produce json
// @Success 200 {object} User
// @Tags Users
// @Param Authorization header string true "Insert your personal access token" default(Bearer <personal access token>)
// @Param login path string true "the user's login name"
// @Param foobar query string false "optional foobar parameter"
// @Param page query int false "for response pagination, page offset number" default(1)
// @Param perPage query int false "for response pagination, max items per page" default(50)
```
```go title="server/model/user.go"
type User struct {
ID int64 `json:"id" xorm:"pk autoincr 'user_id'"`
// ...
} // @name User
```
These guidelines aim to have consistent wording in the swagger doc:
- first word after `@Summary` and `@Summary` are always uppercase
- `@Summary` has no `.` (dot) at the end of the line
- model structs shall use custom short names, to ease life for API consumers, using `@name`
- `@Success` object or array declarations shall be short, this means the actual `model.User` struct must have a `@name` annotation, so that the model can be renderend in Swagger
- when pagination is used, `@Parame page` and `@Parame perPage` must be added manually
- `@Param Authorization` is almost always present, there are just a few un-protected endpoints
There are many examples in the `server/api` package, which you can use a blueprint.
More enhanced information you can find here <https://github.com/swaggo/swag/blob/master/README.md#declarative-comments-format>
### Manual code generation
```bash title="generate the server's Go code containing the Swagger"
make generate-swagger
```
```bash title="update the Markdown in the ./docs folder"
make docs
```
```bash title="auto-format swagger related godoc"
go run github.com/swaggo/swag/cmd/swag@latest fmt -g server/api/z.go
```

81
docs/versioned_docs/version-2.8/92-development/09-testing.md Normal file
View File

@@ -0,0 +1,81 @@
# Testing
## Backend
### Unit Tests
[We use default golang unit tests](https://go.dev/doc/tutorial/add-a-test)
with [`"github.com/stretchr/testify/assert"`](https://pkg.go.dev/github.com/stretchr/testify@v1.9.0/assert) to simplify testing.
### Integration Tests
### Dummy backend
There is a special backend called **`dummy`** which does not execute any commands, but emulates how a typical backend should behave.
To enable it you need to build the agent or cli with the `test` build tag.
An example pipeline config would be:
```yaml
when:
event: manual
steps:
- name: echo
image: dummy
commands: echo "hello woodpecker"
environment:
SLEEP: '1s'
services:
echo:
image: dummy
commands: echo "i am a sevice"
```
This could be executed via `woodpecker-cli --log-level trace exec --backend-engine dummy example.yaml`:
```none
9:18PM DBG pipeline/pipeline.go:94 > executing 2 stages, in order of: CLI=exec
9:18PM DBG pipeline/pipeline.go:104 > stage CLI=exec StagePos=0 Steps=echo
9:18PM DBG pipeline/pipeline.go:104 > stage CLI=exec StagePos=1 Steps=echo
9:18PM TRC pipeline/backend/dummy/dummy.go:75 > create workflow environment taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM DBG pipeline/pipeline.go:176 > prepare CLI=exec step=echo
9:18PM DBG pipeline/pipeline.go:203 > executing CLI=exec step=echo
9:18PM TRC pipeline/backend/dummy/dummy.go:81 > start step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM TRC pipeline/backend/dummy/dummy.go:167 > tail logs of step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM DBG pipeline/pipeline.go:209 > complete CLI=exec step=echo
[echo:L0:0s] StepName: echo
[echo:L1:0s] StepType: service
[echo:L2:0s] StepUUID: 01J10P578JQE6E25VV1A2DNQN9
[echo:L3:0s] StepCommands:
[echo:L4:0s] ------------------
[echo:L5:0s] echo ja
[echo:L6:0s] ------------------
[echo:L7:0s] 9:18PM DBG pipeline/pipeline.go:176 > prepare CLI=exec step=echo
9:18PM DBG pipeline/pipeline.go:203 > executing CLI=exec step=echo
9:18PM TRC pipeline/backend/dummy/dummy.go:81 > start step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM TRC pipeline/backend/dummy/dummy.go:167 > tail logs of step echo taskUUID=01J10P578JQE6E25VV1EQF0745
[echo:L0:0s] StepName: echo
[echo:L1:0s] StepType: commands
[echo:L2:0s] StepUUID: 01J10P578JQE6E25VV1DFSXX1Y
[echo:L3:0s] StepCommands:
[echo:L4:0s] ------------------
[echo:L5:0s] echo ja
[echo:L6:0s] ------------------
[echo:L7:0s] 9:18PM TRC pipeline/backend/dummy/dummy.go:108 > wait for step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM TRC pipeline/backend/dummy/dummy.go:187 > stop step echo taskUUID=01J10P578JQE6E25VV1EQF0745
9:18PM DBG pipeline/pipeline.go:209 > complete CLI=exec step=echo
9:18PM TRC pipeline/backend/dummy/dummy.go:208 > delete workflow environment taskUUID=01J10P578JQE6E25VV1EQF0745
```
There are also environment variables to alter step behaviour:
- `SLEEP: 10` will let the step wait 10 seconds
- `EXPECT_TYPE` allows to check if a step is a `clone`, `service`, `plugin` or `commands`
- `STEP_START_FAIL: true` if set will simulate a step to fail before actually being started (e.g. happens when the container image can not be pulled)
- `STEP_TAIL_FAIL: true` if set will error when we simulate to read from stdout for logs
- `STEP_EXIT_CODE: 2` if set will be used as exit code, default is 0
- `STEP_OOM_KILLED: true` simulates a step being killed by memory constrains
You can let the setup of a whole workflow fail by setting it's UUID to `WorkflowSetupShouldFail`.

4
docs/versioned_docs/version-2.8/92-development/_category_.yaml Normal file
View File

@@ -0,0 +1,4 @@
label: 'Development'
# position: 3
collapsible: true
collapsed: true

16
docs/versioned_docs/version-2.8/92-development/ui-proxy.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 7.5 KiB

BIN
docs/versioned_docs/version-2.8/92-development/vscode-debug.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

BIN
docs/versioned_docs/version-2.8/92-development/vscode-run-test.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 11 KiB

BIN
docs/versioned_docs/version-2.8/92-development/woodpecker-architecture.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 61 KiB