Commit Graph

280 Commits

Author SHA1 Message Date
Daniel Kłobuszewski
6db742fc10 fix fluentd-gcp-scaler to look at correct fluentd-gcp version 2018-02-15 16:15:41 +01:00
Kubernetes Submit Queue
d3bacb914c
Merge pull request #59657 from x13n/manual-fluentd-gcp-scaler
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Enable scaling fluentd-gcp resources using ScalingPolicy.

See https://github.com/justinsb/scaler for more details about ScalingPolicy resource.

**What this PR does / why we need it**:
This is adding a way to override fluentd-gcp resources in a running cluster. The resources syncing for fluentd-gcp is decoupled from addon manager.

**Special notes for your reviewer**:

**Release note**:
```release-note
fluentd-gcp resources can be modified via a ScalingPolicy
```

cc @kawych @justinsb
2018-02-15 03:42:14 -08:00
Kubernetes Submit Queue
bc9c6df31d
Merge pull request #59103 from Random-Liu/upload-container-runtime-log
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Upload container runtime log to sd/es.

I've verified this in my environment. My stackdriver has an extra `container-runtime` entry for node log, and it collects container runtime daemon log correctly.

@yujuhong @feiskyer @crassirostris @piosz 
@kubernetes/sig-node-pr-reviews @kubernetes/sig-instrumentation-pr-reviews 
Signed-off-by: Lantao Liu <lantaol@google.com>

**Release note**:

```release-note
Container runtime daemon (e.g. dockerd) logs in GCE cluster will be uploaded to stackdriver and elasticsearch with tag `container-runtime`
```
2018-02-14 03:33:21 -08:00
Lantao Liu
8d920d095c Upload container runtime log to sd/es.
Signed-off-by: Lantao Liu <lantaol@google.com>
2018-02-13 18:25:02 +00:00
Daniel Kłobuszewski
3b178ee45c Review #1 2018-02-13 10:23:58 +01:00
Kubernetes Submit Queue
7ef11bd964
Merge pull request #59237 from tanshanshan/addons1
Automatic merge from submit-queue (batch tested with PRs 59767, 56454, 59237, 59730, 55479). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Change critical pods’ template to use priority

**What this PR does / why we need it**:
Change critical pods’ template to use priority
Thanks.
**Which issue(s) this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close the issue(s) when PR gets merged)*:
ref #57471

**Special notes for your reviewer**:

**Release note**:

```release-note

```
2018-02-12 15:44:36 -08:00
Daniel Kłobuszewski
2eb24f9ae1 Enable scaling fluentd-gcp resources using ScalingPolicy.
See https://github.com/justinsb/scaler for more details about ScalingPolicy resource.
2018-02-09 14:33:33 +01:00
tanshanshan
95b2b94b1b Change critical pods’ template to use priority 2018-02-08 15:06:27 +08:00
Tim Hockin
3586986416 Switch to k8s.gcr.io vanity domain
This is the 2nd attempt.  The previous was reverted while we figured out
the regional mirrors (oops).

New plan: k8s.gcr.io is a read-only facade that auto-detects your source
region (us, eu, or asia for now) and pulls from the closest.  To publish
an image, push k8s-staging.gcr.io and it will be synced to the regionals
automatically (similar to today).  For now the staging is an alias to
gcr.io/google_containers (the legacy URL).

When we move off of google-owned projects (working on it), then we just
do a one-time sync, and change the google-internal config, and nobody
outside should notice.

We can, in parallel, change the auto-sync into a manual sync - send a PR
to "promote" something from staging, and a bot activates it.  Nice and
visible, easy to keep track of.
2018-02-07 21:14:19 -08:00
Robert Bailey
7cd213ae82 Remove salt configuration from the fluentd-gcp configuration. 2018-01-22 21:06:09 -08:00
Ross Light
6831581f1c Bump fluentd-gcp version 2018-01-12 10:16:13 -08:00
Daniel Kłobuszewski
dca74f17fd
Bump fluentd-gcp image used to 2.0.13 2018-01-08 14:54:26 +01:00
Daniel Kłobuszewski
2eded687be
Bump fluentd-gcp version 2018-01-03 11:46:13 +01:00
Tim Hockin
e9dd8a68f6 Revert k8s.gcr.io vanity domain
This reverts commit eba5b6092a.

Fixes https://github.com/kubernetes/kubernetes/issues/57526
2017-12-22 14:36:16 -08:00
Tim Hockin
eba5b6092a Use k8s.gcr.io vanity domain for container images 2017-12-18 09:18:34 -08:00
Daniel Kłobuszewski
d2cbc37c05
Bump fluentd-gcp version 2017-12-07 14:23:05 +01:00
Rohit Agarwal
ad05928c6e Add wildcard tolerations to kube-proxy.
fluend-gcp already has these tolerations. kube-proxy when it runs as a
static pod gets wildcard `NoExecute` toleration (all static pods get
that). So, added the same toleration to kube-proxy when it runs as a
daemonset. Also added wildcard `NoSchedule` toleration to kube-proxy.
2017-11-29 12:36:58 -08:00
Marian Lobur
3d7a694aed Fix typo in component name of prometheus-to-sd config. 2017-11-28 09:02:08 +01:00
Kubernetes Submit Queue
f85649c6cd
Merge pull request #55950 from crassirostris/fluentd-gcp-setting-resources-fix
Automatic merge from submit-queue (batch tested with PRs 56207, 55950). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Fix setting resources in fluentd-gcp plugin

Currently if some of the variables are not set, scripts prints error, which is not critical, since the function is executed in a separate process, but it leads to the wrong resulting values

```release-note
NONE
```

/cc @piosz @x13n 
/assign @roberthbailey @mikedanese 
Could you please approve?
2017-11-22 05:21:25 -08:00
Mik Vyatskov
e9322b929c Fix setting resources in fluentd-gcp plugin
Signed-off-by: Mik Vyatskov <vmik@google.com>
2017-11-22 12:40:50 +01:00
Lantao Liu
09b614abbb Do not add new field in fluentd CRI log format. 2017-11-22 01:25:24 +00:00
Lantao Liu
2bc0532eb3 Add fluentd support. 2017-11-21 00:36:01 +00:00
Kubernetes Submit Queue
b2125f5aa8
Merge pull request #55509 from tallclair/psp-addons
Automatic merge from submit-queue (batch tested with PRs 54602, 54877, 55243, 55509, 55128). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

PodSecurityPolicies for addons

**What this PR does / why we need it**:

1. Colocate addon PodSecurityPolicy config with the addons (in a `podsecuritypolicies` subdirectory). 
2. Add policies for addons that are currently missing policies (not in the default GCE suite)
3. Remove HostPath SSL certs from several heapster deployments, so that heapster doesn't require a special PSP

**Which issue(s) this PR fixes**:
#43538

**Release note**:
```release-note
- Add PodSecurityPolicies for cluster addons
- Remove SSL cert HostPath volumes from heapster addons
```
2017-11-14 03:03:30 -08:00
Tim Allclair
a1513161b3
Reorganize addon PodSecurityPolicies 2017-11-10 13:57:32 -08:00
Lantao Liu
53d7494b9e Fix CRI fluentd config.
Signed-off-by: Lantao Liu <lantaol@google.com>
2017-11-10 20:55:56 +00:00
Lantao Liu
70a0cdfa8e Add CRI log format support in fluentd. 2017-10-30 06:25:52 +00:00
Kubernetes Submit Queue
949ec719c3
Merge pull request #54635 from loburm/prom-to-sd
Automatic merge from submit-queue (batch tested with PRs 54635, 54250, 54657, 54696, 54700). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Bump version of prometheus-to-sd to 0.2.2.

Bump version of prometheus-to-sd to improve logging, add pod_name and
pod_namespace flags and remove deprecated flags.

Fixes #54583 

```release-note
NONE
```
2017-10-27 14:38:21 -07:00
Kubernetes Submit Queue
fc8bfe2d89 Merge pull request #54395 from crassirostris/fluentd-gcp-rollback-host-networking
Automatic merge from submit-queue (batch tested with PRs 50776, 54395). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

Move fluentd-gcp out of host network

Since metadata proxy doesn't filter service account after all, make fluentd-gcp addon run in its own network

This will mitigate the problem with port collision

```release-note
[fluentd-gcp addon] Fluentd now runs in its own network, not in the host one.
```
2017-10-27 03:09:25 -07:00
Marian Lobur
5b62eb29d2 Bump version of prometheus-to-sd to 0.2.2.
Bump version of prometheus-to-sd to improve logging, add pod_name and
pod_namespace flags and remove deprecated flags.
2017-10-26 15:54:54 +02:00
Tim Allclair
b18edfec7a
Update fluentd-gcp DaemonSet
- Use a dedicated service account to run the fluentd-gcp DS
- Update prometheus-to-sd from v0.1.3 to v0.2.1
- Use the certificates in the prometheus-to-sd image rather than mounting the host certs
2017-10-25 13:11:35 -07:00
Mik Vyatskov
d30af4d8a0 Move fluentd-gcp out of host network 2017-10-23 12:02:54 +02:00
Kubernetes Submit Queue
74604d123d Merge pull request #52565 from crassirostris/sd-logging-update-plugin-0.6.7
Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https://github.com/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>..

[fluentd-gcp addon] Update Stackdriver plugin to version 0.6.7

A new gem among all fixes Java logging severity parsing and string timestamp parsing

Also sync the buffer size with the gem guidelines, making it 1M instead of 2M.

/cc @igorpeshansky
2017-09-20 16:09:31 -07:00
Mik Vyatskov
e73eb86dda [fluentd-gcp addon] Remove audit logs from the fluentd configuration 2017-09-20 11:08:59 +02:00
Mik Vyatskov
ae132a0900 [fluentd-gcp] Update Stackdriver plugin to version 0.6.7 2017-09-15 22:30:15 +02:00
Mik Vyatskov
42be772234 [fluentd-gcp addon] By default ingest audit logs in JSON format 2017-09-15 15:38:30 +02:00
Mik Vyatskov
d8525f8bd1 [fluentd-gcp addon] Trim too long log entries due to Stackdriver limitation 2017-09-13 10:27:17 +02:00
Kubernetes Submit Queue
36b43013c6 Merge pull request #52335 from crassirostris/sd-logging-add-metric
Automatic merge from submit-queue (batch tested with PRs 51041, 52297, 52296, 52335, 52338)

[fluentd-gcp addon] Restore the metric for the number of read log entries

This metric, previously removed, will allow to monitor the number of log entries, that were read, but weren't sent by the output plugin because of liveness probe removing the data.
2017-09-12 11:10:15 -07:00
Mik Vyatskov
683fc23000 [fluentd-gcp addon] Restore the metric for the number of read log entries 2017-09-12 13:24:55 +02:00
Mik Vyatskov
5447a5a2b2 [fluentd-gcp addon] Update event-exporter to address metrics problem 2017-09-11 10:07:10 +02:00
Kubernetes Submit Queue
3b2e403a37 Merge pull request #51011 from xilabao/rbac-v1-in-yaml
Automatic merge from submit-queue (batch tested with PRs 50489, 51070, 51011, 51022, 51141)

update to rbac v1 in yaml file

**What this PR does / why we need it**:
ref to https://github.com/kubernetes/kubernetes/pull/49642
ref https://github.com/kubernetes/features/issues/2

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:
cc @liggitt 

**Release note**:

```release-note
NONE
```
2017-08-23 19:54:28 -07:00
Marian Lobur
d48e2d338d Add flags for prometheus-to-sd components.
Configure prometheus-to-sd-endpoint and prometheus-to-sd-prefix base on
the environment.
2017-08-22 08:30:08 +02:00
Chen Rong
d23df051e1 update to rbac v1 in yaml file 2017-08-21 17:29:37 +08:00
Mik Vyatskov
d64c8809af Update Stackdriver event exporter version 2017-08-11 12:11:25 +02:00
Mik Vyatskov
46f53daef0 Refactored the fluentd-es addon files, moved the fluentd configuration to ConfigMap 2017-08-04 20:05:15 +02:00
icereval
60872db873
log node-problem-detector 2017-07-16 16:46:25 -04:00
Kubernetes Submit Queue
455e44b616 Merge pull request #48812 from crassirostris/change-fluentd-monitoring
Automatic merge from submit-queue (batch tested with PRs 48812, 48276)

Change fluentd-gcp monitoring to use metrics exposed by SD plugin

Following https://github.com/GoogleCloudPlatform/fluent-plugin-google-cloud/pull/135, make fluentd-gcp expose metrics in Prometheus registry and use them instead of counting records in the pipeline.

/cc @piosz @igorpeshansky

```release-note
Fluentd-gcp DaemonSet exposes different set of metrics.
```
2017-07-14 04:43:42 -07:00
Kubernetes Submit Queue
b507254bb0 Merge pull request #46825 from zjj2wry/fix_doc_spell
Automatic merge from submit-queue

Update docs for user-guide

**What this PR does / why we need it**:

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: fixes #

**Special notes for your reviewer**:

**Release note**:

```release-note
NONE
```
2017-07-13 18:50:36 -07:00
Mik Vyatskov
125f759907 Change fluentd-gcp monitoring to use metrics exposed by SD plugin 2017-07-12 17:07:15 +02:00
Mik Vyatskov
b11084e76c Bump event-exporter version 2017-07-10 17:32:40 +02:00
gmarek
3f57d8dba3 Add generic NoSchedule toleration to fluentd in gcp config as a quick-fix for #44445 2017-06-28 10:35:58 +02:00
Mik Vyatskov
b6a0e442ce Fix fluentd-gcp configuration to facilitate JSON parsing 2017-06-27 16:16:00 +02:00
zhengjiajin
2ab0c65506 Update docs for user-guide 2017-06-27 12:21:49 +08:00
Kubernetes Submit Queue
93ef2f2f59 Merge pull request #47860 from crassirostris/fluentd-log-to-stdio
Automatic merge from submit-queue (batch tested with PRs 47860, 47170)

Make fluentd log to stdio instead of a dedicated file

Lower verbosity also, to reduce volume of system logs exported to the backend.

Fix https://github.com/kubernetes/kubernetes/issues/43772

/cc @piosz
2017-06-26 14:03:04 -07:00
Piotr Szczesniak
77fd8a1898 Revert "Decrese fluentd cpu request" 2017-06-26 16:18:24 +02:00
Kubernetes Submit Queue
0cbd0ca189 Merge pull request #47915 from crassirostris/fix-event-exporter-noise
Automatic merge from submit-queue

Bump event-exporter version to reduce warnings noise

Fixes https://github.com/kubernetes/kubernetes/issues/47914
2017-06-23 03:45:20 -07:00
Kubernetes Submit Queue
045a6dca07 Merge pull request #47906 from gmarek/fluentd
Automatic merge from submit-queue

Decrese fluentd cpu request

Fix #47905

cc @piosz - this should fix your tests.
cc @dchen1107
2017-06-22 12:27:13 -07:00
Mik Vyatskov
3932622303 Bump event-exporter version to reduce warnings noise 2017-06-22 19:49:23 +02:00
gmarek
8427d5a274 Decrease fluentd cpu request 2017-06-22 18:30:21 +02:00
Jeff Grafton
4c7c865ff6 Update fluentd-gcp to 2.0.7 2017-06-21 14:08:12 -07:00
Jeff Grafton
b43bb842ad Update event-exporter to v0.1.0-r2 and prometheus-to-sd to v0.1.2-r2 2017-06-21 14:08:09 -07:00
Mik Vyatskov
1d1c126783 Make fluentd log to stdio instead of a dedicated file 2017-06-21 20:14:43 +02:00
Jeff Grafton
641f8c1f29 Revert "Update fluentd-gcp to 2.0.6"
This reverts commit 0bcc271b28.
2017-06-15 11:46:17 -07:00
Jeff Grafton
0bcc271b28 Update fluentd-gcp to 2.0.6 2017-06-14 13:32:03 -07:00
Mik Vyatskov
1cc2235c17 Change port on which fluentd exposes its metrics 2017-06-13 08:15:34 +02:00
Kubernetes Submit Queue
695d438508 Merge pull request #46539 from crassirostris/fluentd-gcp-make-privileged
Automatic merge from submit-queue

Make fluentd-gcp run with host network

Fluentd-gcp should have access to instance's platform-dependent service account in order to work.

/cc @piosz
2017-06-12 10:13:21 -07:00
Kubernetes Submit Queue
a03bb6fc5f Merge pull request #46787 from crassirostris/fluentd-gcp-update
Automatic merge from submit-queue

Update the fluentd-gcp image

Rolled back fluentd version to 0.12 to avoid performance problems and unnecessary noise in logs: https://github.com/kubernetes/contrib/pull/2625

Fixes https://github.com/kubernetes/kubernetes/issues/46990
2017-06-06 01:53:40 -07:00
Kubernetes Submit Queue
8df56da448 Merge pull request #46700 from crassirostris/add-event-exporter-deployment
Automatic merge from submit-queue

Add event exporter deployment to the fluentd-gcp addon

Introduce event exporter deployment to the fluentd-gcp addon so that by default if logging to Stackdriver is enabled, events will be available there also.

In this release, event exporter is a non-critical pod in BestEffort QoS class to avoid preempting actual workload in tightly loaded clusters. It will become critical in one of the future releases.


```release-note
Stackdriver cluster logging now deploys a new component to export Kubernetes events.
```
2017-06-06 00:00:49 -07:00
Mik Vyatskov
527206c36b Add event exporter deployment to the fluentd-gcp addon 2017-06-02 17:00:24 +02:00
Mik Vyatskov
f41a61586d Update the fluentd-gcp image 2017-06-01 19:08:58 +02:00
Mik Vyatskov
a4c3279a68 Update fluentd-gcp image version 2017-06-01 14:28:17 +02:00
Mik Vyatskov
d3c809d900 Make fluentd-gcp run with host network 2017-05-26 22:32:28 +02:00
Mik Vyatskov
a6ccc89541 Add version for fluentd-gcp config 2017-05-18 16:59:05 +02:00
Mik Vyatskov
dcd3ce3bcb Export process start time metric in fluentd-gcp 2017-05-12 16:37:36 +02:00
gmarek
4d7d6b72b3 Add general NoExecute Toleration to fluentd in gcp configuration 2017-05-12 11:23:23 +02:00
Mik Vyatskov
fb3e30729e Add metrics exporter to the fluentd-gcp deployment 2017-04-28 13:27:19 +02:00
Mik Vyatskov
0625697dd5 Remove too verbose label from fluentd metrics 2017-04-27 17:16:25 +02:00
Kubernetes Submit Queue
13e2f5d9d7 Merge pull request #43379 from crassirostris/fluentd-gcp-docs
Automatic merge from submit-queue

Add readme for fluentd addons

/cc @steveperry-53
2017-03-23 02:08:56 -07:00
Mik Vyatskov
7ae8eee1cd Add readme for fluentd-elasticsearch addon 2017-03-22 11:27:48 +01:00
Piotr Szczesniak
8968ac5c36 Renamed fluentd-ds-ready annotation 2017-03-21 20:48:13 +01:00
Mik Vyatskov
48e750f5a0 Increate memory limit for fluentd-gcp 2017-03-21 10:44:01 +01:00
Mik Vyatskov
0fb9e03869 Add readme for fluentd-gcp addon 2017-03-20 13:55:16 +01:00
Tim St. Clair
b707032198
Enable RollingUpdates for the fluentd daemonset addon 2017-03-13 10:51:45 -07:00
Tim St. Clair
f2ee6f1c96
Bump fluentd-gcp to 2.0.2 2017-03-08 17:20:32 -08:00
Mik Vyatskov
d40f9b7a0c Merge pull request #42693 from Crassirostris/fix-owners-files
Fix username case in OWNERS files
2017-03-08 00:55:18 -08:00
Mikhail Vyatskov
0d363f2892 Fix username case in OWNERS files 2017-03-07 16:45:40 -08:00
Tim St. Clair
56a7080e56 Bump fluentd-gcp to version 2.0.1 2017-03-07 16:37:04 -08:00
Kubernetes Submit Queue
4092da38a6 Merge pull request #42127 from crassirostris/remove-fluentd-gcp-image
Automatic merge from submit-queue (batch tested with PRs 42070, 42127)

Remove fluentd-gcp image sources

This PR removes fluentd-gcp image sources from the main kubernetes repo to move it the `contrib`: https://github.com/kubernetes/contrib/pull/2426

Once image is moved, it will be maintained by Stackdriver team (@igorpeshansky, @qingling128 and @dhrupadb)

CC @ixdy @timstclair
2017-03-04 12:58:40 -08:00
Kubernetes Submit Queue
c432e7af37 Merge pull request #42130 from Crassirostris/update-fluentd-gcp-master-config
Automatic merge from submit-queue (batch tested with PRs 42126, 42130, 42232, 42245, 41932)

Update fluentd-gcp configuration for hosted masters

This PR makes use of the new fluentd-gcp image, which is not configured per se, for the hosted masters, which cannot use configmaps.

Mirroring https://github.com/kubernetes/kubernetes/pull/42126
2017-03-02 02:07:22 -08:00
Kubernetes Submit Queue
113cd7da0a Merge pull request #42126 from Crassirostris/move-fluentd-config-to-configmap
Automatic merge from submit-queue

Move fluentd DS config to configmap

This is the logical continuation of https://github.com/kubernetes/kubernetes/pull/41998. This PR makes fluentd-gcp DaemonSet use the new image configured using ConfigMap.

This PR doesn't change the way fluentd-gcp works in case master is not registered, that'll be fixed in a separate PR

CC @ixdy @timstclair @igorpeshansky @qingling128 @dhrupadb

**Release note:**

```release-note
Fluentd-gcp containers spawned by DaemonSet are now configured using ConfigMap
```
2017-03-02 01:43:57 -08:00
Mik Vyatskov
2e7eea594c Remove fluentd-gcp image sources 2017-03-01 01:47:01 +01:00
Avesh Agarwal
99bba40682 Update addons yaml files. 2017-02-27 09:21:59 -05:00
Kubernetes Submit Queue
7265908e7f Merge pull request #41998 from timstclair/fluentd-debian-base
Automatic merge from submit-queue

Cleanup fluentd-gcp image, rebase on debian-base

**Why we need this PR**:

There are several problems with our current fluentd-gcp image:

- It pulls in lots of unused packages, which expose unnecessary risk and create noise in CVE scans (and scare customers). The most notable example is the fluent-ui, which pulls in rails.
- `curl | sh ` is not a good practice for a Dockerfile. First, the script is not checked in the same source control branch, so builds are not reproducible. Second, the actions it is taking are opaque. Third, in this case, using non-standard packages means they're harder to manage with CVE scans & upstream fixes.

**What is changed by this PR?**

- Rather than relying on td-agent (which includes fluent-ui), use standard upstream packages. This is largely based off the [official fluentd debian-based image](https://github.com/fluent/fluentd-docker-image/blob/master/v0.12/debian/Dockerfile).
- Rebases the image on debian-base (depends on https://github.com/kubernetes/kubernetes/pull/41915). We would like to move towards a single full-distro base image we can maintain. This change should be relatively minor.

As a result of these changes, the image size is reduced from 360.6 MB to 185.8 MB (nearly half). Many packages were removed, and the full diff (focus on the unversioned files) is listed here: 3fb704f977

**Which issue this PR fixes** https://github.com/kubernetes/kubernetes/issues/40248

**Special notes for your reviewer**:

This change both addresses security concerns, and is expected to greatly reduce the maintenance burden of the fluentd-gcp image. I'd *really* like to get this into 1.6, so please prioritize this review if possible.

I tested this by running the default e2e suite on a private e2e cluster using the new image. If there are other tests you'd like me to run, please let me know ASAP.

**Release note**:
```release-note
Cleanup fluentd-gcp image: rebase on debian-base, switch to upstream packages, remove fluent-ui & rails
```
2017-02-26 19:15:11 -08:00
Mik Vyatskov
42e064fc90 Move fluentd DS config to configmap 2017-02-27 00:05:48 +01:00
Mik Vyatskov
bd89af6863 Update fluentd-gcp configuration for hosted masters 2017-02-27 00:00:41 +01:00
Zihong Zheng
d4fa046d56 Adds the new addon-manager labels on cluster addon templates 2017-02-24 16:53:12 -08:00
Tim St. Clair
4b4c3e4944
Cleanup fluentd-gcp image, rebase on debian-base 2017-02-24 13:45:12 -08:00
Kubernetes Submit Queue
bb5fdff58b Merge pull request #41567 from Crassirostris/fluentd-gcp-monitoring
Automatic merge from submit-queue (batch tested with PRs 39855, 41433, 41567, 41887, 41652)

Add fluentd monitoring to fluentd-gcp image

Right now we are not able to monitor the state of fluentd in cluster, which may result in logging subsystem quietly failing. This PR tries to address that problem by introducing the fluentd container monitoring:

* fluentd internal metrics, like number of buffers and number of data in buffers
* `logging_line_count`, number of lines, read by fluentd from application containers' logs
    * Has `tag` label, corresponding to the fluentd tag of the entry
* `logging_entry_count`, number of entries, emitted to the output plugin
    * With label `component` set to `container`, generated by application containers
    * With label `component` set to `system`, generated by system components like kubelet, docker, scheduler, etc.
    * Has `tag` label, corresponding to the fluentd tag of the entry

CC @fabxc @igorpeshansky @edsiper
2017-02-23 09:36:33 -08:00
Mik Vyatskov
5d59d4d27b Turn fluentd supervisor off for fluentd-gcp 2017-02-21 13:50:47 +01:00
Mik Vyatskov
8d2d91070a Add fluentd monitoring to fluentd-gcp image 2017-02-16 19:04:32 +01:00
Kubernetes Submit Queue
30e8953fad Merge pull request #41564 from Crassirostris/fluentd-gcp-plugin-version-bump
Automatic merge from submit-queue

Bump fluentd-gcp google_cloud plugin version

Bump the version of `fluent-plugin-google-cloud` in fluentd-gcp image, because it's broken for version `0.5.2`.

Recently, gem `google-api-client` was updated to version `0.10.0`. The new version broke `fluent-plugin-google-cloud` which doesn't specify the upper version of `google-api-client` gem. I'm bumping the version used in our image to allow future changes in this release to be run and tested.

This PR doesn't bump the version, since no effective changes has happened, leaving this for the next PR to do.

CC @igorpeshansky
2017-02-16 09:20:12 -08:00
Mik Vyatskov
e8de31623f Bump fluentd-gcp google_cloud plugin version 2017-02-16 16:49:16 +01:00
Kubernetes Submit Queue
627c6ce2b8 Merge pull request #41489 from Crassirostris/fluentd-add-toleration
Automatic merge from submit-queue (batch tested with PRs 40000, 41508, 41489)

Add toleration to fluentd daemonset to make it run on master

Because of https://github.com/kubernetes/kubernetes/pull/41172 fluentd pods stopped being allocated on master node.

This PR introduces toleration for master taint for fluentd.

CC @davidopp @janetkuo @kubernetes/sig-scheduling-bugs

Unfortunately, we don't have e2e tests to ensure that master logs are being ingested. This problem is a great signal to work on https://github.com/kubernetes/kubernetes/issues/41411
2017-02-16 01:52:08 -08:00
Mik Vyatskov
f6730bd334 Make fluentd use default dns instead of cluster dns to make it work on master 2017-02-15 20:53:32 +01:00
Mik Vyatskov
cbba60cc7d Add toleration to fluentd daemonset to make it run on master 2017-02-15 18:11:45 +01:00
Anthony Yeh
257a8745e3 fluentd-gcp: Add kube-apiserver-audit.log. 2017-02-14 14:23:36 -08:00
Mik Vyatskov
a1ec542d7c Fix copying systemd libraries upon fluentd-gcp startup 2017-02-14 15:41:15 +01:00
Vishnu kannan
10e7902a12 make fluentd a critical pod
Signed-off-by: Vishnu kannan <vishnuk@google.com>
2017-02-06 12:16:32 -08:00
Mik Vyatskov
7b194d496f Fix fluentd-gcp configuration to handle different timezones on the node 2017-01-24 11:53:15 +01:00
Antoine Pelisse
62af7dd33d OWNERS: Update latest OWNERS files
These files have been created lately, so we don't have much information
about them anyway, so let's just:
- Remove assignees and make them approvers
- Copy approves as reviewers
2017-01-23 10:05:48 -08:00
thomasschickinger
42fbf93fb0 Add rule for detecting exceptions to fluentd config for GKE logging.
Bump version of gcp-fluentd container to 1.34
2017-01-19 15:51:47 +01:00
Mik Vyatskov
83df5b8495 Remove library copying from fluentd image 2017-01-17 15:00:48 +01:00
Mik Vyatskov
edf1ffc074 Remove fluentd buffers if fluentd is stuck 2017-01-16 13:47:23 +01:00
Jeff Grafton
1c2ea28080 Update images that use ubuntu-slim base image to :0.6 2017-01-11 15:07:04 -08:00
Jeff Grafton
19aafd291c Always --pull in docker build to ensure recent base images 2017-01-10 16:21:05 -08:00
Piotr Szczesniak
da7b81c4d8 Added owners to monitoring and logging related directories 2017-01-10 12:14:10 +01:00
Mik Vyatskov
57ec7b77fd Fix fluentd-gcp image config by avoiding processing its own logs 2017-01-09 10:05:33 +01:00
Kubernetes Submit Queue
2d15499984 Merge pull request #39151 from Crassirostris/fluentd-gcp-default-format
Automatic merge from submit-queue

Try parse golang logs by default

Glog by default logs to stderr, so Stackdriver Logging shows them all as errors. This PR makes fluentd try to parse messages using glog format and if succeeded, set timestamp and severity accordingly.

CC @piosz @fgrzadkowski
2017-01-03 05:50:33 -08:00
Kubernetes Submit Queue
274a9f0f70 Merge pull request #38927 from luxas/remove_maintainer
Automatic merge from submit-queue

Remove all MAINTAINER statements in the codebase as they are deprecated

**What this PR does / why we need it**:
ref: https://github.com/docker/docker/pull/25466

**Release note**:

```release-note
Remove all MAINTAINER statements in Dockerfiles in the codebase as they are deprecated by docker
```
@ixdy @thockin (who else should be notified?)
2016-12-29 16:41:24 -08:00
Dawn Chen
9d3f4d7152 Revert "Make fluentd pods critical" 2016-12-22 15:58:36 -08:00
Kubernetes Submit Queue
0e9cb8f86b Merge pull request #39146 from Crassirostris/make-fluentd-critical
Automatic merge from submit-queue

Make fluentd pods critical

Related to https://github.com/kubernetes/kubernetes/issues/38322

Make fluentd critical so it will be evicted with less probability.

CC @piosz @fgrzadkowski
2016-12-22 05:17:25 -08:00
Mik Vyatskov
4a7b3d7528 Make fluentd pods critical 2016-12-21 19:56:46 +01:00
Mik Vyatskov
a3940ba874 Add liveness probe for fluentd-gcp 2016-12-21 17:32:28 +01:00
Mik Vyatskov
5105102940 Try parse golang logs by default 2016-12-20 17:31:13 +01:00
Lucas Käldström
3c5b5f5963 Remove all MAINTAINER statements in the codebase as they aren't very useful and now deprecated 2016-12-17 20:34:10 +02:00
Piotr Szczesniak
c00e57789d Added upgrade story from manifest pod to ds 2016-12-15 13:48:32 +01:00
Piotr Szczesniak
a52637f09f Migrated fluentd to daemon set 2016-12-15 13:48:32 +01:00
Mik Vyatskov
e52c3e77e2 Avoid exporting fluentd-gcp own logs 2016-12-14 14:43:05 +01:00
Mik Vyatskov
6215089796 Unify fluentd-gcp configurations 2016-11-30 14:47:24 +01:00
Mik Vyatskov
ba19a27974 Bump version of the fluentd-gcp image 2016-11-22 11:12:28 +01:00
Mik Vyatskov
b5b47620f5 Add kube-proxy logs to fluentd configs 2016-11-21 15:43:28 +01:00
Mik Vyatskov
aa31f7bc02 Bump fluentd-gcp image version 2016-11-17 21:21:10 +01:00
Mik Vyatskov
cf845eda7c Remove fixed docker daemon log time format in fluentd configs 2016-11-17 15:52:07 +01:00
Mik Vyatskov
279e20ed13 Fix flunetd-gcp image Dockerfile 2016-11-08 15:14:09 +01:00
Mik Vyatskov
d478307106 Fix config file names inside fluentd-gcp image 2016-11-07 20:31:12 +01:00
Kubernetes Submit Queue
04a81cdd3e Merge pull request #36363 from Crassirostris/fluentd-gcp-image-build-fix
Automatic merge from submit-queue

Fix fluentd-gcp Dockerfile to reduce image size

Change reduces image size by 150MB.

@piosz
2016-11-07 07:56:11 -08:00
Mik Vyatskov
d3465e5b8c Add rescheduler logs to the fluentd-gcp configuration 2016-11-07 15:10:52 +01:00
Mik Vyatskov
220168c9aa Fix fluentd-gcp Dockerfile to reduce image size 2016-11-07 15:01:38 +01:00
Mik Vyatskov
b4c7dfa674 Switched from google-fluentd to configuring fluentd from scratch 2016-11-02 12:08:36 +01:00
Mik Vyatskov
24361fce06 Update fluentd-gcp configuration 2016-11-02 12:06:01 +01:00
Jess Frazelle
cde5559d09
Update gcloud docker commands to use gcloud docker -- ARGS
We can then avoid the following warning:
```
WARNING: The '--' argument must be specified between gcloud specific args on the left and DOCKER_ARGS on the right. IMPORTANT: previously, commands allowed the omission of the --, and unparsed arguments were treated as implementation args. This usage is being deprecated and will be removed in March 2017.
This will be strictly enforced in March 2017. Use 'gcloud beta docker' to see new behavior.
```

Signed-off-by: Jess Frazelle <acidburn@google.com>
2016-10-10 13:42:34 -07:00
Manuel de Brito Fontes
e2c50152dd Remove kubernetes metadata plugin 2016-08-23 08:21:46 -03:00
Manuel de Brito Fontes
5753a3196c Reduce the size of fluentd images 2016-08-23 08:21:45 -03:00
Rick Buskens
77bda6fd16 Updated configuration to spawn multiple threads for processing 2016-08-01 15:08:11 -04:00
David McMahon
ef0c9f0c5b Remove "All rights reserved" from all the headers. 2016-06-29 17:47:36 -07:00
Alex Robinson
19bf9d0624 Support journal logs in fluentd-gcp.
Only run the systemd-journal plugin when on a platform that requests it.
The plugin crashes the fluentd process if the journal isn't present, so
it can't just be run blindly in all configurations.
2016-06-24 00:30:32 +00:00
Alex Robinson
bb51c94172 Add collection of the new glbc and cluster-autoscaler logs 2016-06-03 15:50:52 +00:00
Alex Robinson
3216204668 Update to the latest version of google-fluentd.
It includes some performance improvements for parsing JSON (which is
very important for us, since all Docker logs are JSON) as well as a
couple new settings, like forcing of a flush of multiline logs after a
time period rather than having to wait until a new log is seen before
feeling confident flushing the previous one.
2016-03-05 00:18:31 +00:00
Alex Robinson
29eb6020fb Merge pull request #22074 from a-robinson/images
Update the logging images to parse sub-second docker timestamps.
2016-03-04 09:04:33 -08:00
Alex Robinson
feea376b5c Update the logging images to parse sub-second docker timestamps. 2016-03-03 10:21:30 -08:00
Alex Robinson
f8965a6f41 Fix parsing of kube logs to handle logs split across lines.
I didn't expect glog to split single log statements onto multiple lines,
but apparently it does if they're long enough. This groups them back
together appropriately.
2016-02-26 22:58:46 +00:00