kubernetes - Google Cloud Logging + google-fluentd Dropping Messages -
i have rather small (1-2 node) kubernetes cluster running in gke ±40 pods running. problem @ hand it's not logging gce console properly. see lots of messages fluentd container(s) in following format:
$ kubectl logs fluentd-cloud-logging-gke-xxxxxxxx-node-xxxx 2016-02-02 23:30:09 +0000 [warn]: dropping 10 log message(s) error_class="google::apiclient::clienterror" error="project has not enabled api. please use google developers console activate 'logging' api project." 2016-02-02 23:30:09 +0000 [warn]: dropping 1 log message(s) error_class="google::apiclient::clienterror" error="project has not enabled api. please use google developers console activate 'logging' api project." 2016-02-02 23:30:09 +0000 [warn]: dropping 3 log message(s) error_class="google::apiclient::clienterror" error="project has not enabled api. please use google developers console activate 'logging' api project." 2016-02-02 23:30:09 +0000 [warn]: dropping 41 log message(s) error_class="google::apiclient::clienterror" error="project has not enabled api. please use google developers console activate 'logging' api project." 2016-02-02 23:30:09 +0000 [warn]: dropping 5 log message(s) error_class="google::apiclient::clienterror" error="project has not enabled api. please use google developers console activate 'logging' api project."
...and on. i'm seeing ~5 of these messages per second, know things producing logs. however, in compute engine console see following:
so somewhere in between i'm loosing lots of messages. strange though, i'm not loosing all these messages!
the cluster configured logging.write , monitoring.all privileges suggested in gh issue #15727
it's confusing of logs showing up. given error message, i'd expect none of logs showing in viewer, since sounds logging api hasn't been enabled project yet.
you can developers console, here. try going there, clicking enable api
button, , seeing whether errors keep coming.
Comments
Post a Comment