Skip to content

Commit c132584

Browse files
author
Bill Prin
committed
Add GAE and GKE fluentd Handlers
On GAE and GKE with the plugin installed, there is a fluentd plugin that collects logs from files. However without the right formatting, metadata like log_level is lost. Furthermore, the fluentd agents are configured to set the correct resources types, which could be done in the main handler as well, but it’s easier to rely on the fluentd configurations. This adds two new handlers and some helper functions to detect when they should be used.
1 parent b92f0cf commit c132584

19 files changed

Lines changed: 594 additions & 98 deletions

‎docs/index.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,8 @@
111111
logging-sink
112112
logging-stdlib-usage
113113
logging-handlers
114+
logging-handlers-app-engine
115+
logging-handlers-container-engine
114116
logging-transports-sync
115117
logging-transports-thread
116118
logging-transports-base

‎docs/logging-handlers-app-engine.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Google App Engine flexible Log Handler
2+
======================================
3+
4+
.. automodule:: google.cloud.logging.handlers.app_engine
5+
:members:
6+
:show-inheritance:

‎docs/logging-handlers-container-engine.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Google Container Engine Log Handler
2+
===================================
3+
4+
.. automodule:: google.cloud.logging.handlers.container_engine
5+
:members:
6+
:show-inheritance:

‎docs/logging-usage.rst‎

Lines changed: 119 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,3 +267,122 @@ Delete a sink:
267267
:start-after: [START sink_delete]
268268
:end-before: [END sink_delete]
269269
:dedent: 4
270+
Integration with Python logging module
271+
--------------------------------------
272+
273+
It's possible to tie the Python :mod:`logging` module directly into Google
274+
Stackdriver Logging. There are different handler options to accomplish this.
275+
To automatically pick the default for your current environment, use
276+
:meth:`~google.cloud.logging.client.Client.get_default_handler`.
277+
278+
.. code-block:: python
279+
280+
>>> import logging
281+
>>> import google.cloud.logging # Don't conflict with standard logging
282+
>>> from google.cloud.logging.handlers import CloudLoggingHandler
283+
>>> client = google.cloud.logging.Client()
284+
>>> handler = client.get_default_handler()
285+
>>> cloud_logger = logging.getLogger('cloudLogger')
286+
>>> cloud_logger.setLevel(logging.INFO)
287+
>>> cloud_logger.addHandler(handler)
288+
>>> cloud_logger.error('bad news')
289+
290+
It is also possible to attach the handler to the root Python logger, so that
291+
for example a plain ``logging.warn`` call would be sent to Stackdriver Logging,
292+
as well as any other loggers created. A helper method
293+
:meth:`~google.cloud.logging.client.Client.setup_logging` is provided
294+
to configure this automatically.
295+
296+
.. code-block:: python
297+
298+
>>> import logging
299+
>>> import google.cloud.logging # Don't conflict with standard logging
300+
>>> client = google.cloud.logging.Client()
301+
>>> client.setup_logging(log_level=logging.INFO)
302+
303+
.. note::
304+
305+
To reduce cost and quota usage, do not enable Stackdriver logging
306+
handlers while testing locally.
307+
308+
You can also exclude certain loggers:
309+
310+
.. code-block:: python
311+
312+
>>> client.setup_logging(log_level=logging.INFO,
313+
excluded_loggers=('werkzeug',))
314+
315+
Cloud Logging Handler
316+
=====================
317+
318+
If you prefer not to use
319+
:meth:`~google.cloud.logging.client.Client.get_default_handler`, you can
320+
directly create a
321+
:class:`~google.cloud.logging.handlers.handlers.CloudLoggingHandler` instance
322+
which will write directly to the API.
323+
324+
.. code-block:: python
325+
326+
>>> import logging
327+
>>> import google.cloud.logging
328+
>>> from google.cloud.logging.handlers import CloudLoggingHandler
329+
>>> client = google.cloud.logging.Client()
330+
>>> handler = CloudLoggingHandler(client)
331+
>>> cloud_logger = logging.getLogger('cloudLogger')
332+
>>> cloud_logger.setLevel(logging.INFO)
333+
>>> cloud_logger.addHandler(handler)
334+
>>> cloud_logger.error('bad news')
335+
336+
.. note::
337+
338+
This handler by default uses an asynchronous transport that sends log
339+
entries on a background thread. However, the API call will still be made
340+
in the same process. For other transport options, see the transports
341+
section.
342+
343+
All logs will go to a single custom log, which defaults to "python". The name
344+
of the Python logger will be included in the structured log entry under the
345+
"python_logger" field. You can change it by providing a name to the handler:
346+
347+
.. code-block:: python
348+
349+
>>> handler = CloudLoggingHandler(client, name='mycustomlog')
350+
351+
fluentd logging handlers
352+
========================
353+
354+
Besides :class:`~google.cloud.logging.handlers.handlers.CloudLoggingHandler`,
355+
which writes directly to the API, two other handlers are provided.
356+
:class:`~google.cloud.logging.handlers.app_engine.AppEngineHandler`, which is
357+
recommended when running on the Google App Engine Flexible vanilla runtimes
358+
(i.e. your app.yaml contains ``runtime: python``), and
359+
:class:`~google.cloud.logging.handlers.container_engine.ContainerEngineHandler`
360+
, which is recommended when running on `Google Container Engine`_ with the
361+
Stackdriver Logging plugin enabled.
362+
363+
:meth:`~google.cloud.logging.client.Client.get_default_handler` and
364+
:meth:`~google.cloud.logging.client.Client.setup_logging` will attempt to use
365+
the environment to automatically detect whether the code is running in
366+
these platforms and use the appropriate handler.
367+
368+
In both cases, the fluentd agent is configured to automatically parse log files
369+
in an expected format and forward them to Stackdriver logging. The handlers
370+
provided help set the correct metadata such as log level so that logs can be
371+
filtered accordingly.
372+
373+
Cloud Logging Handler transports
374+
=================================
375+
376+
The :class:`~google.cloud.logging.handlers.handlers.CloudLoggingHandler`
377+
logging handler can use different transports. The default is
378+
:class:`~google.cloud.logging.handlers.BackgroundThreadTransport`.
379+
380+
1. :class:`~google.cloud.logging.handlers.BackgroundThreadTransport` this is
381+
the default. It writes entries on a background
382+
:class:`python.threading.Thread`.
383+
384+
1. :class:`~google.cloud.logging.handlers.SyncTransport` this handler does a
385+
direct API call on each logging statement to write the entry.
386+
387+
388+
.. _Google Container Engine: https://cloud.google.com/container-engine/

‎logging/google/cloud/logging/__init__.py‎

Lines changed: 0 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,6 @@
1515
"""Google Stackdriver Logging API wrapper."""
1616

1717

18-
from google.cloud.logging.client import Client
19-
20-
2118
ASCENDING = 'timestamp asc'
2219
"""Query string to order by ascending timestamps."""
2320
DESCENDING = 'timestamp desc'

‎logging/google/cloud/logging/client.py‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414

1515
"""Client for interacting with the Google Stackdriver Logging API."""
1616

17+
import logging
1718
import os
1819

1920
try:
@@ -34,6 +35,12 @@
3435
from google.cloud.logging._http import _LoggingAPI as JSONLoggingAPI
3536
from google.cloud.logging._http import _MetricsAPI as JSONMetricsAPI
3637
from google.cloud.logging._http import _SinksAPI as JSONSinksAPI
38+
from google.cloud.logging.handlers import CloudLoggingHandler
39+
from google.cloud.logging.handlers import AppEngineHandler
40+
from google.cloud.logging.handlers import ContainerEngineHandler
41+
from google.cloud.logging.handlers import setup_logging
42+
from google.cloud.logging.handlers.handlers import EXCLUDED_LOGGER_DEFAULTS
43+
3744
from google.cloud.logging.logger import Logger
3845
from google.cloud.logging.metric import Metric
3946
from google.cloud.logging.sink import Sink
@@ -42,6 +49,15 @@
4249
_DISABLE_GAX = os.getenv(DISABLE_GRPC, False)
4350
_USE_GAX = _HAVE_GAX and not _DISABLE_GAX
4451

52+
_APPENGINE_FLEXIBLE_ENV_VM = 'GAE_APPENGINE_HOSTNAME'
53+
"""Environment variable set in App Engine when vm:true is set."""
54+
55+
_APPENGINE_FLEXIBLE_ENV_FLEX = 'GAE_INSTANCE'
56+
"""Environment variable set in App Engine when env:flex is set."""
57+
58+
_CONTAINER_ENGINE_ENV = 'KUBERNETES_SERVICE'
59+
"""Environment variable set in a Google Container Engine environment."""
60+
4561

4662
class Client(JSONClient):
4763
"""Client to bundle configuration needed for API requests.
@@ -264,3 +280,40 @@ def list_metrics(self, page_size=None, page_token=None):
264280
"""
265281
return self.metrics_api.list_metrics(
266282
self.project, page_size, page_token)
283+
284+
def get_default_handler(self):
285+
"""Return the default logging handler based on the local environment.
286+
287+
:rtype: :class:`logging.Handler`
288+
:returns: The default log handler based on the environment
289+
"""
290+
if (_APPENGINE_FLEXIBLE_ENV_VM in os.environ or
291+
_APPENGINE_FLEXIBLE_ENV_FLEX in os.environ):
292+
return AppEngineHandler()
293+
elif _CONTAINER_ENGINE_ENV in os.environ:
294+
return ContainerEngineHandler()
295+
else:
296+
return CloudLoggingHandler(self)
297+
298+
def setup_logging(self, log_level=logging.INFO,
299+
excluded_loggers=EXCLUDED_LOGGER_DEFAULTS):
300+
"""Attach default Stackdriver logging handler to the root logger.
301+
302+
This method uses the default log handler, obtained by
303+
:meth:`~get_default_handler`, and attaches it to the root Python
304+
logger, so that a call such as ``logging.warn``, as well as all child
305+
loggers, will report to Stackdriver logging.
306+
307+
:type log_level: int
308+
:param log_level: (Optional) Python logging log level. Defaults to
309+
:const:`logging.INFO`.
310+
311+
:type excluded_loggers: tuple
312+
:param excluded_loggers: (Optional) The loggers to not attach the
313+
handler to. This will always include the
314+
loggers in the path of the logging client
315+
itself.
316+
"""
317+
handler = self.get_default_handler()
318+
setup_logging(handler, log_level=log_level,
319+
excluded_loggers=excluded_loggers)

‎logging/google/cloud/logging/handlers/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,5 +14,8 @@
1414

1515
"""Python :mod:`logging` handlers for Google Cloud Logging."""
1616

17+
from google.cloud.logging.handlers.app_engine import AppEngineHandler
18+
from google.cloud.logging.handlers.container_engine import (
19+
ContainerEngineHandler)
1720
from google.cloud.logging.handlers.handlers import CloudLoggingHandler
1821
from google.cloud.logging.handlers.handlers import setup_logging

‎logging/google/cloud/logging/handlers/_helpers.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Copyright 2016 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Helper functions for logging handlers."""
16+
17+
import math
18+
import json
19+
20+
21+
def format_stackdriver_json(record, message):
22+
"""Helper to format a LogRecord in in Stackdriver fluentd format.
23+
24+
:rtype: str
25+
:returns: JSON str to be written to the log file.
26+
"""
27+
subsecond, second = math.modf(record.created)
28+
29+
payload = {
30+
'message': message,
31+
'timestamp': {
32+
'seconds': int(second),
33+
'nanos': int(subsecond * 1e9),
34+
},
35+
'thread': record.thread,
36+
'severity': record.levelname,
37+
}
38+
39+
return json.dumps(payload)

‎logging/google/cloud/logging/handlers/app_engine.py‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
# Copyright 2016 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Logging handler for App Engine Flexible
16+
17+
Logs to the well-known file that the fluentd sidecar container on App Engine
18+
Flexible is configured to read from and send to Stackdriver Logging.
19+
20+
See the fluentd configuration here:
21+
22+
https://github.com/GoogleCloudPlatform/appengine-sidecars-docker/tree/master/fluentd_logger
23+
"""
24+
25+
# This file is largely copied from:
26+
# https://github.com/GoogleCloudPlatform/python-compat-runtime/blob/master
27+
# /appengine-vmruntime/vmruntime/cloud_logging.py
28+
29+
import logging.handlers
30+
import os
31+
32+
from google.cloud.logging.handlers._helpers import format_stackdriver_json
33+
34+
_LOG_PATH_TEMPLATE = '/var/log/app_engine/app.{pid}.json'
35+
_MAX_LOG_BYTES = 128 * 1024 * 1024
36+
_LOG_FILE_COUNT = 3
37+
38+
39+
class AppEngineHandler(logging.handlers.RotatingFileHandler):
40+
"""A handler that writes to the App Engine fluentd Stackdriver log file.
41+
42+
Writes to the file that the fluentd agent on App Engine Flexbile is
43+
configured to discover logs and send them to Stackdriver Logging.
44+
Log entries are wrapped in JSON and with appropriate metadata. The
45+
process of converting the user's formatted logs into a JSON payload for
46+
Stackdriver Logging consumption is implemented as part of the handler
47+
itself, and not as a formatting step, so as not to interfere with
48+
user-defined logging formats.
49+
50+
Logging calls can also alternatively 'trace_id' in as a field in the
51+
'extra' dict, which will be used preferentially to fill out the Trace ID
52+
metadata.
53+
"""
54+
55+
def __init__(self):
56+
"""Construct the handler
57+
58+
Large log entries will get mangled if multiple workers write to the
59+
same file simultaneously, so we'll use the worker's PID to pick a log
60+
filename.
61+
"""
62+
self.filename = _LOG_PATH_TEMPLATE.format(pid=os.getpid())
63+
super(AppEngineHandler, self).__init__(self.filename,
64+
maxBytes=_MAX_LOG_BYTES,
65+
backupCount=_LOG_FILE_COUNT)
66+
67+
def format(self, record):
68+
"""Format the specified record into the expected JSON structure.
69+
70+
:type record: :class:`~logging.LogRecord`
71+
:param record: the log record
72+
73+
:rtype: str
74+
:returns: JSON str to be written to the log file
75+
"""
76+
message = super(AppEngineHandler, self).format(record)
77+
return format_stackdriver_json(record, message)

0 commit comments

Comments
 (0)