Skip to content

Commit 5d59291

Browse files
dhermesdhermes
authored
Merge pull request #2668 from waprin/newhandlers
Add GAE and GKE fluentd Handlers
2 parents 0e56db8 + 02ae299 commit 5d59291

18 files changed

+625
-96
lines changed

‎docs/index.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,8 @@
111111
logging-sink
112112
logging-stdlib-usage
113113
logging-handlers
114+
logging-handlers-app-engine
115+
logging-handlers-container-engine
114116
logging-transports-sync
115117
logging-transports-thread
116118
logging-transports-base

‎docs/logging-handlers-app-engine.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Google App Engine flexible Log Handler
2+
======================================
3+
4+
.. automodule:: google.cloud.logging.handlers.app_engine
5+
:members:
6+
:show-inheritance:

‎docs/logging-handlers-container-engine.rst‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
Google Container Engine Log Handler
2+
===================================
3+
4+
.. automodule:: google.cloud.logging.handlers.container_engine
5+
:members:
6+
:show-inheritance:

‎docs/logging-usage.rst‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -267,3 +267,108 @@ Delete a sink:
267267
:start-after: [START sink_delete]
268268
:end-before: [END sink_delete]
269269
:dedent: 4
270+
271+
Integration with Python logging module
272+
--------------------------------------
273+
274+
It's possible to tie the Python :mod:`logging` module directly into Google
275+
Stackdriver Logging. There are different handler options to accomplish this.
276+
To automatically pick the default for your current environment, use
277+
:meth:`~google.cloud.logging.client.Client.get_default_handler`.
278+
279+
.. literalinclude:: logging_snippets.py
280+
:start-after: [START create_default_handler]
281+
:end-before: [END create_default_handler]
282+
:dedent: 4
283+
284+
It is also possible to attach the handler to the root Python logger, so that
285+
for example a plain ``logging.warn`` call would be sent to Stackdriver Logging,
286+
as well as any other loggers created. A helper method
287+
:meth:`~google.cloud.logging.client.Client.setup_logging` is provided
288+
to configure this automatically.
289+
290+
.. literalinclude:: logging_snippets.py
291+
:start-after: [START setup_logging]
292+
:end-before: [END setup_logging]
293+
:dedent: 4
294+
295+
.. note::
296+
297+
To reduce cost and quota usage, do not enable Stackdriver logging
298+
handlers while testing locally.
299+
300+
You can also exclude certain loggers:
301+
302+
.. literalinclude:: logging_snippets.py
303+
:start-after: [START setup_logging_excludes]
304+
:end-before: [END setup_logging_excludes]
305+
:dedent: 4
306+
307+
Cloud Logging Handler
308+
=====================
309+
310+
If you prefer not to use
311+
:meth:`~google.cloud.logging.client.Client.get_default_handler`, you can
312+
directly create a
313+
:class:`~google.cloud.logging.handlers.handlers.CloudLoggingHandler` instance
314+
which will write directly to the API.
315+
316+
.. literalinclude:: logging_snippets.py
317+
:start-after: [START create_cloud_handler]
318+
:end-before: [END create_cloud_handler]
319+
:dedent: 4
320+
321+
.. note::
322+
323+
This handler by default uses an asynchronous transport that sends log
324+
entries on a background thread. However, the API call will still be made
325+
in the same process. For other transport options, see the transports
326+
section.
327+
328+
All logs will go to a single custom log, which defaults to "python". The name
329+
of the Python logger will be included in the structured log entry under the
330+
"python_logger" field. You can change it by providing a name to the handler:
331+
332+
.. literalinclude:: logging_snippets.py
333+
:start-after: [START create_named_handler]
334+
:end-before: [END create_named_handler]
335+
:dedent: 4
336+
337+
fluentd logging handlers
338+
========================
339+
340+
Besides :class:`~google.cloud.logging.handlers.handlers.CloudLoggingHandler`,
341+
which writes directly to the API, two other handlers are provided.
342+
:class:`~google.cloud.logging.handlers.app_engine.AppEngineHandler`, which is
343+
recommended when running on the Google App Engine Flexible vanilla runtimes
344+
(i.e. your app.yaml contains ``runtime: python``), and
345+
:class:`~google.cloud.logging.handlers.container_engine.ContainerEngineHandler`
346+
, which is recommended when running on `Google Container Engine`_ with the
347+
Stackdriver Logging plugin enabled.
348+
349+
:meth:`~google.cloud.logging.client.Client.get_default_handler` and
350+
:meth:`~google.cloud.logging.client.Client.setup_logging` will attempt to use
351+
the environment to automatically detect whether the code is running in
352+
these platforms and use the appropriate handler.
353+
354+
In both cases, the fluentd agent is configured to automatically parse log files
355+
in an expected format and forward them to Stackdriver logging. The handlers
356+
provided help set the correct metadata such as log level so that logs can be
357+
filtered accordingly.
358+
359+
Cloud Logging Handler transports
360+
=================================
361+
362+
The :class:`~google.cloud.logging.handlers.handlers.CloudLoggingHandler`
363+
logging handler can use different transports. The default is
364+
:class:`~google.cloud.logging.handlers.BackgroundThreadTransport`.
365+
366+
1. :class:`~google.cloud.logging.handlers.BackgroundThreadTransport` this is
367+
the default. It writes entries on a background
368+
:class:`python.threading.Thread`.
369+
370+
1. :class:`~google.cloud.logging.handlers.SyncTransport` this handler does a
371+
direct API call on each logging statement to write the entry.
372+
373+
374+
.. _Google Container Engine: https://cloud.google.com/container-engine/

‎docs/logging_snippets.py‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -324,6 +324,44 @@ def sink_pubsub(client, to_delete):
324324
to_delete.pop(0)
325325

326326

327+
@snippet
328+
def logging_handler(client):
329+
# [START create_default_handler]
330+
import logging
331+
handler = client.get_default_handler()
332+
cloud_logger = logging.getLogger('cloudLogger')
333+
cloud_logger.setLevel(logging.INFO)
334+
cloud_logger.addHandler(handler)
335+
cloud_logger.error('bad news')
336+
# [END create_default_handler]
337+
338+
# [START create_cloud_handler]
339+
from google.cloud.logging.handlers import CloudLoggingHandler
340+
handler = CloudLoggingHandler(client)
341+
cloud_logger = logging.getLogger('cloudLogger')
342+
cloud_logger.setLevel(logging.INFO)
343+
cloud_logger.addHandler(handler)
344+
cloud_logger.error('bad news')
345+
# [END create_cloud_handler]
346+
347+
# [START create_named_handler]
348+
handler = CloudLoggingHandler(client, name='mycustomlog')
349+
# [END create_named_handler]
350+
351+
352+
@snippet
353+
def setup_logging(client):
354+
import logging
355+
# [START setup_logging]
356+
client.setup_logging(log_level=logging.INFO)
357+
# [END setup_logging]
358+
359+
# [START setup_logging_excludes]
360+
client.setup_logging(log_level=logging.INFO,
361+
excluded_loggers=('werkzeug',))
362+
# [END setup_logging_excludes]
363+
364+
327365
def _line_no(func):
328366
return func.__code__.co_firstlineno
329367

‎logging/google/cloud/logging/client.py‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414

1515
"""Client for interacting with the Google Stackdriver Logging API."""
1616

17+
import logging
1718
import os
1819

1920
try:
@@ -34,6 +35,12 @@
3435
from google.cloud.logging._http import _LoggingAPI as JSONLoggingAPI
3536
from google.cloud.logging._http import _MetricsAPI as JSONMetricsAPI
3637
from google.cloud.logging._http import _SinksAPI as JSONSinksAPI
38+
from google.cloud.logging.handlers import CloudLoggingHandler
39+
from google.cloud.logging.handlers import AppEngineHandler
40+
from google.cloud.logging.handlers import ContainerEngineHandler
41+
from google.cloud.logging.handlers import setup_logging
42+
from google.cloud.logging.handlers.handlers import EXCLUDED_LOGGER_DEFAULTS
43+
3744
from google.cloud.logging.logger import Logger
3845
from google.cloud.logging.metric import Metric
3946
from google.cloud.logging.sink import Sink
@@ -42,6 +49,15 @@
4249
_DISABLE_GAX = os.getenv(DISABLE_GRPC, False)
4350
_USE_GAX = _HAVE_GAX and not _DISABLE_GAX
4451

52+
_APPENGINE_FLEXIBLE_ENV_VM = 'GAE_APPENGINE_HOSTNAME'
53+
"""Environment variable set in App Engine when vm:true is set."""
54+
55+
_APPENGINE_FLEXIBLE_ENV_FLEX = 'GAE_INSTANCE'
56+
"""Environment variable set in App Engine when env:flex is set."""
57+
58+
_CONTAINER_ENGINE_ENV = 'KUBERNETES_SERVICE'
59+
"""Environment variable set in a Google Container Engine environment."""
60+
4561

4662
class Client(JSONClient):
4763
"""Client to bundle configuration needed for API requests.
@@ -264,3 +280,40 @@ def list_metrics(self, page_size=None, page_token=None):
264280
"""
265281
return self.metrics_api.list_metrics(
266282
self.project, page_size, page_token)
283+
284+
def get_default_handler(self):
285+
"""Return the default logging handler based on the local environment.
286+
287+
:rtype: :class:`logging.Handler`
288+
:returns: The default log handler based on the environment
289+
"""
290+
if (_APPENGINE_FLEXIBLE_ENV_VM in os.environ or
291+
_APPENGINE_FLEXIBLE_ENV_FLEX in os.environ):
292+
return AppEngineHandler()
293+
elif _CONTAINER_ENGINE_ENV in os.environ:
294+
return ContainerEngineHandler()
295+
else:
296+
return CloudLoggingHandler(self)
297+
298+
def setup_logging(self, log_level=logging.INFO,
299+
excluded_loggers=EXCLUDED_LOGGER_DEFAULTS):
300+
"""Attach default Stackdriver logging handler to the root logger.
301+
302+
This method uses the default log handler, obtained by
303+
:meth:`~get_default_handler`, and attaches it to the root Python
304+
logger, so that a call such as ``logging.warn``, as well as all child
305+
loggers, will report to Stackdriver logging.
306+
307+
:type log_level: int
308+
:param log_level: (Optional) Python logging log level. Defaults to
309+
:const:`logging.INFO`.
310+
311+
:type excluded_loggers: tuple
312+
:param excluded_loggers: (Optional) The loggers to not attach the
313+
handler to. This will always include the
314+
loggers in the path of the logging client
315+
itself.
316+
"""
317+
handler = self.get_default_handler()
318+
setup_logging(handler, log_level=log_level,
319+
excluded_loggers=excluded_loggers)

‎logging/google/cloud/logging/handlers/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,5 +14,8 @@
1414

1515
"""Python :mod:`logging` handlers for Google Cloud Logging."""
1616

17+
from google.cloud.logging.handlers.app_engine import AppEngineHandler
18+
from google.cloud.logging.handlers.container_engine import (
19+
ContainerEngineHandler)
1720
from google.cloud.logging.handlers.handlers import CloudLoggingHandler
1821
from google.cloud.logging.handlers.handlers import setup_logging

‎logging/google/cloud/logging/handlers/_helpers.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# Copyright 2016 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Helper functions for logging handlers."""
16+
17+
import math
18+
import json
19+
20+
21+
def format_stackdriver_json(record, message):
22+
"""Helper to format a LogRecord in in Stackdriver fluentd format.
23+
24+
:rtype: str
25+
:returns: JSON str to be written to the log file.
26+
"""
27+
subsecond, second = math.modf(record.created)
28+
29+
payload = {
30+
'message': message,
31+
'timestamp': {
32+
'seconds': int(second),
33+
'nanos': int(subsecond * 1e9),
34+
},
35+
'thread': record.thread,
36+
'severity': record.levelname,
37+
}
38+
39+
return json.dumps(payload)

‎logging/google/cloud/logging/handlers/app_engine.py‎

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Copyright 2016 Google Inc. All Rights Reserved.
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
"""Logging handler for App Engine Flexible
16+
17+
Logs to the well-known file that the fluentd sidecar container on App Engine
18+
Flexible is configured to read from and send to Stackdriver Logging.
19+
20+
See the fluentd configuration here:
21+
22+
https://github.com/GoogleCloudPlatform/appengine-sidecars-docker/tree/master/fluentd_logger
23+
"""
24+
25+
# This file is largely copied from:
26+
# https://github.com/GoogleCloudPlatform/python-compat-runtime/blob/master
27+
# /appengine-vmruntime/vmruntime/cloud_logging.py
28+
29+
import logging.handlers
30+
import os
31+
32+
from google.cloud.logging.handlers._helpers import format_stackdriver_json
33+
34+
_LOG_PATH_TEMPLATE = '/var/log/app_engine/app.{pid}.json'
35+
_MAX_LOG_BYTES = 128 * 1024 * 1024
36+
_LOG_FILE_COUNT = 3
37+
38+
39+
class AppEngineHandler(logging.handlers.RotatingFileHandler):
40+
"""A handler that writes to the App Engine fluentd Stackdriver log file.
41+
42+
Writes to the file that the fluentd agent on App Engine Flexible is
43+
configured to discover logs and send them to Stackdriver Logging.
44+
Log entries are wrapped in JSON and with appropriate metadata. The
45+
process of converting the user's formatted logs into a JSON payload for
46+
Stackdriver Logging consumption is implemented as part of the handler
47+
itself, and not as a formatting step, so as not to interfere with
48+
user-defined logging formats.
49+
"""
50+
51+
def __init__(self):
52+
"""Construct the handler
53+
54+
Large log entries will get mangled if multiple workers write to the
55+
same file simultaneously, so we'll use the worker's PID to pick a log
56+
filename.
57+
"""
58+
self.filename = _LOG_PATH_TEMPLATE.format(pid=os.getpid())
59+
super(AppEngineHandler, self).__init__(self.filename,
60+
maxBytes=_MAX_LOG_BYTES,
61+
backupCount=_LOG_FILE_COUNT)
62+
63+
def format(self, record):
64+
"""Format the specified record into the expected JSON structure.
65+
66+
:type record: :class:`~logging.LogRecord`
67+
:param record: the log record
68+
69+
:rtype: str
70+
:returns: JSON str to be written to the log file
71+
"""
72+
message = super(AppEngineHandler, self).format(record)
73+
return format_stackdriver_json(record, message)

0 commit comments

Comments
 (0)