Skip to content

Commit 27321b3

Browse files
author
David Noble
committed
First complete documentation draft
All code elements intended for consumption by end-users are documented with correct restructured text markup and present as well as any of our other Python API documentation sets. Signed-off-by: David Noble <dnoble@splunk.com>
1 parent 7c23477 commit 27321b3

8 files changed

Lines changed: 360 additions & 276 deletions

File tree

‎docs/searchcommands.rst‎

Lines changed: 34 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,19 +3,51 @@ splunklib.searchcommands
33

44
.. automodule:: splunklib.searchcommands
55

6-
.. autofunction:: dispatch
6+
.. autofunction:: dispatch(command_class[, argv=sys.argv, input_file=sys.stdin, output_file=sys.stdout, module_name=None])
77

88
.. autoclass:: GeneratingCommand
99
:members:
1010
:inherited-members:
11+
:exclude-members: ConfigurationSettings, generate, process
12+
13+
.. autoclass:: splunklib.searchcommands::GeneratingCommand.ConfigurationSettings
14+
:members:
15+
:inherited-members:
16+
:exclude-members: configuration_settings, fix_up, items, keys
17+
18+
.. automethod:: splunklib.searchcommands::GeneratingCommand.generate
19+
20+
.. automethod:: splunklib.searchcommands::GeneratingCommand.process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout])
1121

1222
.. autoclass:: ReportingCommand
1323
:members:
1424
:inherited-members:
25+
:exclude-members: ConfigurationSettings, map, process, reduce
26+
27+
.. autoclass:: splunklib.searchcommands::ReportingCommand.ConfigurationSettings
28+
:members:
29+
:inherited-members:
30+
:exclude-members: configuration_settings, fix_up, items, keys
31+
32+
.. automethod:: splunklib.searchcommands::ReportingCommand.map
33+
34+
.. automethod:: splunklib.searchcommands::ReportingCommand.process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout])
35+
36+
.. automethod:: splunklib.searchcommands::ReportingCommand.reduce
1537

1638
.. autoclass:: StreamingCommand
1739
:members:
1840
:inherited-members:
41+
:exclude-members: ConfigurationSettings, process, stream
42+
43+
.. autoclass:: splunklib.searchcommands::StreamingCommand.ConfigurationSettings
44+
:members:
45+
:inherited-members:
46+
:exclude-members: configuration_settings, fix_up, items, keys
47+
48+
.. automethod:: splunklib.searchcommands::StreamingCommand.process(args=sys.argv[, input_file=sys.stdin, output_file=sys.stdout])
49+
50+
.. automethod:: splunklib.searchcommands::StreamingCommand.stream
1951

2052
.. autoclass:: Configuration
2153
:members:
@@ -24,6 +56,7 @@ splunklib.searchcommands
2456
.. autoclass:: Option
2557
:members:
2658
:inherited-members:
59+
:exclude-members: Item, View, fix_up
2760

2861
.. autoclass:: Boolean
2962
:members:

‎splunklib/searchcommands/__init__.py‎

Lines changed: 115 additions & 98 deletions
Original file line numberDiff line numberDiff line change
@@ -12,134 +12,131 @@
1212
# License for the specific language governing permissions and limitations
1313
# under the License.
1414

15-
""" Splunk search command library
15+
"""Design Notes
16+
---------------
1617
17-
#Design Notes
18+
1. Commands are constrained to this ABNF grammar::
1819
19-
1. Command lines are constrained to this ABNF grammar::
20+
command = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
21+
command-name = alpha *( alpha / digit )
22+
option = option-name [wsp] "=" [wsp] option-value
23+
option-name = alpha *( alpha / digit / "_" )
24+
option-value = word / quoted-string
25+
word = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
26+
quoted-string = dquote *( word / wsp / "\" dquote / dquote dquote ) dquote
27+
field-name = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )
2028
21-
command = command-name *[wsp option] *[wsp [dquote] field-name [dquote]]
22-
command-name = alpha *( alpha / digit )
23-
option = option-name [wsp] "=" [wsp] option-value
24-
option-name = alpha *( alpha / digit / "_" )
25-
option-value = word / quoted-string
26-
word = 1*( %01-%08 / %0B / %0C / %0E-1F / %21 / %23-%FF ) ; Any character but DQUOTE and WSP
27-
quoted-string = dquote *( word / wsp / "\" dquote / dquote dquote ) dquote
28-
field-name = ( "_" / alpha ) *( alpha / digit / "_" / "." / "-" )
29+
It is Constrained to an 8-bit character set. It does not show that
30+
:code:`field-name` values may be comma-separated. This is because Splunk strips
31+
commas from the command line. A search command will never see them.
2932
30-
**Note:**
33+
3. Commands must be statically configured as follows:
3134
32-
This grammar is constrained to an 8-bit character set.
35+
.. code-block:: text
36+
:linenos:
3337
34-
**Note:**
38+
[commandname]
39+
filename = commandname.py
40+
supports_getinfo = true
41+
supports_rawargs = true
3542
36-
This grammar does not show that `field-name` values may be comma-separated
37-
when in fact they may be. This is because Splunk strips commas from the
38-
command line. A custom search command will never see them.
43+
No other static configuration is required or expected and may interfere with
44+
command execution.
3945
4046
2. Commands support dynamic probing for settings.
4147
42-
Splunk probes for settings dynamically when `supports_getinfo=true`.
43-
44-
3. Commands do not support static probing for settings.
45-
46-
This module expects that commands are statically configured as follows::
47-
48-
[<command-name>]
49-
filename = <command-name>.py
50-
supports_getinfo = true
51-
52-
No other static configuration is required or expected and may interfere
53-
with command execution.
48+
Splunk probes for settings dynamically when :code:`supports_getinfo=true`.
49+
You must add this line to the commands.conf stanza for each of your search
50+
commands.
5451
5552
4. Commands do not support parsed arguments on the command line.
5653
57-
Splunk parses arguments when `supports_rawargs=false`. This ``SearchCommand``
58-
class sets this value unconditionally. You cannot override it.
54+
Splunk parses arguments when :code:`supports_rawargs=false`. The
55+
:code:`SearchCommand` class sets this value unconditionally. You cannot
56+
override it.
5957
60-
**Rationale:**
58+
**Rationale**
6159
6260
Splunk parses arguments by stripping quotes, nothing more. This may be useful
6361
in some cases, but doesn't work well with our chosen grammar.
6462
6563
5. Commands consume input headers.
6664
67-
An input header is provided by Splunk when `enableheader=true`. The
68-
``SearchCommand`` class sets this value unconditionally. You cannot override
69-
it.
65+
An input header is provided by Splunk when :code:`enableheader=true`. The
66+
:class:`SearchCommand` class sets this value unconditionally. You cannot
67+
override it.
7068
7169
6. Commands produce an output messages header.
7270
7371
Splunk expects a command to produce an output messages header when
74-
`outputheader=true`. The ``SearchCommand`` class sets this value
72+
:code:`outputheader=true`. The :class:`SearchCommand` class sets this value
7573
unconditionally. You cannot override it.
7674
7775
7. Commands support multi-value fields.
7876
7977
Multi-value fields are provided and consumed by Splunk when
80-
`supports_multivalue=true`. The ``SearchCommand`` class sets this value
81-
unconditionally. You cannot override it.
78+
:code:`supports_multivalue=true`. This value is fixed. You cannot override
79+
it.
8280
8381
8. Commands represent all fields on the output stream as multi-value fields.
8482
8583
Splunk represents multi-value fields with a pair of fields:
8684
87-
+ `<field-name>`
85+
======================== ====================================================
86+
field-name Contains the text from which the multi-value field
87+
was derived.
8888
89-
Contains the text from which the multi-value field was derived.
89+
:code:`__mv_field-name` Contains an encoded list. Values in the list are
90+
wrapped in dollar signs ($) and separated by
91+
semi-colons (;). Dollar signs ($) within a value are
92+
represented by a pair of dollar signs ($$). Empty
93+
lists are represented by the empty string.
94+
Single-value lists are represented by the single
95+
value.
96+
======================== ====================================================
9097
91-
+ `__mv_<field-name>`
98+
On input this class processes and hides all :code:`__mv_` fields. On output
99+
backing :code:`__mv_` fields are produced for all fields, thereby enabling a
100+
command to reduce its memory footprint by using streaming I/O. This is done
101+
at the cost of one extra byte of data per field per record on the output
102+
stream and some extra processing time by the next processor in the pipeline.
92103
93-
Contains an encoded list. Values in the list are wrapped in dollar
94-
signs ('$') and separated by semi-colons (';). Dollar signs ('$')
95-
within a value are represented by a pair of dollar signs ('$$').
96-
Empty lists are represented by the empty string. Single-value lists
97-
are represented by the single value.
104+
9. A :class:`ReportingCommand` must override :meth:`~ReportingCommand.reduce`
105+
and may override :meth:`~ReportingCommand.map`. Map/reduce commands on the
106+
Splunk processing pipeline are distinguished as this example illustrates.
98107
99-
On input this class processes and hides all **__mv_** fields. On output
100-
this class produces backing **__mv_** fields for all fields, thereby
101-
enabling a command to reduce its memory footprint by using streaming
102-
I/O. This is done at the cost of one extra byte of data per field per
103-
record on the output stream and extra processing time by the next
104-
processor in the pipeline.
108+
**Splunk command**
105109
106-
9. A ReportingCommand may implement a `map` method (a.k.a, a streaming preop)
107-
and must implement a `reduce` operation (a.k.a., a reporting operation).
110+
.. code-block:: text
108111
109-
Map/reduce command lines are distinguished by this module as exemplified
110-
here:
112+
sum total=total_date_hour date_hour
111113
112-
**Command**::
114+
**Map command line**
113115
114-
...| sum total=total_date_hour date_hour
116+
.. code-block:: text
115117
116-
**Reduce command line**::
118+
sum __GETINFO__ __map__ total=total_date_hour date_hour
119+
sum __EXECUTE__ __map__ total=total_date_hour date_hour
117120
118-
sum __GETINFO__ total=total_date_hour date_hour
119-
sum __EXECUTE__ total=total_date_hour date_hour
121+
**Reduce command line**
120122
121-
**Map command line**::
123+
.. code-block:: text
122124
123-
sum __GETINFO__ __map__ total=total_date_hour date_hour
124-
sum __EXECUTE__ __map__ total=total_date_hour date_hour
125+
sum __GETINFO__ total=total_date_hour date_hour
126+
sum __EXECUTE__ total=total_date_hour date_hour
125127
126-
The `__map__` argument is introduced by the `ReportingCommand._execute`
127-
method. ReportingCommand authors cannot influence the contents of the
128-
command line in this release.
128+
The :code:`__map__` argument is introduced by
129+
:meth:`ReportingCommand._execute`. Search command authors cannot influence
130+
the contents of the command line in this release.
129131
130-
#References
132+
References
133+
----------
131134
132-
1. [Commands.conf.spec](http://docs.splunk.com/Documentation/Splunk/5.0.5/Admin/Commandsconf)
133-
2. [Search command style guide](http://docs.splunk.com/Documentation/Splunk/6.0/Search/Searchcommandstyleguide)
134-
135-
"""
135+
1. `Search command style guide <http://docs.splunk.com/Documentation/Splunk/6.0/Search/Searchcommandstyleguide>`_
136136
137-
# TODO: Do not use [defaults] stanza in commands.conf
138-
# It doesn't work as advertised. Make sure the ERD/PRD/User documentation is
139-
# updated. The source code already is.
137+
2. `Commands.conf.spec <http://docs.splunk.com/Documentation/Splunk/5.0.5/Admin/Commandsconf>`_
140138
141-
# TODO: Multi-line message headers are incorrectly formatted in Splunk Web UI
142-
# What should we be using as a newline terminator? Try url encoding.
139+
"""
143140

144141
from __future__ import absolute_import
145142

@@ -158,46 +155,66 @@ class sets this value unconditionally. You cannot override it.
158155
import os
159156
msvcrt.setmode(sys.stdout.fileno(), os.O_BINARY)
160157

161-
162158
def dispatch(command_class, argv=sys.argv, input_file=sys.stdin, output_file=
163159
sys.stdout, module_name=None):
164160
""" Instantiates and executes a search command class
165161
166-
This function implements a [conditional script stanza](http://goo.gl/OFaox6)
167-
based on the value of `module_name`::
162+
This function implements a `conditional script stanza <http://goo.gl/OFaox6>`_
163+
based on the value of :code:`module_name`::
168164
169165
if module_name is None or module_name == '__main__':
170166
# execute command
171167
172-
If you would like this function's caller to act as either a reusable module
173-
or a standalone program, call it at module scope with `__name__` as the
174-
value of `module_name`. Otherwise, if you wish this function to
175-
unconditionally instantiate and execute `command_class`, pass `None` as the
176-
value of `module_name`.
168+
Call this function at module scope with :code:`module_name=__name__`, if you
169+
would like your module to act as either a reusable module or a standalone
170+
program. Otherwise, if you wish this function to unconditionally instantiate
171+
and execute :code:`command_class`, pass :const:`None` as the value of
172+
:code:`module_name`.
177173
178174
:param command_class: Class to instantiate and execute.
179-
:type command_class: ``.search_command.SearchCommand``
175+
:type command_class: :code:`SearchCommand`
180176
:param argv: List of arguments to the command.
181-
:type argv: ``list``
177+
:type argv: :code:`list`
182178
:param input_file: File from which the command will read data.
183-
:type input_file: ``file``
179+
:type input_file: :code:`file`
184180
:param output_file: File to which the command will write data.
185-
:type output_file: ``file``
186-
:param module_name: Name of the module calling dispatch or `None`.
187-
:type module_name: ``str``
188-
:returns: ``None``
181+
:type output_file: :code:`file`
182+
:param module_name: Name of the module calling :code:`dispatch` or :const:`None`.
183+
:type module_name: :code:`str`
184+
:returns: :const:`None`
189185
190-
**Example**::
186+
**Example**
187+
188+
.. code-block:: python
189+
:linenos:
191190
192191
#!/usr/bin/env python
193-
...
194-
class CountMatchesCommand(StreamingCommand):
192+
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
193+
@Configuration()
194+
class SomeStreamingCommand(StreamingCommand):
195195
...
196+
def stream(records):
197+
...
198+
dispatch(SomeStreamingCommand, module_name=__name__)
199+
200+
Dispatches the :code:`SomeStreamingCommand`, if and only if
201+
:code:`__name__` is equal to :code:`'__main__'`.
196202
197-
dispatch(CountMatchesCommand, module_name=__name__)
198203
199-
Dispatches the `CountMatchesCommand`, if and only if `__name__` is equal to
200-
`__main__`.
204+
**Example**
205+
206+
.. code-block:: python
207+
:linenos:
208+
209+
from splunklib.searchcommands import dispatch, StreamingCommand, Configuration, Option, validators
210+
@Configuration()
211+
class SomeStreamingCommand(StreamingCommand):
212+
...
213+
def stream(records):
214+
...
215+
dispatch(SomeStreamingCommand)
216+
217+
Unconditionally dispatches :code:`SomeStreamingCommand`.
201218
202219
"""
203220
if module_name is not None and module_name != '__main__':

0 commit comments

Comments
 (0)