Skip to content

Commit dda36f5

Browse files
dhermesdhermes
committed
Moving all Bigtable row filters into their own module.
In the process also lowering the line limits to levels before they were raised to accomodate the final changes to the Bigtable Row implementations.
1 parent 72617b3 commit dda36f5

16 files changed

Lines changed: 1875 additions & 1831 deletions

‎docs/bigtable-row-filters.rst‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
Bigtable Row Filters
2+
====================
3+
4+
It is possible to use a
5+
:class:`RowFilter <gcloud.bigtable.row_filters.RowFilter>`
6+
when adding mutations to a
7+
:class:`ConditionalRow <gcloud.bigtable.row.ConditionalRow>` and when
8+
reading row data with :meth:`read_row() <gcloud.bigtable.table.Table.read_row>`
9+
:meth:`read_rows() <gcloud.bigtable.table.Table.read_rows>`.
10+
11+
As laid out in the `RowFilter definition`_, the following basic filters
12+
are provided:
13+
14+
* :class:`SinkFilter <.row_filters.SinkFilter>`
15+
* :class:`PassAllFilter <.row_filters.PassAllFilter>`
16+
* :class:`BlockAllFilter <.row_filters.BlockAllFilter>`
17+
* :class:`RowKeyRegexFilter <.row_filters.RowKeyRegexFilter>`
18+
* :class:`RowSampleFilter <.row_filters.RowSampleFilter>`
19+
* :class:`FamilyNameRegexFilter <.row_filters.FamilyNameRegexFilter>`
20+
* :class:`ColumnQualifierRegexFilter <.row_filters.ColumnQualifierRegexFilter>`
21+
* :class:`TimestampRangeFilter <.row_filters.TimestampRangeFilter>`
22+
* :class:`ColumnRangeFilter <.row_filters.ColumnRangeFilter>`
23+
* :class:`ValueRegexFilter <.row_filters.ValueRegexFilter>`
24+
* :class:`ValueRangeFilter <.row_filters.ValueRangeFilter>`
25+
* :class:`CellsRowOffsetFilter <.row_filters.CellsRowOffsetFilter>`
26+
* :class:`CellsRowLimitFilter <.row_filters.CellsRowLimitFilter>`
27+
* :class:`CellsColumnLimitFilter <.row_filters.CellsColumnLimitFilter>`
28+
* :class:`StripValueTransformerFilter <.row_filters.StripValueTransformerFilter>`
29+
* :class:`ApplyLabelFilter <.row_filters.ApplyLabelFilter>`
30+
31+
In addition, these filters can be combined into composite filters with
32+
33+
* :class:`RowFilterChain <.row_filters.RowFilterChain>`
34+
* :class:`RowFilterUnion <.row_filters.RowFilterUnion>`
35+
* :class:`ConditionalRowFilter <.row_filters.ConditionalRowFilter>`
36+
37+
These rules can be nested arbitrarily, with a basic filter at the lowest
38+
level. For example:
39+
40+
.. code:: python
41+
42+
# Filter in a specified column (matching any column family).
43+
col1_filter = ColumnQualifierRegexFilter(b'columnbia')
44+
45+
# Create a filter to label results.
46+
label1 = u'label-red'
47+
label1_filter = ApplyLabelFilter(label1)
48+
49+
# Combine the filters to label all the cells in columnbia.
50+
chain1 = RowFilterChain(filters=[col1_filter, label1_filter])
51+
52+
# Create a similar filter to label cells blue.
53+
col2_filter = ColumnQualifierRegexFilter(b'columnseeya')
54+
label2 = u'label-blue'
55+
label2_filter = ApplyLabelFilter(label2)
56+
chain2 = RowFilterChain(filters=[col2_filter, label2_filter])
57+
58+
# Bring our two labeled columns together.
59+
row_filter = RowFilterUnion(filters=[chain1, chain2])
60+
61+
----
62+
63+
.. automodule:: gcloud.bigtable.row_filters
64+
:members:
65+
:undoc-members:
66+
:show-inheritance:
67+
68+
.. _RowFilter definition: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/1ff247c2e3b7cd0a2dd49071b2d95beaf6563092/bigtable-protos/src/main/proto/google/bigtable/v1/bigtable_data.proto#L195

‎docs/bigtable-row.rst‎

Lines changed: 0 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -1,67 +1,7 @@
11
Bigtable Row
22
============
33

4-
It is possible to use a :class:`RowFilter <gcloud.bigtable.row.RowFilter>`
5-
when adding mutations to a
6-
:class:`ConditionalRow <gcloud.bigtable.row.ConditionalRow>` and when
7-
reading row data with :meth:`read_row() <gcloud.bigtable.table.Table.read_row>`
8-
:meth:`read_rows() <gcloud.bigtable.table.Table.read_rows>`.
9-
10-
As laid out in the `RowFilter definition`_, the following basic filters
11-
are provided:
12-
13-
* :class:`SinkFilter <.row.SinkFilter>`
14-
* :class:`PassAllFilter <.row.PassAllFilter>`
15-
* :class:`BlockAllFilter <.row.BlockAllFilter>`
16-
* :class:`RowKeyRegexFilter <.row.RowKeyRegexFilter>`
17-
* :class:`RowSampleFilter <.row.RowSampleFilter>`
18-
* :class:`FamilyNameRegexFilter <.row.FamilyNameRegexFilter>`
19-
* :class:`ColumnQualifierRegexFilter <.row.ColumnQualifierRegexFilter>`
20-
* :class:`TimestampRangeFilter <.row.TimestampRangeFilter>`
21-
* :class:`ColumnRangeFilter <.row.ColumnRangeFilter>`
22-
* :class:`ValueRegexFilter <.row.ValueRegexFilter>`
23-
* :class:`ValueRangeFilter <.row.ValueRangeFilter>`
24-
* :class:`CellsRowOffsetFilter <.row.CellsRowOffsetFilter>`
25-
* :class:`CellsRowLimitFilter <.row.CellsRowLimitFilter>`
26-
* :class:`CellsColumnLimitFilter <.row.CellsColumnLimitFilter>`
27-
* :class:`StripValueTransformerFilter <.row.StripValueTransformerFilter>`
28-
* :class:`ApplyLabelFilter <.row.ApplyLabelFilter>`
29-
30-
In addition, these filters can be combined into composite filters with
31-
32-
* :class:`RowFilterChain <.row.RowFilterChain>`
33-
* :class:`RowFilterUnion <.row.RowFilterUnion>`
34-
* :class:`ConditionalRowFilter <.row.ConditionalRowFilter>`
35-
36-
These rules can be nested arbitrarily, with a basic filter at the lowest
37-
level. For example:
38-
39-
.. code:: python
40-
41-
# Filter in a specified column (matching any column family).
42-
col1_filter = ColumnQualifierRegexFilter(b'columnbia')
43-
44-
# Create a filter to label results.
45-
label1 = u'label-red'
46-
label1_filter = ApplyLabelFilter(label1)
47-
48-
# Combine the filters to label all the cells in columnbia.
49-
chain1 = RowFilterChain(filters=[col1_filter, label1_filter])
50-
51-
# Create a similar filter to label cells blue.
52-
col2_filter = ColumnQualifierRegexFilter(b'columnseeya')
53-
label2 = u'label-blue'
54-
label2_filter = ApplyLabelFilter(label2)
55-
chain2 = RowFilterChain(filters=[col2_filter, label2_filter])
56-
57-
# Bring our two labeled columns together.
58-
row_filter = RowFilterUnion(filters=[chain1, chain2])
59-
60-
----
61-
624
.. automodule:: gcloud.bigtable.row
635
:members:
646
:undoc-members:
657
:show-inheritance:
66-
67-
.. _RowFilter definition: https://github.com/GoogleCloudPlatform/cloud-bigtable-client/blob/1ff247c2e3b7cd0a2dd49071b2d95beaf6563092/bigtable-protos/src/main/proto/google/bigtable/v1/bigtable_data.proto#L195

‎docs/index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,7 @@
6969
bigtable-table
7070
bigtable-column-family
7171
bigtable-row
72+
bigtable-row-filters
7273
bigtable-row-data
7374
happybase-connection
7475
happybase-pool

‎gcloud/bigquery/test_job.py‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
# pylint: disable=C0302
21
# Copyright 2015 Google Inc. All rights reserved.
32
#
43
# Licensed under the Apache License, Version 2.0 (the "License");

‎gcloud/bigtable/happybase/batch.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@
2121
import six
2222

2323
from gcloud._helpers import _datetime_from_microseconds
24-
from gcloud.bigtable.row import TimestampRange
24+
from gcloud.bigtable.row_filters import TimestampRange
2525

2626

2727
_WAL_SENTINEL = object()

‎gcloud/bigtable/happybase/table.py‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -30,14 +30,14 @@
3030
from gcloud.bigtable.happybase.batch import _get_column_pairs
3131
from gcloud.bigtable.happybase.batch import _WAL_SENTINEL
3232
from gcloud.bigtable.happybase.batch import Batch
33-
from gcloud.bigtable.row import CellsColumnLimitFilter
34-
from gcloud.bigtable.row import ColumnQualifierRegexFilter
35-
from gcloud.bigtable.row import FamilyNameRegexFilter
36-
from gcloud.bigtable.row import RowFilterChain
37-
from gcloud.bigtable.row import RowFilterUnion
38-
from gcloud.bigtable.row import RowKeyRegexFilter
39-
from gcloud.bigtable.row import TimestampRange
40-
from gcloud.bigtable.row import TimestampRangeFilter
33+
from gcloud.bigtable.row_filters import CellsColumnLimitFilter
34+
from gcloud.bigtable.row_filters import ColumnQualifierRegexFilter
35+
from gcloud.bigtable.row_filters import FamilyNameRegexFilter
36+
from gcloud.bigtable.row_filters import RowFilterChain
37+
from gcloud.bigtable.row_filters import RowFilterUnion
38+
from gcloud.bigtable.row_filters import RowKeyRegexFilter
39+
from gcloud.bigtable.row_filters import TimestampRange
40+
from gcloud.bigtable.row_filters import TimestampRangeFilter
4141
from gcloud.bigtable.table import Table as _LowLevelTable
4242

4343

‎gcloud/bigtable/happybase/test_batch.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ def test_constructor_defaults(self):
4646

4747
def test_constructor_explicit(self):
4848
from gcloud._helpers import _datetime_from_microseconds
49-
from gcloud.bigtable.row import TimestampRange
49+
from gcloud.bigtable.row_filters import TimestampRange
5050

5151
table = object()
5252
timestamp = 144185290431

‎gcloud/bigtable/happybase/test_table.py‎

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1113,7 +1113,7 @@ def test_invalid_type(self):
11131113

11141114
def test_success(self):
11151115
from gcloud._helpers import _datetime_from_microseconds
1116-
from gcloud.bigtable.row import TimestampRange
1116+
from gcloud.bigtable.row_filters import TimestampRange
11171117

11181118
timestamp = 1441928298571
11191119
ts_dt = _datetime_from_microseconds(1000 * timestamp)
@@ -1219,7 +1219,7 @@ def test_no_filters(self):
12191219
self._callFUT()
12201220

12211221
def test_single_filter(self):
1222-
from gcloud.bigtable.row import CellsColumnLimitFilter
1222+
from gcloud.bigtable.row_filters import CellsColumnLimitFilter
12231223

12241224
versions = 1337
12251225
result = self._callFUT(versions=versions)
@@ -1229,7 +1229,7 @@ def test_single_filter(self):
12291229
self.assertEqual(result.num_cells, versions)
12301230

12311231
def test_existing_filters(self):
1232-
from gcloud.bigtable.row import CellsColumnLimitFilter
1232+
from gcloud.bigtable.row_filters import CellsColumnLimitFilter
12331233

12341234
filters = []
12351235
versions = 1337
@@ -1244,9 +1244,9 @@ def test_existing_filters(self):
12441244

12451245
def _column_helper(self, num_filters, versions=None, timestamp=None,
12461246
column=None, col_fam=None, qual=None):
1247-
from gcloud.bigtable.row import ColumnQualifierRegexFilter
1248-
from gcloud.bigtable.row import FamilyNameRegexFilter
1249-
from gcloud.bigtable.row import RowFilterChain
1247+
from gcloud.bigtable.row_filters import ColumnQualifierRegexFilter
1248+
from gcloud.bigtable.row_filters import FamilyNameRegexFilter
1249+
from gcloud.bigtable.row_filters import RowFilterChain
12501250

12511251
if col_fam is None:
12521252
col_fam = 'cf1'
@@ -1282,7 +1282,7 @@ def test_column_unicode(self):
12821282
col_fam=u'cfU', qual=u'qualN')
12831283

12841284
def test_with_versions(self):
1285-
from gcloud.bigtable.row import CellsColumnLimitFilter
1285+
from gcloud.bigtable.row_filters import CellsColumnLimitFilter
12861286

12871287
versions = 11
12881288
result = self._column_helper(num_filters=3, versions=versions)
@@ -1295,8 +1295,8 @@ def test_with_versions(self):
12951295

12961296
def test_with_timestamp(self):
12971297
from gcloud._helpers import _datetime_from_microseconds
1298-
from gcloud.bigtable.row import TimestampRange
1299-
from gcloud.bigtable.row import TimestampRangeFilter
1298+
from gcloud.bigtable.row_filters import TimestampRange
1299+
from gcloud.bigtable.row_filters import TimestampRangeFilter
13001300

13011301
timestamp = 1441928298571
13021302
result = self._column_helper(num_filters=3, timestamp=timestamp)
@@ -1330,7 +1330,7 @@ def test_no_columns(self):
13301330
self._callFUT(columns)
13311331

13321332
def test_single_column(self):
1333-
from gcloud.bigtable.row import FamilyNameRegexFilter
1333+
from gcloud.bigtable.row_filters import FamilyNameRegexFilter
13341334

13351335
col_fam = 'cf1'
13361336
columns = [col_fam]
@@ -1339,10 +1339,10 @@ def test_single_column(self):
13391339
self.assertEqual(result, expected_result)
13401340

13411341
def test_column_and_column_families(self):
1342-
from gcloud.bigtable.row import ColumnQualifierRegexFilter
1343-
from gcloud.bigtable.row import FamilyNameRegexFilter
1344-
from gcloud.bigtable.row import RowFilterChain
1345-
from gcloud.bigtable.row import RowFilterUnion
1342+
from gcloud.bigtable.row_filters import ColumnQualifierRegexFilter
1343+
from gcloud.bigtable.row_filters import FamilyNameRegexFilter
1344+
from gcloud.bigtable.row_filters import RowFilterChain
1345+
from gcloud.bigtable.row_filters import RowFilterUnion
13461346

13471347
col_fam1 = 'cf1'
13481348
col_fam2 = 'cf2'
@@ -1378,7 +1378,7 @@ def test_no_rows(self):
13781378
self._callFUT(row_keys)
13791379

13801380
def test_single_row(self):
1381-
from gcloud.bigtable.row import RowKeyRegexFilter
1381+
from gcloud.bigtable.row_filters import RowKeyRegexFilter
13821382

13831383
row_key = b'row-key'
13841384
row_keys = [row_key]
@@ -1387,8 +1387,8 @@ def test_single_row(self):
13871387
self.assertEqual(result, expected_result)
13881388

13891389
def test_many_rows(self):
1390-
from gcloud.bigtable.row import RowFilterUnion
1391-
from gcloud.bigtable.row import RowKeyRegexFilter
1390+
from gcloud.bigtable.row_filters import RowFilterUnion
1391+
from gcloud.bigtable.row_filters import RowKeyRegexFilter
13921392

13931393
row_key1 = b'row-key1'
13941394
row_key2 = b'row-key2'

0 commit comments

Comments
 (0)