Decoding step is terribly slow

In which file did you encounter the issue?

https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/bigtable/hello/main.py

Did you change the file? If so, how?

I pointed the code to my custom bigtable instance containing time series data. The data ingested was much larger than that used in the sample.

Describe the issue

The decoding loop in the scan takes a long time in python

# [START scanning_all_rows]
print('Scanning for all greetings:')
partial_rows = table.read_rows()
partial_rows.consume_all()

for row_key, row in partial_rows.rows.items():
    key = row_key.decode('utf-8')
    cell = row.cells[column_family_id][column_id][0]
    value = cell.value.decode('utf-8')
    print('\t{}: {}'.format(key, value))
# [END scanning_all_rows]

Up until partial_rows.consume_all() takes only 1 second. The following loop where the partial rows are decoded into a python data structure takes 2 seconds, tripling the total access time. I have tried condensing the data into into a single row to see if performance would improve, but it did not help much. Is there a faster way to get the decoded data out of the big table row structure and into another python data structure (such as a pandas dataframe or a numpy array of floats)?

@crwilcox

@crwilcox can you verify if this issue is obsolete, and if so, please close?

@seankerman

@seankerman what is the schema of a row and what are you doing to decode right now? I think perhaps I need a better reproduction.

For instance, I made a table with a string and a datetime with 10k rows. Reading those rows takes .02 seconds.

import argparse
import datetime

from google.cloud import bigtable
from google.cloud.bigtable import column_family
from google.cloud.bigtable import row_filters

def scanning(project_id, instance_id, table_id):
    client = bigtable.Client(project=project_id, admin=True)
    instance = client.instance(instance_id)
    table = instance.table(table_id)
    column_family_id = 'cf1'
    greeting_column_id = 'greeting'.encode()
    date_column_id = 'date'.encode()

    print('Scanning for all greetings:')
    partial_rows = table.read_rows()
    partial_rows.consume_all()

    start = datetime.datetime.utcnow()
    for row_key, row in partial_rows.rows.items():
        key = row_key.decode('utf-8')

        greeting_cell = row.cells[column_family_id][greeting_column_id][0]
        greeting_value = greeting_cell.value.decode('utf-8')

        date_cell = row.cells[column_family_id][date_column_id][0]
        date_value = datetime.datetime.fromisoformat(date_cell.value.decode('utf-8'))

        #print('\t{}: {}, {}'.format(key, greeting_value, date_value))
    end = datetime.datetime.utcnow()
    print(start, end)

def setup(project_id, instance_id, table_id):
    client = bigtable.Client(project=project_id, admin=True)
    instance = client.instance(instance_id)

    # Create Table
    print('Creating the {} table.'.format(table_id))
    table = instance.table(table_id)

    # Create Column Family
    print('Creating column family cf1 with Max Version GC rule...')
    # Create a column family with GC policy : most recent N versions
    # Define the GC policy to retain only the most recent 2 versions
    max_versions_rule = column_family.MaxVersionsGCRule(2)
    column_family_id = 'cf1'
    column_families = {column_family_id: max_versions_rule}
    if not table.exists():
        table.create(column_families=column_families)
    else:
        print("Table {} already exists.".format(table_id))

    # Populate Table
    print('Writing some greetings to the table.')
    greetings = [f"Hello World {i}!" for i in range(10000)]
    rows = []
    greeting_column = 'greeting'.encode()
    date_column = 'date'.encode()

    for i, value in enumerate(greetings):
        # Note: This example uses sequential numeric IDs for simplicity,
        # but this can result in poor performance in a production
        # application.  Since rows are stored in sorted order by key,
        # sequential keys can result in poor distribution of operations
        # across nodes.
        #
        # For more information about how to design a Bigtable schema for
        # the best performance, see the documentation:
        #
        #     https://cloud.google.com/bigtable/docs/schema-design
        row_key = 'greeting{}'.format(i).encode()
        row = table.row(row_key)
        row.set_cell(column_family_id,
                     greeting_column,
                     value,
                     timestamp=datetime.datetime.utcnow())
        row.set_cell(column_family_id,
                     date_column,
                     datetime.datetime.utcnow().isoformat(),
                     timestamp=datetime.datetime.utcnow())
        rows.append(row)
    table.mutate_rows(rows)


if __name__ == '__main__':
    project_id = 'project-test'
    instance_id = 'project-test-20191119'
    table_id = 'test_table_3'
    #setup(project_id, instance_id, table_id)
    scanning(project_id, instance_id, table_id)

As this is rather old I am going to close it for now. Please feel free to open this again, or file a new issue with a repro.

Navigation Menu

Repository navigation

Decoding step is terribly slow #1224

In which file did you encounter the issue?

Did you change the file? If so, how?

Describe the issue

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Decoding step is terribly slow #1224

Description

In which file did you encounter the issue?

Did you change the file? If so, how?

Describe the issue

Activity

leahecole commented on Nov 15, 2019

crwilcox commented on Nov 19, 2019

crwilcox commented on Nov 21, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions