Skip to content

Missing last chunk when it's less than max_chunk_size in job.get_chunk_data_frame() #948

@leeway-liu

Description

@leeway-liu

Expected Behavior

Return all the chunks of data including the last one when its size is less than the max_chunk_size

Current Behavior

Doesn't return last chunk when its size is less than the max_chunk_size

Steps to reproduce

Run any data which has a last chunk when its size is less than the max_chunk_size

Specifications

  • Version: 0.5.0
  • Platform: Python 3
  • Subsystem:

Possible Solution

In get_chunk_data_frame() function, please update if not records to if records

        # Handle for last chunk that is < max_chunk_size
        if records:
            yield pd.DataFrame.from_records(records)

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions