BigQuery: Allow choice of compression for loading from dataframe #7701

Activity

1. tswast

   added

   api: bigqueryIssues related to the BigQuery API.Issues related to the BigQuery API.

   type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.

   good first issueThis issue is a good place to started contributing to this repository.This issue is a good place to started contributing to this repository.

   on Apr 12, 2019
2. tswast commented on Apr 12, 2019

   

   tswast

   on Apr 12, 2019

   Contributor

   More actions

   Thanks for the report @awgymer . This is a great suggestion. I agree that the snappy system requirement can be painful to install. I think we can mirror the compression argument from pandas in the load_table_from_dataframe method.

   Happy to accept a PR for this.
3. awgymer commented on Apr 20, 2019

   

   awgymer

   on Apr 20, 2019

   Author

   More actions

   So on a little more looking, the pandas.Dataframe.to_parquet() allows three (four if you count uncompressed) options: snappy, gzip, and brotli, whilst the official docs say that

   BigQuery supports Snappy, GZip, and LZO_1X codecs for compressed data blocks in Parquet files

   So I don't know now if it is in fact more confusing to move to allowing 2/3 of the options (again uncompressed too?) rather than remaining fixed to 1 and simply making the requirement for snappy more clear in the docs?
4. tswast commented on Apr 24, 2019

   

   tswast

   on Apr 24, 2019

   Last edited by tswast

   Contributor

   More actions

   What dependencies are needed for gzip? Is it a noticeable difference in performance? I'm open to changing the default if it's easier to install and not a noticeable difference.

   Also I'm actually a bit wary of exposing compression, since I'd like to keep the option open to change the file format we serialize to. Parquet happens to be the best match to BigQuery's data types right now, but it'd be good to keep the option open to move to something else in the future.
5. awgymer commented on Apr 24, 2019

   

   awgymer

   on Apr 24, 2019

   Author

   More actions

   As far as I can tell it uses the standard python gzip/zlib which use the zlib library. I couldn't speak for performance difference at the moment, but I appreciate your point about not exposing the parameter, so perhaps just making it clearer that snappy is a dependency for that function in the docs would be a suitable solution?

   👍React with 👍1Reacted by Tim Sweña (Swast)
6. tswast commented on Apr 24, 2019

   

   tswast

   on Apr 24, 2019

   Contributor

   More actions

   making it clearer that snappy is a dependency for that function in the docs would be a suitable solution?

   Yeah, that seems reasonable. We already talk about needing pyarrow. We should mention snappy, too.

   Related: The required system packages for snappy are one of the reasons I think conda is still relevant, despite wheels and why I publish the google-cloud-bigquery package to conda-forge.
7. plamut

   self-assigned this

   on Aug 5, 2019
8. plamut

   closed this as completedin #8938on Aug 6, 2019