[BigTable] Convert byte to int after table.read_row() call issue #2678

Activity

1. dmitry-saritasa commented on Nov 4, 2016

   

   dmitry-saritasa

   on Nov 4, 2016

   Author

   More actions

   v = connection.table('viewability')
   r = v.read_row('1#1478224240#100#7')
   
   In [8]: r.to_dict()
   Out[8]: 
   {'stat:advertiser': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366b50>],
    'stat:age': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366c10>],
    'stat:asset': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366c90>],
    'stat:browser': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366d10>],
    'stat:browser_ver': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366d90>],
    'stat:campaign': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366e10>],
    'stat:country': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366e90>],
    'stat:created': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366f10>],
    'stat:desktop': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff2366f90>],
    'stat:developer': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386050>],
    'stat:game': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13860d0>],
    'stat:ip_addr': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386150>],
    'stat:os': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13861d0>],
    'stat:os_ver': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386250>],
    'stat:session': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13862d0>],
    'stat:sex': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386350>],
    'stat:spot': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13863d0>],
    'stat:tick_size': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386450>],
    'stat:user': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff13864d0>],
    'stat:viewability': [<google.cloud.bigtable.row_data.Cell at 0x7f8ff1386550>]}
   
   In [9]: r.row_key
   Out[9]: '1#1478224240#100#7'

   first integer in row_key is value for field 'stat:asset', when I'm trying to obtain the value:

   In [7]: r.cells['stat']['asset'][0].value
   Out[7]: '\x00\x00\x00\x00\x00\x00\x00\x01'
   struct.unpack('>I', r.cells['stat']['asset'][0].value)
   ---------------------------------------------------------------------------
   error                                     Traceback (most recent call last)
   <ipython-input-12-1b4c35dc411e> in <module>()
   ----> 1 struct.unpack('>I', r.cells['stat']['asset'][0].value)
   
   error: unpack requires a string argument of length 4
   
2. dmitry-saritasa

   changed the title [-][Bi[/-] [+][BigTable] Convert byte to int after table.read_row() call issue[/+] on Nov 4, 2016
3. dmitry-saritasa commented on Nov 4, 2016

   

   dmitry-saritasa

   on Nov 4, 2016

   Author

   More actions

   I see, the answer is this
   https://docs.python.org/3/library/struct.html

   In [43]: struct.unpack('>Q', r.cells['stat']['asset'][0].value)
   Out[43]: (1,)
   

   however is it possible to conver the int/long values automatically to python type once read_row is called?
   set_cell doesn't require mandatory conversion to bytes so you simply submit int, char values and they're converted automatically.

   I think it's great for end developer to get actual 'python' value instead of bytes.

   Maybe I'm missing something and it has it's own purpose?

   thanks,
   Dmitry
4. dhermes commented on Nov 4, 2016

   

   dhermes

   on Nov 4, 2016

   Contributor

   More actions

   @dmitry-saritasa Bigtable stores no metadata about the types being stored, so storing typed data and converting back to types isn't something we could infer without using a different backend. So unfortunately, this is an impossible feature for us to add.

   If you'd like a way to specify a type and have it converted, we could look into serialization / deserialization helpers, but you can also roll your own. For example, you could convert integers directly to the bytes representing the digits, and this would be efficient storage-wise until you hit 99999999 (max int is 9223372036854775807 which has 19 digits, min is -9223372036854775808)

   What you are looking for is the Datastore API, which stores typed data.

   Please let me know if there are any other questions about this.
5. dhermes

   closed this as completedon Nov 4, 2016
6. dhermes

   added

   api: bigtableIssues related to the Bigtable API.Issues related to the Bigtable API.

   on Nov 4, 2016
7. JustinBeckwith

   assigned

   dhermes

   on Feb 1, 2021
8. parthea

   added a commit that references this issue on Oct 21, 2023

   automl: add natural language text classification ga samples [(#2678)](G…

   ...

   016e02b