Skip to content

Bigtable and float values #2614

Closed
Closed

Description

@dmitry-saritasa

tried to store 0.73 as numeric value in bigtable cell and got this error:

TypeError: 0.67 could not be converted to bytes

Trace

/home/dmitry/Projects/demo/storm/libs/db/models.py in save(self, record, instance_id)
    205             for col in columns:
    206                 row.set_cell(
--> 207                     column_family, col, getattr(record, col, None)
    208                 )
    209         row.commit()
/home/dmitry/Projects/experiments/storm/petrel/chapter3/example1/petrel/lib/python2.7/site-packages/gcloud/bigtable/row.pyc in set_cell(self, column_family_id, column, value, timestamp)
    300         """
    301         self._set_cell(column_family_id, column, value, timestamp=timestamp,
--> 302                        state=None)
    303 
    304     def delete(self):

/home/dmitry/Projects/experiments/storm/petrel/chapter3/example1/petrel/lib/python2.7/site-packages/gcloud/bigtable/row.pyc in _set_cell(self, column_family_id, column, value, timestamp, state)
    126         if isinstance(value, six.integer_types):
    127             value = _PACK_I64(value)
--> 128         value = _to_bytes(value)
    129         if timestamp is None:
    130             # Use -1 for current Bigtable server time.

/home/dmitry/Projects/experiments/storm/petrel/chapter3/example1/petrel/lib/python2.7/site-packages/gcloud/_helpers.pyc in _to_bytes(value, encoding)
    457         return result
    458     else:
--> 459         raise TypeError('%r could not be converted to bytes' % (value,))
    460 
    461 

Checked the source code and see that only int or bytes are allowed to be stored. Therefore I have to convert float value into INT *i.e. 0.63 *100 = 63 and store that as a value

Am I correct and why only ints/bytes are allowed?

thanks,
Dmitry

Activity

  1. dhermes commented on Oct 25, 2016

    @dhermes
    Contributor

    @dmitry-saritasa Thanks for filing.

    1. Can you provide the code you used to trigger this error?
    2. The backend allows only bytes but has special methods for counters, which are stored as >q (i.e. a long long / 64-bit signed integer in big-endian byte order).
    3. If you'd like to store a Python float (double), you'd need to convert to bytes e.g. struct.pack('>d', 0.75)

    @jonparrott WDYT of supporting types other than bytes or integers? It'd be easy enough to do, but may go against the spirit of the API.

  2. dmitry-saritasa commented on Oct 25, 2016

    @dmitry-saritasa
    Author

    clear - thanks @dhermes

  3. theacodes commented on Oct 26, 2016

    @theacodes
    Contributor

    @dhermes I think it affords discussion - Python's representation of other types may not work if the data is read in another language.

  4. added
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on Apr 19, 2017
  5. lukesneeringer commented on Aug 11, 2017

    @lukesneeringer
    Contributor

    I am going to declare bankruptcy on this issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

api: bigtableIssues related to the Bigtable API.priority: p2Moderately-important priority. Fix may not be included in next release.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions