Repository navigation
BigQuery Job objects not pickle-able #5866
Description
@tseaver suggested I start a new issue after adding to a closed issue: #3191
BigQuery Job Objects are not pickle-able:
In [16]: job = client.copy_table(source_table,source_table)
In [17]: job
Out[17]: <google.cloud.bigquery.job.CopyJob at 0x1163a3320>
In [18]: pickle.dumps(job)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-18-0c1255e2d696> in <module>()
----> 1 pickle.dumps(job)
AttributeError: Can't pickle local object 'if_exception_type.<locals>.if_exception_type_predicate'
I'm guessing that's because it represents some sort of future? IMO it would be fine if __getstate__ deleted the polling and let a reference that could be checked.
@tseaver suggests:
As a workaround, you could pickle the result of job._to_api_repr, and then reconstitute it using client.job_from_resource.
which works. This could also be done by __getstate__ itself.
Thanks
Activity
c1dc-candidate-22 commentedon Oct 3, 2018
@tseaver CopyJob doesnt seem to have to_api_repr. Can you suggest which Class type does the job you are referring to belongs to ?
tseaver commentedon Oct 3, 2018
@c1dc-candidate-22 Hmm, looking at the code, the method I had in mind isn't called
to_api_repr: it isCopyJob._build_resource. ISTM that renaming itto_api_reprwould be reasonable (of course leaving behind an alias for backward compatibility). @shollyman, @tswast WDYT?shollyman commentedon Oct 4, 2018
This is where I fully admit I have not fully internalized this veneer yet, so I'm happy to defer to others.
_build_resourceseems to revolve around ensuring a viable resource suitable for calling the backend for insertion (e.g. solely the configuration part of a job representation), whereas the goal here is the ability to serialize the whole representation of job: configuration, jobreference, statistics etc.Exposing the full representation seems fine, but its not clear that aliasing is the right thing to do here without further reading on my part. Perhaps @tswast can offer more insight?
tswast commentedon Oct 4, 2018
Job is the one set of resources that we didn't fully refactor during the 1.0 rewrite to follow the
to_api_repr/from_api_reprpattern. I agree that having these methods is desirable.It'll be a little bit tricky since Job mutates itself with the private
reload()function. It'd be good to remove some of the fancy logic that Job has for building it's internal state and use the_propertiesdictionary directly, as we do for other resources.wilberh commentedon Oct 5, 2018
Maybe the reason for the error is the data format used is not Python-specific?
Also, don't forget to "import pickle" and "import client".
tseaver commentedon Oct 5, 2018
@tswast If
_build_resourceas it stands today were renamedto_api_repr, that would satisfy the OP's usecase.tswast commentedon Oct 8, 2018
Yes, it looks like
_build_resourcehas no side effects. We can rename toto_api_repr.Expose 'to_api_repr' method for jobs. (#6176)