Repository navigation
blob.py is unable to handle a partial rewrite and fails with KeyError on lack of resource #3150
Description
871: self._set_properties(api_response['resource'])
This relies on the API to always return 'resource'. However, it is optional as per the doc.
https://cloud.google.com/storage/docs/json_api/v1/objects/rewrite
"""
This property is present in the response only when copying completes.
"""
As such, when the write was partial, done: false is returned and resource not present. Thus failing with the following KeyError.
Traceback (most recent call last):
File "xxx.py", line 304, in test_011_CopyFileFromBucketToBucketNewName
new_destination_file_name=new_file_name)
File "xxx.py, line 276, in copyFileFromBucketToBucket
token, rewritten, total = destination_blob.rewrite(source_blob, token=token)
File "/home/jenkins/venv/lib/python2.7/site-packages/google/cloud/storage/blob.py", line 681, in rewrite
self._set_properties(api_response['resource'])
KeyError: 'resource'
The most easy fix would be to push _set_properties down under after done: false check:
E.g.;
api_response = client._connection.api_request(
method='POST', path=source.path + '/rewriteTo' + self.path,
query_params=query_params, data=self._properties, headers=headers,
_target_object=self)
# self._set_properties(api_response['resource']) ## Comment out this line
rewritten = int(api_response['totalBytesRewritten'])
size = int(api_response['objectSize'])
if api_response['done']:
return None, rewritten, size
self._set_properties(api_response['resource']) # and move here
return api_response['rewriteToken'], rewritten, size
Activity
verbal commentedon Mar 16, 2017
I don't think your solution will work. I came across the same issue, but when it is not done, it falls through. You probably want to move it inside the if
api_response['done'].80nashi commentedon Mar 16, 2017
Ah, you are totally right. Scrap my suggestion.
I wasn't sure how _set_properties is used in the API and I mistook the "done" check in the wrong way, of course.
daspecster commentedon Mar 16, 2017
Thanks for reporting @80nashi!
If I understand correctly from the docs, the
resourceis only available when the copy is complete?@tseaver, @dhermes maybe we should be raising an exception here?
I'm not sure from the docs, but is there polling that should be available for rewriting so you can check if the copy is done?
80nashi commentedon Mar 17, 2017
As I understand the method and the API, the method is expected to return the rewrite token without raising an exception, in case of partial write (
done: falseandresourcenon-existent).Then the user program can call the method again with the non-null, non empty string rewrite token, repeat it until
done: trueandresourcepresent.80nashi commentedon Mar 17, 2017
So yeah, the following can be a possible fix as verba suggested.
However, I'm not sure what _set_properties is expected to do in case of
done: false.Fixes googleapis#3150 - Ignores resource key when it isn't present
daspecster commentedon Mar 21, 2017
@80nashi if you want to try and install from
masterit appears that @lukesneeringer addressed this.Fixes googleapis#3150 - Ignores resource key when it isn't present (g…
Fixes #3150 - Ignores resource key when it isn't present (#3177)