Skip to content

TypeError: an integer is required (got type bytes) #39

@louismartin

Description

@louismartin

Since upgrading to python 3.8 I can't access my old jobs' submission pickle (error below).

The problem might be related to this issue or this one but I have no clue what it means.

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-17-0248393e65dd> in <module>
     30         if state != "COMPLETED":
     31             continue
---> 32         row = job.submission().kwargs
     33         row["scores"] = job.result()
     34         row["exp_name"] = exp_name

~/dev/ext/submitit/submitit/core/core.py in submission(self)
    206             self.paths.submitted_pickle.exists()
    207         ), f"Cannot find job submission pickle: {self.paths.submitted_pickle}"
--> 208         return utils.DelayedSubmission.load(self.paths.submitted_pickle)
    209 
    210     def cancel_at_deletion(self, value: bool = True) -> "Job[R]":

~/dev/ext/submitit/submitit/core/utils.py in load(cls, filepath)
    133     @classmethod
    134     def load(cls: Type["DelayedSubmission"], filepath: Union[str, Path]) -> "DelayedSubmission":
--> 135         obj = pickle_load(filepath)
    136         # following assertion is relaxed compared to isinstance, to allow flexibility
    137         # (Eg: copying this class in a project to be able to have checkpointable jobs without adding submitit as dependency)

~/dev/ext/submitit/submitit/core/utils.py in pickle_load(filename)
    271     # this is used by cloudpickle as well
    272     with open(filename, "rb") as ifile:
--> 273         return pickle.load(ifile)
    274 
    275 

TypeError: an integer is required (got type bytes)

Repro:
Start a job with python 3.7 and then try to access it in python 3.8.
In python 3.7

Python 3.7.4 (default, Aug 13 2019, 20:35:49)
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import submitit
>>>
>>> def add(a, b):
...     return a + b
...
>>> executor = submitit.AutoExecutor(folder="log_test")
>>> executor.update_parameters(timeout_min=1, slurm_partition="dev")
>>> job = executor.submit(add, 5, 7)
>>> print(job.job_id)
33389760
>>> job.submission()
<submitit.core.utils.DelayedSubmission object at 0x7f42f5952bd0>

In python 3.8

Python 3.8.5 | packaged by conda-forge | (default, Jul 24 2020, 01:25:15)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import submitit
>>> job = submitit.SlurmJob("log_test", "33389760")
>>> job.submission()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/core.py", line 208, in submission
    return utils.DelayedSubmission.load(self.paths.submitted_pickle)
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/utils.py", line 135, in load
    obj = pickle_load(filepath)
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/utils.py", line 273, in pickle_load
    return pickle.load(ifile)
TypeError: an integer is required (got type bytes)
>>> job.submission()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/core.py", line 208, in submission
    return utils.DelayedSubmission.load(self.paths.submitted_pickle)
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/utils.py", line 135, in load
    obj = pickle_load(filepath)
  File "/private/home/louismartin/dev/ext/submitit/submitit/core/utils.py", line 273, in pickle_load
    return pickle.load(ifile)
TypeError: an integer is required (got type bytes)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions