New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot save AutoEncoder #256
Comments
|
Sorry for being late on this. I recall some people mentioned that pickle may work. Haven't investigate. Should possibly do some experiment |
|
I am having the same issue, I found a solution for the AutoEncoder thanks to this answer: #88 (comment) (pickle or dill do not work for me) but I have the same problem with SOGAAL and MOGAAL and I don't know how to solve it. |
|
Yes, I think it's important for PyOD models to have a unified save/load API. Right now, it randomly breaks based on the underlying library each model uses. I temporarily got around this by creating a wrapper class with different save/load logic for sklearn vs TF models. |
|
@kennysong can u share ur wrapper? |
|
Unfortunately, the wrapper won't be that useful for you since it's for ensembles and is not a complete implementation. I'll share some tips that might be a starting point for you, though.
Here's a snippet as reference. class EnsembleDetector:
...
def save(self, folder):
'''Saves the EnsembleDetector (as multiple files) in a given folder.'''
# Save TF-based AutoEncoders in separate sub-directories (they don't pickle)
tf_models = {} # {index for self.models: model}
for i, model in enumerate(self.models):
if 'AutoEncoder' in str(type(model)):
model.model_.save(Path(folder)/str(i))
tf_models[i] = model.model_
model.model_ = None # Remove non-pickleable TF models from self so we can pickle self
if 'VAE' in str(type(model)):
raise Exception('VAE is not supported when saving the ensemble yet, since it uses a Lambda layer.')
# Pickle the entire EnsembleDetector after the TF models are removed
Path(folder).mkdir(parents=True, exist_ok=True)
joblib.dump(self, Path(folder)/'ensemble_detector.joblib')
# Add the TF model objects back into self
for i in tf_models: self.models[i].model_ = tf_models[i]
@staticmethod
def load(folder):
'''Loads the EnsembleDetector (from multiple files) in a given folder.'''
# Unpickle the EnsembleDetector object
ed = joblib.load(Path(folder)/'ensemble_detector.joblib')
# Load TF-based AutoEncoders from separate sub-directories (they don't pickle)
for i, model in enumerate(ed.models):
if 'AutoEncoder' in str(type(model)):
model.model_ = keras.models.load_model(Path(folder)/str(i))
return ed |
|
Having custom saving for keras model, would be useful |
The official instructions say to use joblib for pickling PyOD models.
This fails for AutoEncoders, or any other TensorFlow-backed model as far as I can tell. The error is:
Note that it's not sufficient to save the underlying Keras Sequential model, since I need the methods & variables of BaseDetector (like
.decision_scores_or.decision_function().The text was updated successfully, but these errors were encountered: