Skip to content

Add possibility to save stores itself...#385

Merged
dwhswenson merged 15 commits into
openpathsampling:masterfrom
jhprinz:addsavestore
Nov 25, 2015
Merged

Add possibility to save stores itself...#385
dwhswenson merged 15 commits into
openpathsampling:masterfrom
jhprinz:addsavestore

Conversation

@jhprinz

@jhprinz jhprinz commented Nov 18, 2015

Copy link
Copy Markdown
Contributor

This was part of the original PR #380 and is only the part that affects netcdfplus by adding the possibility to save the list of used stores within the storage itself. This provides better backward compatibility and will allow to always open a store as a file.

This is in total a much better way to define a file format.

Of course the used stores require implementations which need to be available and these code is not stored!

@jhprinz jhprinz changed the title Add possibility to save stores itself... [WIP] Add possibility to save stores itself... Nov 18, 2015
@jhprinz

jhprinz commented Nov 24, 2015

Copy link
Copy Markdown
Contributor Author

Ready for review

@jhprinz jhprinz changed the title [WIP] Add possibility to save stores itself... Add possibility to save stores itself... Nov 24, 2015
@dwhswenson

Copy link
Copy Markdown
Member

Also depends on addfeatureagain; same question as in #384.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When do you have type(obj) is abc.ABCMeta while trying to store the object? My intuition would have been that it was impossible for that to occur, since abc.ABCMeta shouldn't get instantiated.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is something that is caused by the use of a metaclass to implements abc. In the classes that use abstract I placed a __metaclass__ = abc.ABCMeta which replaces the function type by the function abc.ABCMeta as the class constructor (not to be confused with the class instance constructor which is the __init__ method). When building any class that is a potential subclass of this abstract base class the metaclass is also inherited. That means that all subclasses lead to type(class) == abc.ABCMeta and not type(class) == type. It took me some time to figure that out.

This part was not needed before since usually we do not store a class, but rather instances and their attributes. The reference to a class was only necessary to provide a store with the possibility to store for which class this store is meant. This is necessary for all object_stores.

If you have a better idea to check if an obj is a class or an abstract class that would be great. The more pythonic way is probably to use inspect.isclass() but that does more or less the same thing and check for more cases and is potentially slower. But I can change that to remore the necessity to import abc.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understood what the code was doing, but I still don't quite understand why. (And I think that, for what it is doing, that's a fine way of doing it). But I don't get what you're saying here:

This part was not needed before since usually we do not store a class, but rather instances and their attributes. The reference to a class was only necessary to provide a store with the possibility to store for which class this store is meant. This is necessary for all object_stores.

Can you explain what is going on that we are storing classes, as opposed to the attributes of an instance of a given class?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. The underlying idea was that a store (the class instances that stores objects) should be storable objects themselves. In doing so the stored file contains most of the structure in it, not only the data. I think that was clear. Now to save a store I needed to add a new variable type for the to_json functions. The thing that I named simplifier. It takes a dict (presumable the one from to_dict) and converts all objects that cannot directly turned into json into a little dict that will be reconstructed when loading. So

object >(to_dict())> dict >(simplifier))> simple_dict >(json.decode())> string

The simplifier never needed to encode a class itself. And my implementation checked for type(obj) == type and if the class was subclassed of storableobjects (we do not allow storing unknown classes). In that case we take the name of the class and store it with a _class key. When reconstructing it realizes the _class key and picks the appropriate class from the list of registered storable objects.

Now to your question. The point is that before it just never occurred that an instance had an attribute that pointed to a class. Now for storing stores themselves the store has an attribute that points to a class. Namely the base class to be stored, which I call content_class.

Hence the need to update the simplifier to handle now abstract classes as well. This would have failed before too, but since we never stored abstract classes the error never showed up.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That explains it. Thanks for the clarification!

@dwhswenson

Copy link
Copy Markdown
Member

Would it be reasonable to add from storage.storage import Storage to the OPS __init__.py? I find the extra lines of import openpathsampling.storage as st to be a little annoying, since the user will only ever directly use one class from that subpackage.

@dwhswenson

Copy link
Copy Markdown
Member

Aside from comments above, this all looks reasonable. I'll plan to merge this one first, since it has a number of code overlaps with #384; I'll review that after this is merged. Let's finish the discussion on this one as soon as possible.

@jhprinz

jhprinz commented Nov 25, 2015

Copy link
Copy Markdown
Contributor Author

Yes, this is independent of all other PRs and should be merged first. The from storage.storage import Storage is good as long as it does not create import cycles (which I will check). I agree that this is annoying.

@jhprinz

jhprinz commented Nov 25, 2015

Copy link
Copy Markdown
Contributor Author

Works. You can now use

paths.Storage()
paths.AnalysisStorage()

@jhprinz

jhprinz commented Nov 25, 2015

Copy link
Copy Markdown
Contributor Author

This passed tests so should be ready to merge.

@dwhswenson

Copy link
Copy Markdown
Member

Works. You can now use

paths.Storage()
paths.AnalysisStorage()

Excellent! That definitely feels more natural.

Merging this, but please respond to the line notes above. They're mainly me asking for clarification; the code looks fine and runs fine.

dwhswenson added a commit that referenced this pull request Nov 25, 2015
Add possibility to save stores itself...
@dwhswenson dwhswenson merged commit f6a9415 into openpathsampling:master Nov 25, 2015
@jhprinz jhprinz deleted the addsavestore branch July 12, 2016 23:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants