Add possibility to save stores itself...#385
Conversation
|
Ready for review |
|
Also depends on |
There was a problem hiding this comment.
When do you have type(obj) is abc.ABCMeta while trying to store the object? My intuition would have been that it was impossible for that to occur, since abc.ABCMeta shouldn't get instantiated.
There was a problem hiding this comment.
This is something that is caused by the use of a metaclass to implements abc. In the classes that use abstract I placed a __metaclass__ = abc.ABCMeta which replaces the function type by the function abc.ABCMeta as the class constructor (not to be confused with the class instance constructor which is the __init__ method). When building any class that is a potential subclass of this abstract base class the metaclass is also inherited. That means that all subclasses lead to type(class) == abc.ABCMeta and not type(class) == type. It took me some time to figure that out.
This part was not needed before since usually we do not store a class, but rather instances and their attributes. The reference to a class was only necessary to provide a store with the possibility to store for which class this store is meant. This is necessary for all object_stores.
If you have a better idea to check if an obj is a class or an abstract class that would be great. The more pythonic way is probably to use inspect.isclass() but that does more or less the same thing and check for more cases and is potentially slower. But I can change that to remore the necessity to import abc.
There was a problem hiding this comment.
I understood what the code was doing, but I still don't quite understand why. (And I think that, for what it is doing, that's a fine way of doing it). But I don't get what you're saying here:
This part was not needed before since usually we do not store a class, but rather instances and their attributes. The reference to a class was only necessary to provide a store with the possibility to store for which class this store is meant. This is necessary for all object_stores.
Can you explain what is going on that we are storing classes, as opposed to the attributes of an instance of a given class?
There was a problem hiding this comment.
I see. The underlying idea was that a store (the class instances that stores objects) should be storable objects themselves. In doing so the stored file contains most of the structure in it, not only the data. I think that was clear. Now to save a store I needed to add a new variable type for the to_json functions. The thing that I named simplifier. It takes a dict (presumable the one from to_dict) and converts all objects that cannot directly turned into json into a little dict that will be reconstructed when loading. So
object >(to_dict())> dict >(simplifier))> simple_dict >(json.decode())> string
The simplifier never needed to encode a class itself. And my implementation checked for type(obj) == type and if the class was subclassed of storableobjects (we do not allow storing unknown classes). In that case we take the name of the class and store it with a _class key. When reconstructing it realizes the _class key and picks the appropriate class from the list of registered storable objects.
Now to your question. The point is that before it just never occurred that an instance had an attribute that pointed to a class. Now for storing stores themselves the store has an attribute that points to a class. Namely the base class to be stored, which I call content_class.
Hence the need to update the simplifier to handle now abstract classes as well. This would have failed before too, but since we never stored abstract classes the error never showed up.
There was a problem hiding this comment.
That explains it. Thanks for the clarification!
|
Would it be reasonable to add |
|
Aside from comments above, this all looks reasonable. I'll plan to merge this one first, since it has a number of code overlaps with #384; I'll review that after this is merged. Let's finish the discussion on this one as soon as possible. |
|
Yes, this is independent of all other PRs and should be merged first. The |
|
Works. You can now use |
|
This passed tests so should be ready to merge. |
Excellent! That definitely feels more natural. Merging this, but please respond to the line notes above. They're mainly me asking for clarification; the code looks fine and runs fine. |
Add possibility to save stores itself...
This was part of the original PR #380 and is only the part that affects netcdfplus by adding the possibility to save the list of used stores within the storage itself. This provides better backward compatibility and will allow to always open a store as a file.
This is in total a much better way to define a file format.
Of course the used stores require implementations which need to be available and these code is not stored!