Can we have a generic Type[C]?

The type object occupies a rather strange place in the type hierarchy:

>>> type(type) is type
True

(I'm pretty sure that's a flat lie, since you can't instantiate something from itself, but regardless...)

>>> isinstance(type, object)
True
>>> isinstance(object, type)
True
>>> isinstance(type, type)
True

In Java, the (very rough) equivalent is a class, specifically Class<T>. It's also generic; the type variable refers to the instance. Java has it easy because they don't support metaclasses. Classes are not first class in Java, so their type hierarchy doesn't have to deal with the strange loops shown above.

I realize metaclasses in their full generality are out of scope (at least for now), but a generic Type[T] like Java's would be nice to have. So far as I can tell from the Mypy documentation, it doesn't currently exist.

Here's some example code which might like to have this feature:

def make_foo(class_: Type[T]) -> T:
    # Instantiate and return a T

Unless this is trivial to add to mypy I propose to punt this to a future version of type hints.

FWIW type(X) is a shorthand for X.__class__ and type.__class__ is indeed type. Not a lie.

Yeah, but I'm pretty sure you have to reassign type.__class__ at some point to get there...

Anyway, I recognize that this is probably a lot more complex than it looks, and that the first 3.5 beta is rather alarmingly close, but I do feel this is essential for anyone using first-class classes in any significant way. Otherwise, you just have a type object, which is basically Callable[[???], object] (any idea what to put for the args?). You'd need a lot of casting to make that useful. I mean, I suppose you can treat it like Type[Any]/Callable[Any, Any], but then you're just being too permissive instead of too strict.

Any is your friend. :-)

Here's an example use case, straight from the code of my new framework:

@typechecked
def register_extension_type(ext_type: str, extension_class: type, replace: bool=False):
    """
    Adds a new extension type that can be used with a dictionary based configuration.

    :param ext_type: the extension type identifier
    :param extension_class: a class that implements IExtension
    :param replace: ``True`` to replace an existing type
    """

    assert_subclass('extension_class', extension_class, IExtension)
    if ext_type in extension_types and not replace:
        raise ValueError('Extension type "{}" already exists'.format(ext_type))

    extension_types[ext_type] = extension_class

I would like to declare the second argument as extension_class: Type[IExtension] (or Class[IExtension], doesn't matter to me). Likewise, the type hint for extension_types should be Dict[str, Type[IExtension]].

I understand the time pressure but without Type[T] typehinting system looks a bit incomplete. I hope it will be added soon.

Could you submit a patch to mypy? That would make a world of difference.

Until then, incompleteness of the typesystem doesn't bother me that much -- unlike statically-typed languages, Python programs don't have to be fully specified, you can always use Any.

I totally agree, one of the good points of gradual typing is the possibility to introduce it gradually :-) IMO it is perfectly pythonic - one could take exactly as much typing as one wants, practicality beats purity here. And yes I will try to make a patch.

@ilevkivskyi

@ilevkivskyi If you're serious about making a patch, make sure type inference of this line doesn't crash or break things too badly:

x = type

You could infer x to be a Type[Type[Type[...]]], but that's probably not a Good Idea. I'd just make it Type[object] or Type[Any]. Not sure whether this is an actual problem without looking at mypy's source, but it's something to watch out for.

Another issue related to Type[X] is whether a value with such type should be callable. Since __init__ is special (it is often overridden with an incompatible signature) just taking the signature of __init__ is not safe, unless there is a way to annotate __init__ to require a compatible override in all subclasses.

For example, consider x with type Type[object]. If it would be callable, x() should probably be accepted, as object() is fine. Now the runtime valueof x could be slice, as it's a subclass of object -- like any class is. However, in this case the call would fail at runtime since slice() is not valid.

@JukkaL

@JukkaL It gets worse:

class Foo:
    def __init__(self, args):
        # do something
    @classmethod
    def bar(cls, args):
        # cls is Type[Foo]
        self = cls(args)  # type: Foo
        # do something else
        return self

I agree with Jukka that this code is wrong. We can easily fix it by splitting the "do something" off into a separate (underscore-prefixed) method and calling super().__new__() from bar(). But unlike the examples we've been discussing so far, this is something the average user would actually run into. If we talk about Type[Foo] in the error message, we'll induce exploding head syndrome in the end user. It would be nice if we could avoid that.

The pattern (helper class methods that create and return instances) is indeed very popular. I think it's usually done for classes that won't be subclassed, or where the subclasses don't override __init__ (at least not in an incompatible way). In the former case one could use a static method and name the class explicitly, but the second use case is popular in some circles.

I wonder if a good type checker should be complaining only about an __init__ override that changes the signature, and only when the base class constructs instances of itself through a class method's cls argument.

Frankly I don't understand what the fuss is about. Type[Foo] would not guarantee anything beyond the type being a subclass of Foo. It would not guarantee compatibility of method signatures. Just like when annotating an argument as Foo, it would still accept an instance of any Foo's subclass, which may have arbitrary method overrides, yes?

@agronholm

@agronholm That's why we have the Liskov substitution principle. Usually, constructors and initializers are exempt from that rule since they're (effectively) static, but when you start to allow covariant first-class classes, that decision becomes a bit more questionable. Should static/class methods be exempt from LSP at all?

My point was that Python does not adhere to the LSP since you can arbitrarily override any methods, even if the overridden signatures are incompatible. In Java, this would simply cause the superclass method to be called in such cases. Are you proposing that the LSP should now be enforced somehow?

@gvanrossum

@gvanrossum In addition to my first explanation, I think CT does not represent metaclass, it still represents a type, you could even substitute a Union in place of it.

Hm, I agree CT does not represent a metaclass. The example Type[User] makes this pretty clear. It's Type itself that's like a metaclass (it inherits from 'type'). This is also pretty clear from the new_user() example:

def new_user(user_class: XXX):
    ...
new_user(User)

Compare this to

def foo(arg: int):
    ...
foo(42)

Here, 42 is an instance of int, just like in the previous example User is an instance of type. So where the annotation for arg is a class, the annotation for user_class is a metaclass.

Now back to variance. For this we need a different "regular" example. Let's use

def bar(args: Sequence[int]):
    ...

If we had a subclass of int, MyInt, then Sequence[MyInt] would be a valid type for a call to bar(), and that's what covariance means.

So in our Type example, we have Type[User] as the argument's annotation. For Type to be covariant would mean that if we have another function with an argument declared as Type[ProUser], we could call new_user() with that argument:

def new_pro_user(pro_user_class: Type[ProUser]):
    user = new_user(pro_user_class)
    ...

Indeed that sounds fine to me, so I agree intuitively (without having proven so rigorously) that Type is indeed covariant. (That's a big deal because previously I was just repeating "Type is covariant" because that's what the experts said; I hadn't actually visualized what that statement meant. :-)

Having gotten this far, I like Ivan's first sentence:

"Type is covariant in its parameter, because Type[Derived] is a subtype of Type[Base]."

I don't think the second sentence adds much (even though I agree it's true).

I agree that Type should be covariant. Here's my logic, which is basically an expanded version of your "if we have another function with an argument declared as Type[ProUser], we could call new_user() with that argument: [...] Indeed that sounds fine to me".

To test whether Type[Derived] should be a subtype of Type[Base], we need to check whether a value of type Type[Derived] is legal in any context where a value of type Type[Base] is. In order to do that we should start by writing down the typing rules for using values of type Type[T]. My suggestion is:

If c has type Type[T], C is a class, and T is a subtype of C, then if the expression C(...) type checks, then c(...) also type checks and has type T. [Construction via class variable]
If c has type Type[T], C is a class, and T is a subtype of C, then if the expression C.m(...) type checks, then c.m(...) also type checks and has the same type as C.m(...). [Invocation of class method via class variable]

(Why do these rules not start simply "If c has type Type[C], then if..."? The argument to Type might not be a class, but instead a Union or a type variable. In the case of a type variable, we need to be able to invoke class methods of the upper bound of the type variable (if it is a class). So I think we are forced into rules of this form.)

Now suppose S is a subtype of T and d has type Type[S]. Take the first rule and substitute d for c leaving the rest of the statement unchanged. Then S is a subtype of C, because S <: T <: C, and so we can apply the typing rule for d and S to conclude that d(...) has type S. Again since S <: T, d(...) also has type T. So, the value d also satisfies the typing rule for T, meaning it can be substituted for c. Similar logic applies to the second rule. So according to these rules, Type should indeed be covariant.

One problem with the typing rules above is that they don't actually correspond to python's runtime semantics. For example, every class is a subclass of object, but object's constructor can take 0 arguments, which a typical class's constructor cannot. So we expect a = A to have type Type[A], and then a() should be legal according to the above rule (with C = object), but it could fail at runtime. This is the LSP issue mentioned earlier in this thread. And this issue is unavoidable; we want the new_user example from the PEP to type check, but it really could fail at runtime if BasicUser's constructor isn't compatible with User's.

The language in the PEP about compatibility of method signatures is the assumption that is needed for the typing rule to match the runtime reality. mypy currently checks compatibility for ordinary methods, but not for the constructor. The true covariance rule for runtime behavior is something like "Type[D] is a subtype of Type[C] when D is a subclass of C which overrides the constructor and methods of C in a compatible manner".

Operationally I think mypy should implement these rules by just picking C to be the most specific class that is known to be a supertype of the type T, since that is the class that c is most likely to be compatible with (more so than any of C's superclasses, like object).

@gvanrossum

Also, I quite like @gvanrossum's type_map example. A related example would be code that does something like

class A: ...
class AImpl1(A): ...
class AImpl2(A): ...

if some_condition:
    aImpl = AImpl1  # type: Type[A]
else:
    aImpl = AImpl2

# use class methods on aImpl

Technically this does not require variance since the rule for typing classes could be "C has type Type[T] if T is a supertype of C", but it's nicer to just say "C has type Type[C]" and make use of covariance.

OK, thanks for the (semi-?)formal proof, it helps to know that I didn't miss a case (reasoning about variance just doesn't come naturally to me, I'm probably a closet Eiffel programmer :-).

Of course the weakness of the whole scheme is that the constructor signature of a subclass doesn't have to match that of the base class -- but that's not specific to the argument for covariance. There's already text in the PEP that promises to get back to this issue in the future.

The other thought that this triggered for me is that a class method might itself be a factory that uses Type[T] in its return type, for some type variable T whose upper bound is C. Then c.m() needn't have the same type as C.m() -- wherever the return type of C.m() uses C, the return type of c.m() uses c. But honestly I don't want to weigh the PEP down with this formalism anyways, so we can hand-wave this away.

It should probably reject isinstance(..., Type[...]) and issubclass(.., Type[...]).

I'm not sure which "it" you're referring to here: mypy or typing.py's runtime implementation. If these are rejected at runtime, then I would tend to prefer that they be rejected by mypy too.

Trying this out, mypy already gives an error

x/type.py:7: error: Generic type is prohibited as a runtime expression (use a type alias or '# type:' comment)

but with a type alias mypy accepts the isinstance call since Type[T] is a subtype of type. Seems like it would take a special case for mypy to reject this form, so maybe it should be allowed at runtime too?

Running under python gave a baffling error, looks like an unrelated issue?:

Traceback (most recent call last):
  File "x/type.py", line 7, in <module>
    isinstance(1, Type_int)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/typing.py", line 996, in __instancecheck__
    return self.__subclasscheck__(instance.__class__)
TypeError: __subclasscheck__() takes exactly one argument (0 given)

For context, that __instancecheck__ is a method of GenericMeta and the full program I am running is

from typing import TypeVar, Generic

T = TypeVar('T')
class Type(type, Generic[T], extra=type): pass

Type_int = Type[int]
isinstance(1, Type_int)

The other thought that this triggered for me is that a class method might itself be a factory that uses Type[T] in its return type, for some type variable T whose upper bound is C. Then c.m() needn't have the same type as C.m() -- wherever the return type of C.m() uses C, the return type of c.m() uses c. But honestly I don't want to weigh the PEP down with this formalism anyways, so we can hand-wave this away.

Something like this crossed my mind too; I think it might fit better with the SelfType stuff discussed elsewhere and we should not worry about it right now.

(Our remarks about constructor signatures and LSP crossed, but I think we're in agreement. Note that object is even more special than most other classes, because it also has a wacko rule about compatibility between __new__ and __init__ if you implement one of them but not the other. But I digress.)

I'm not sure which "it" you're referring to here

Me neither, but most likely I was thinking about the type checker. Or perhaps the PEP. Since at runtime isinstance() involving special stuff is almost always forbidden, and issubclass() is forbidden by the PEP (though the runtime still allows it -- ripping it out is still a task (#136) but it's stalled by some unforeseen problems).

@gvanrossum

@gvanrossum I agree that my second sentence does not add much, but I do not see a shorter way of explaining the covariance than @rwbarton did. I think probably it would be better to simply add a short example, illustrating that a value of type Type[Derived] is legal in a context where a value of type Type[Base] is. Something like this:

``Type`` is covariant in its parameter, because ``Type[Derived]`` is a subtype of ``Type[Base]``::
  def new_pro_user(pro_user_class: Type[ProUser]):
      user = new_user(pro_user_class) # OK
    ...

Improve wording on covariance of Type[] (Ivan L, #107).

Done! I've also added a similar call to new_user() to the earlier Union example (implicitly acknowledging covariance there without mentioning it, to avoid scaring people unnecessarily).

Closing -- both the PEP and typing.py have been updated. We're waiting for mypy but that's also very close (and not essential to this issue): python/mypy#1569

Navigation Menu

Repository navigation

Can we have a generic Type[C]? #107

50 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Can we have a generic Type[C]? #107

Description

Activity

gvanrossum commented on May 8, 2015

NYKevin commented on May 9, 2015

gvanrossum commented on May 9, 2015

agronholm commented on May 18, 2015

ilevkivskyi commented on May 18, 2015

gvanrossum commented on May 18, 2015

ilevkivskyi commented on May 18, 2015

NYKevin commented on May 19, 2015

JukkaL commented on May 22, 2015

NYKevin commented on May 22, 2015

gvanrossum commented on May 22, 2015

agronholm commented on May 22, 2015

NYKevin commented on May 22, 2015

agronholm commented on May 22, 2015

50 remaining items

ilevkivskyi commented on May 19, 2016

gvanrossum commented on May 19, 2016

rwbarton commented on May 19, 2016

rwbarton commented on May 20, 2016

rwbarton commented on May 20, 2016

gvanrossum commented on May 20, 2016

rwbarton commented on May 20, 2016

rwbarton commented on May 20, 2016

gvanrossum commented on May 20, 2016

gvanrossum commented on May 20, 2016

ilevkivskyi commented on May 20, 2016

gvanrossum commented on May 20, 2016

gvanrossum commented on Jun 7, 2016

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions