works just fine. As of python3.7, this stringification is done by automatically under a from future import, and will eventually become the default, so the original code will be valid.
> This complexity helped drive the introduction of the auto keyword to C++. There are many situations where writing the type information isn’t workable. This is especially true when dealing with parametric container classes,
This is absolutely wrong! You cannot annotate a function as returning `auto` in C++. `auto` and its variants in other languages are useful for eliding redundant type declarations. Especially long ones that use generics/templates. `auto foo = MyContainer<Tuple<int, str>>();` or whatever is nice than having to double write the type declaration. But, in C++ or java, if you write a function that returns a Mycontainer<Tuple<int, str>>, you have to write that in the function.
This is an intentional choice: function declarations are your apis, and explicit and clear APIs are useful for human readers of your code. That's why you should annotate the public apis[0] even when you can have them be type inferred. Speaking of which, if you want type inference, check out pytype[1], it's like MyPy, but does do type inference on unannotated code. But you should still annotate your public apis. It serves as a sanity check that you aren't accidentally returning something you don't expect.
And of course, being familiar with the differences between iterables, iterators, sequences, containers, etc. is not a bad idea.
"[...]You cannot annotate a function as returning `auto` in C++. `auto` and its variants in other languages are useful for eliding redundant type declarations. Especially long ones that use generics/templates. `auto foo = MyContainer<Tuple<int, str>>();` or whatever is nice than having to double write the type declaration. But, in C++ or java, if you write a function that returns a Mycontainer<Tuple<int, str>>, you have to write that in the function.[...]"
Maybe I'm misunderstanding something, but one can write a function auto foo() in C++ and the return type is inferred. Lambda functions infer the return type by default.
You're correct about Java. It was an explicit decision that the only local variable's type declarations and lambda parameters may be inferred with `var`.
This excludes, as you mentioned, the return types of a function, its parameter's types, as well as the types of fields in a class.
There's a great 5m lightning talk from Pycon 2019 explaining the difference between pytype and mypy[0]. It's great that there's different libraries approaching the problem from different angles. Speaking of which, I'm also curious to see how Facebook's Pyre[1] differs from the former two. I'm surprise there's isn't a comprehensive comparison of the 3 libraries yet.
The good thing about inference (especially with a REPL) is you can write it without the annotation, and then use the inferred type (in Haskell, I usually find that when I resist the temptation to explicitly annotate types, the actual types are more general than I would have specified.)
I agree with that, too. My usual practice with Haskell is leave types off to leverage the information gained from type inference (with the intent of annotating signatures when I'm done), but then tell Haskell what I'm thinking the types should be if things break with impenetrable error messages.
But my coding in Haskell is pretty much personal and toy projects; I like the approach, but it may not be ideal for coding in anger.
This is generally true with pytype as well, but questionable whether it's a good thing.
Inferred, loose parameter types are great, because while I think it's good to know the differences between iterator, generator, iterable, sequence, container, and so on, it's difficult, and inference will give you the loosest one. But with return values, the inferred type will be the tighest, in other words your function will return a Dict (or worse, a defaultdict) instead of a Mapping or MutableMapping. You almost always want a Mapping.
As well as pytype, you can use a separate tool to discover the correct annotations, then verify them and add them to your source code. There are a couple - MonkeyType, pyannotate and pytypes.
I recently added type annotations to my Python 3.7 project and found it very helpful. Just the process of adding the annotations caught a handful of bugs!
Some things I found useful:
1. from __future__ import annotations
As I mentioned in another comment, this lets you write annotations naturally without worrying about when something gets defined.
2. typing.TYPE_CHECKING is only True while type checking.
This allows you to conditionally import files that would otherwise cause a circular dependency so that you can use the symbols for annotations.
3. def foo(bar: str = None) -> str
If an argument defaults to None, then mypy will recognize that its type is actually Optional[whatever]. So in the example above, bar is an Optional[str]. (I'm not sure if this is mypy-specific.) Optional[T] is equivalent to Union[T, None].
Annotations in comments or in type stubs are supported in python2 (you can look at typeshed for typing.Text and conditional py2/3 stuff). There's also some other cases, but they're...unique.
This is not mypy, it's "python-the-language": the annotations are values, and, in this case, the method annotations are evaluated when encountered - which is before the end of the class.
So you are trying to use a value still not defined.
It's like doing something like
def foo() -> Bar:
...
class Bar:
...
obviously the python interpreter cannot find a "Bar" definition during the parsing of the foo function
The case of referring to a class inside its definition is quite different from your "Bar" case. The thing referred to is already declared, it's just not fully defined yet.
There is no logical reason I can see making it impossible for Python to honor the obvious intention of the programmer here. It seems it has just chosen not to do so. Though I'll admit I haven't thought through every weird corner case :)
The second issue is a bit of a judgment call too, I think. If the types were inferred you could end up with errors that you're not able to see because inferred types end up compatible. Personally, I think I'd rather write them manually so that I'm validating whether the method actually matches what it was intended to return.
I can't find a more authoritative source right now, but I also remember seeing the same thing that this SO answer says - that it was a deliberate choice not to do this so that people could incrementally add type annotations: https://stackoverflow.com/a/38775381
I really like using and writing type annotated Python, but every time I do, I wonder if I should just use a language where all the annotation time I put in gets me actual runtime improvements.
I recommend checking F# if you want mostly inferred, strong static types and terse pythonesque Syntax (few braces, significant whitespace). Also much faster.
Considering you can use all of .NET from F# you can benefit from the work on C# as well.
While the first complaint is just a reflect of the lack of good tutorials on mypy, the second one is very valid: I too wish for automatic type detection for obvious cases.
What's more, while I found mypy useful myself in several instances, it's still cumbersome to use:
- you must know to use the magic command line arguments to avoid mypy complaining about other libs and imports all the time
- you need to use List[], Dict[], Set[], Iteratble[], Tuple[], etc. instead of list[], dict[], set[], iter[], tuple[], etc., which means an import in almost every file and a very unnatural workflow.
- you need to use Union and instead of |. This is ridiculous. I have to do:
from typing import Union, List
def mask(...) -> List[Union[bool, int]]
Instead of:
def mask(...) -> list[bool|int]
And Guido explicitly rejected the proposal for those on github despite the fact most imports are due to those.
Now, mypy has improved a lot. It's way faster, shows many more things that before, way less false positive, and has support for duck typing and dunder methods (named "protocol").
But it's sad to thing TypeScript is actually easier to use. Having a JS tech easier to use than a Python tech is a good sign we can improve things.
Why would you keep Python if you want static typing? I mean, there are several modern, performant (more than Python actually) statically typed languages. Rust, Go, Kotlin, Scala, Swift… All of them are mature and have a good library ecosystem. So, except if you need some math/AI/ML packages available only for Python, why bother? Choose the right tool for the job, no?
In general, for a new project of any size, or significance, or where performance/concurrency was a concern; hands down, yes, I would absolutely reach for a strong, statically typed, compiled language.
The fact remains though that Python is an excellent general purpose language, easy to learn, almost universally known, has libraries for everything (and that’s just the stdlib). It is well suited to rapid prototyping, scripting, and data wrangling. Adding incremental typing makes it even better.
No immutable vectors in Python? Are you not aware of tuples?
Lambdas in Python are quite limited, but offset by natural nested local functions.
Regarding performance, that's not always an issue, and depending on the use case can be addressed with async/await or using a multiprocessing pool. I have had issues with multiprocessing and some DB libs ib the past, but recently have had pretty good success.
Multithreading is still an issue because of the GIL (global interpreter lock) and really only useful if you're calling into a C/C++ lib that releases the lock while it does it's native things.
I've written a lot of Python in the last 15 years, and Ive written a ton of scripts that are faster than a well written identical C/C++/C# app, and was written in 1/10th of the time. It just depends on the use case.
Sorry, I meant persistent vectors. Data structures with type Vector[T] instead of Tuple[T0, T1, ...], amortized O(1) updates and list-like syntax for literals, e.g. i[x0, x1]. It's a minor annoyance, in most cases using vanilla Python lists with the convention 'any list that is part of a dataclass / is passed across function boundaries should never be mutated' is good enough. As u/joshuamorton noticed in this thread, the convention can be enforced with mypy and Sequence[T], which is great!
> * lack of immutable vectors, alleviated via style conventions
Note that mypy/typed python fixes this: if you annotate a function as returning (or accepting) a Sequence/Collection/Iterable instead of a List, and attempt to mutate it, you'll get type errors. This is good practice anyway.
Swift runs on Linux and has done since it was made open source in 2015, a year after its initial announcement. Windows support is being worked on but, for the time being, it runs fine in WSL. No lock in.
How's the cross-platform library ecosystem? My distant sentiment is that the bulk of libraries are tied to / provided by Apple, but I'd be happy to be proven wrong.
Cross-platform stuff is mainly non-GUI since the main thrust of the cross-platform scene seems to be server-side Swift, and most of the GUI stuff is targeted to iOS (although with Project Catalyst, that's all implicitly valid macOS code now).
I expect we may see some third-party, cross-platform reimplementations of SwiftUI very soon once the appropriate language features are set in stone and ship with the next version of Swift.
Otherwise, it's fairly healthy — lots of computational libraries, database, web, etc. Plus, Swift pretty much has direct compatibility with C and Python libraries without needing to wrap anything.
C compatibility is one of the original design goals for Swift and is used extensively for the Linux version, for making Swift wrappers around existing libraries, and the Objective-C bridge on macOS. If using Xcode, one would use a bridging header; otherwise, one can use Swift Package Manager to wrap C libraries for use in Swift projects.
Swift recently gained the features needed to call Python code. You can either use the Python module from Swift for Tensorflow or the PythonKit library, both of which allow calling Python code from within Swift.
The features used to allow calling Python from Swift can also be used for other dynamic languages, so there may be the ability to call Ruby and other languages' libraries in future as well.
Quite. My company is using Go for most new projects, but we have five years of Python code that we can't justify rewriting any time soon. We've been able to add annotations incrementally instead.
Getting a whole team to try out static typing on a language that they already know is far easier than getting a whole team to learn an entirely new language.
This is a really poor article with a click-bait headline. I was expecting this to be a case study where Python type annotations just couldn't work due to some interesting specifics of the projects. Instead, it's some petty gripes about the syntax of annotations not being just to the author's liking.
The first point can be rather easily circumvented by using strings as forward references and it only takes a bit of googling to figure that out. The second doesn't seem remotely like a deal breaker to me. Half the reason for having type annotations is to provide a description of what a function or method does at a glance. This is idiomatic even in hardcore functional languages like Haskell. With return type inference, you don't get that benefit.
I think the premise of this blog post is misguided.
Yes, there are two things about introducing types that the author finds annoying. So don't do those two things, and use type annotations for all the rest.
The type annotation system and mypy are designed to give you benefits even if you don't fully annotate everything -- that's the starting point of nearly all existing python code bases.
Other people have mentioned that the self-referential case from this post is doable. But what is not doable (to my knowledge) is this kind of self-reference:
class Node(typing.NamedTuple): # or dataclass
child: 'Node'
The error is example.py:4: error: Recursive types not fully supported yet, nested types replaced with "Any".
This error has been in there for several years at least. And it always bites me when I forget. It's particularly cryptic when the self-reference is many levels deep. I've even had the error message break in these cases where it prints out a line number from the wrong file. So, then you have to hunt to see where the self-reference is hiding.
Just tried this myself, and it seems to only be an issue for typing.NamedTuple. Though, it still manages to do type checking correctly...
from __future__ import annotations
from typing import Optional, NamedTuple
from dataclasses import dataclass
class Foo:
def __init__(self, foo: Foo = None) -> None:
self.child: Optional[Foo] = foo
Foo(Foo(Foo(None))) # Works
Foo(Foo(Foo(1))) # error: Argument 1 to "Foo" has incompatible type "int"; expected "Optional[Foo]"
@dataclass
class Bar:
child: Optional[Bar]
Bar(Bar(Bar(None))) # Works
Bar(Bar(Bar('bar'))) # error: Argument 1 to "Bar" has incompatible type "str"; expected "Optional[Bar]"
class Hep(NamedTuple): # error: Recursive types not fully supported yet, nested types replaced with "Any"
child: Optional[Hep]
Hep(Hep(Hep(None))) # Works
Hep(Hep(Hep({'a': 'b'}))) # error: Argument 1 to "Hep" has incompatible type "Dict[str, str]"; expected "Optional[Hep]"
It looks like the constructor is being typechecked, but probably not attribute access. `val.child.junk` will pass because `val.child` is `Any` and you can do any-thing to an Any.
a = Foo(Foo(Foo(None)))
b = Bar(Bar(Bar(None)))
c = Hep(Hep(Hep(None)))
a.child.junk = 1
# error: Item "Foo" of "Optional[Foo]" has no attribute "junk"
# error: Item "None" of "Optional[Foo]" has no attribute "junk"
b.child.junk = 'a'
# error: Item "Bar" of "Optional[Bar]" has no attribute "junk"
# error: Item "None" of "Optional[Bar]" has no attribute "junk"
c.child.junk = 1.1
# error: Item "Hep" of "Optional[Hep]" has no attribute "junk"
# error: Item "None" of "Optional[Hep]" has no attribute "junk"
a.child = 1
# error: Incompatible types in assignment (expression has type "int", variable has type "Optional[Foo]")
b.child = 1
# error: Incompatible types in assignment (expression has type "int", variable has type "Optional[Bar]")
c.child = 1
# error: Property "child" defined in "Hep" is read-only
# error: Incompatible types in assignment (expression has type "int", variable has type "Optional[Hep]")
Hey, thanks a lot for pointing this out! Yet another reason to use data classes vs. named tuples.
Looks like just plain class level attribute declaration also works:
class Node:
child: 'Node'
I wonder why `typing.NamedTuple` is unique as far as it not working. I know they don't use `eval` and templating to create it anymore. From looking at the code [0], they're using metaclass / __new__. But other than the fact that it uses metaclasses, I'm not sure why it'd have an error. Obviously there's cycle handling in mypy, otherwise none of the examples would work. If I use an example with a metaclass, that also typechecks. So, it's not metaclasses that trigger the error.
class NodeBase(type):
def __new__(cls, name, bases, attrs):
return super().__new__(cls, name, bases, attrs)
class Node(metaclass=NodeBase):
child: 'Node' # works
Edit: it looks like there is now work going into this and other cases where recursive types don't work [1][2]. For example: `Callback = typing.Callable[[str], Callback]` and `Foo = Union[str, List['Foo']]`.
Circular references like that are called "recursive types" in formal type theory. They are one of those big issues that academics like to write papers about.
Even if you just want to write a practical interpreter, and choose to gloss over the issues, they will still come back in some disguised form and either by requiring some sort of implementation kludge, or just by creating weird edge cases.
I feel baking type annotations on top of dynamic, late binding languages is fundamentally misunderstanding what they are about. Python might not go as far as say, smalltalk, but if you're picking a dynamic language just accept the runtime dynamism and don't program against it.
No, the workaround is to stringify "Node".
works just fine. As of python3.7, this stringification is done by automatically under a from future import, and will eventually become the default, so the original code will be valid.> This complexity helped drive the introduction of the auto keyword to C++. There are many situations where writing the type information isn’t workable. This is especially true when dealing with parametric container classes,
This is absolutely wrong! You cannot annotate a function as returning `auto` in C++. `auto` and its variants in other languages are useful for eliding redundant type declarations. Especially long ones that use generics/templates. `auto foo = MyContainer<Tuple<int, str>>();` or whatever is nice than having to double write the type declaration. But, in C++ or java, if you write a function that returns a Mycontainer<Tuple<int, str>>, you have to write that in the function.
This is an intentional choice: function declarations are your apis, and explicit and clear APIs are useful for human readers of your code. That's why you should annotate the public apis[0] even when you can have them be type inferred. Speaking of which, if you want type inference, check out pytype[1], it's like MyPy, but does do type inference on unannotated code. But you should still annotate your public apis. It serves as a sanity check that you aren't accidentally returning something you don't expect.
And of course, being familiar with the differences between iterables, iterators, sequences, containers, etc. is not a bad idea.
[0]: https://google.github.io/styleguide/pyguide.html#3191-genera...
[1]: https://github.com/google/pytype