I admit to not fully understanding the async PEP (partially since I don't do much Python programming anymore). Are there any downsides to having to explicitly mark an async function using the "async def" syntax?

I'm thinking of how in Go I can launch a goroutine with "go foo()", where foo can be any function and not one that happened to be declared explicitly as async.

Or in Perl 6, I can "my $p100 = start { (1..Inf).grep(*.is-prime)[99] }; say await $p100;" This provides the similar blocking "await" keyword, but I'm still free to execute any arbitrary code inside the block denoted by "start".

I have a feeling that these questions are both answered by whatever coroutine functionality existed prior to the introduction of "async def", but that's an area of Python that I have no experience with. I'd appreciate any clarification that could be offered. Thanks!

Edit: re: the P6 example, it looks like it works somewhat like Python, where a "Future-like object" (from the PEP) is provided to await; in P6 start returns a Promise. However it looks like in Python I still need to wrap the enclosing function with "await def". https://www.python.org/dev/peps/pep-0492/#await-expression

Goroutines are, for all practical purposes, threads. They're cheaper than real threads but they still have all the subtle complexity that comes with shared-memory preemptive threading (everything shared needs to be appropriately locked, scheduling is non-deterministic and can happen any time, etc). Async/await is different; context switches can only happen at an `await` expression. This greatly reduces the number of possibilities that must be considered and simplifies the task of concurrent programming (for more, see https://glyph.twistedmatrix.com/2014/02/unyielding.html).

In Python, the closest equivalent to `go foo()` is `threading.Thread(target=foo).start()`. Or you can use gevent to get goroutine-like lightweight threads, but I find this is works better in go than in python because the entire language is designed around it. In python it's common to find libraries that are not compatible with gevent's monkey-patching and so you lose concurrency in a way that is difficult to detect or guard against.

Going to have to disagree with the criticism of gevent. The monkey-patching is very robust, despite the fact that monkey-patching is a sin in general. I've never had any problems with it, with the exception of third party native modules.

If code is using the regular Python socket and threading libraries, it will almost always work seamlessly. Gevent provides the closest thing to goroutines.

Third party native modules are exactly the issue. Many database drivers (etc) are wrappers around native libraries that do not integrate with gevent; the ease of integration with native code is generally one of Python's great strengths. To use gevent you must pay a lot more attention to implementation details of libraries you rely on.

> Are there any downsides to having to explicitly mark an async function using the "async def" syntax?

Yes. One main one is splintering the library ecosystem. So some libraries are using Twisted, some are using base threading system, some run in Tornado, some will use async and so on. They are usually not mixable. One day in the distant future they will all support async, but that means updating all of them.

Special markers ("async" waits points/objects) for IO co-routines tend to propagate vertically through all the API layers. If, say, a low level library you found is returning a Deferred/Promise/Future or generates values, then top level has to handle that as well. Then its parent also has to handle it.

It is infecting the ecosystem with low levels details about how the IO needs special casing because of how the GIL works. Yes, it is more elegant in Go, Rust, Erlang, C#, Java etc.

BTW Python has something like it based on the greenlet library (eventlet and gevent libraries are based on). Those monkeypatch system libraries so it manages to hide this complexity, but there are other costs, unsupported or untested side-effects being one.

What does the GIL have to do with this? All blocking I/O calls release the GIL.

It's more complicated than that. Specifically having a mix of any CPU and I/O bound threads doesn't play out as nicely:

See this:

http://dabeaz.blogspot.com/2010/02/revisiting-thread-priorit...

Or even better for a demo watch David Beazley's Pycon 2015 video (It is an awesome video even if you don't care about the GIL or socket programming).

Looks like this is the video referred to: http://pyvideo.org/video/3432/python-concurrency-from-the-gr...

Yap, thank you for finding. I was just being lazy.

Think of async/await as syntactic sugar for doing the kind of asynchronous programming when you pass callbacks/continuations and have an explicit event loop. For example, assume that you have an http_request function that looks like this:

    http_request(url, callback)

If you want to make a request, get a value, and then make another request depending on that value, you would need something like:

    def callback(result):
        url2 = get_from_value(result)

        def inner_callback(result):
            print('Final result:', result)

        http_request(url2, print)

    http_request(url1, callback)

This is very complicated, specially because in Python it's not easy to inline callbacks like in JavaScript. And even then, it's sometimes messy and becomes a callback hell.

With async/await you could do something like:

    result1 = await http_request(url1)
    result2 = await http_request(get_from_value(result1))
    print('Final result:', result2)

The await keyword doesn't block like in Perl 6. It suspends the execution and returns the control to the even loop. Then the event loop calls other functions in response to other events. All this happens in a single process and a single thread.

This model is more explicit than the Goroutines of Go or the multiple threads of Perl 6. But it has the advantage that the user has more control over when the execution control flows between different parts of the program. For example, if I have this code:

    await request1()
    # some synchronous code
    await request2()

I am completely sure that nothing will run while the code in the middle of the two request is running. This simplifies a lot of synchronization problems.

AFAIU await can only be used with async functions:

  async def http_request(url):
    # ...

  await http_request(u)

If the method I want to use is not an async one, is it possible to dynamically define a lambda ?

Python could do green threads (IO-base coroutines) for a long time -- see eventlet, gevent, even Twisted's inlineCallback pattern. There are probably others, there was even one based on "yields" as well.

Stackless python pairs well with green threads. Very, very well. I believe pypy based a lot of their initial work on the design of stackless python.

Yap, I've used and shipped code based on eventlet. And you are right PyPy supports that as well. I have never tested it using PyPy. Perhaps one day. For large concurrency cases I would probably look at Erlang/Elixir though...

C# has the reactive extensions API https://rx.codeplex.com/ which is the inspiration for the Perl 6 concurrency model which is also quite advanced https://www.youtube.com/watch?v=JBHsdc0IVIg So not sure why you think Python is somehow unique in doing all this.

This is awesome!

I want to ask:

- Does it mean that it will be possible to create realtime apps with python? Will it be possible to use it instead of node.js? Do you think Django will implement that functionality?

- Now that WebAssembly is coming - doesn't it mean that it could be possible to create both backend and frontend for realtime apps in python?

That would be so cool....

It's always been possible to create realtime apps with Python. Callbacks have been available since the beginning, and frameworks like Tornado and Twisted have been doing something very similar to the new async/await functionality with generators since Python 2.5 (or maybe even older). The async/await keywords are a nice performance improvement and work in a few places that "yield" doesn't, but they're not adding fundamentally new capabilities.

> Does it mean that it will be possible to create realtime apps with python?

I'm not sure what you mean by realtime but usually real time means something else and requires more deterministic behavior.

If you mean applications that use non-blocking IO to great extent then yes, but this depends a lot on the ecosystem to provide libraries for async file io, database io and so on. The Python language left good infrastructure to use though.

> Now that WebAssembly is coming - doesn't it mean that it could be possible to create both backend and frontend for realtime apps in python?

Not anymore than we could before with asm.js unfortunately as seen in http://repl.it/languages/python3 the problem is wrapping and the library, I hope a company "picks up the glove" and makes "Python for the frontend", sharing Python code between backend and frontend could be amazing.

People have started to hijack the word realtime. It's truly sad, because now it just muddies up people's conversations. So now people have to ask, is it the one that keeps planes in the air or the one that makes sure your email badge counter updates.

It is unfortunate. In most cases you can assume until proven otherwise that when someone says "realtime" they mean "really really soft soft-realtime" (sometimes referred to as "flaccidtime") rather than hard- or soft-realtime. Who needs planes when you have Javascript?

I wish there was a better alternative term for the web sense that I could promote, just to help un-hijack the word before it gets out of control. But I don't know of one that is likely to succeed. 'Responsive' makes sense to me but it already has another meaning in the web world. What about something like 'reactive' or 'on-the-fly'? People will never go for 'near real-time' because it sounds too weak in terms of its advertising value. Someone else below me suggested 'push', which is pretty decent but again it's an overloaded term... Thoughts or suggestions? :)

I'm not much of a wordsmith, but I am leaning towards "forthcoming".

Yes, for better or for worse, "realtime" (and sometimes "near-realtime") are used in the web community to refer to push technology as opposed to time constraints.

"Will it be possible to use it instead of node.js?"

It's been possible to use Python instead of node.js since before node.js existed. And by "possible" I mean "tons of people have been shipping code", not just "it's theoretically possible but nobody does it".

Node ported a well-established existing technique to Javascript, it didn't invent it.

As for Django, I think this is an exciting idea: https://gist.github.com/andrewgodwin/b3f826a879eb84a70625

I don't know that it's possible to have a realtime application running in an interpreter (which makes no realtime guarantees as far as I know) running on top of a soft-at-best-realtime OS.

One thing I don't really get, why is the "async" keyword necessary at all? Couldn't just the mere presence of "await" in the function body implicitly make the function "async"?

I guess the reason is that it changes the type of the return value. Compare:

    def moo():
        return 5

    async def oink():
        return 5

moo returns 5, oink returns an awaitable.

For a human, without the 'async' marker, you'd have to scan the entire function source to know. I guess also type checkers and documentation tools like to have an idea about the return type.

The presence of "yield" also changes the return type. No extra keywords needed for that.

Ah right, that makes sense.

This is directly addressed in the PEP [1] - the short of it is, without the `async` modifier if you have two functions:

   def important():
       await something()

   def something():
       # ... stuff ...
       await something_else()
       # ... more stuff ...

and you refactor `await something_else()` into a new method then you've changed the return type of `something` from `Awaitable[ReturnType]` to just `ReturnType` and `important` will break at run-time when it tries to `await something()`.

  [1]: https://www.python.org/dev/peps/pep-0492/#importance-of-async-keyword

Self documenting code for one.

I suppose you could follow the .NET Naming convention of appending "Async" to all routines.

Me too. There will be less of a switch in mindset for me after working in JS for a while then switching to Python for the backend.