Is it possible to run python async functions in different Haskell threads?

[qwbarch]

As a quick and dirty test I tried the following:

  runPyInMain [pymain| import asyncio|]
  foo ←
    runPyInMain
      [pyf|
      async def foo():
        print("running foo")
        await asyncio.sleep(10)
        print("ending foo")
      
      return foo
      |]
  _ ←
    forkIO $
      runPy
        [py_|
        asyncio.run(foo_hs())
        |]
  _ ←
    forkIO $
      runPy
        [py_|
        asyncio.run(foo_hs())
        |]

This causes the second thread to wait until the first thread to run its async function (as expected since it's waiting for the global lock). However, top-level async calls are not supported so I can't seem to be able to call async functions from multiple places without it blocking the global lock.

Is it possible run multiple async functions at the same time, but calling it from the Haskell side?

[Shimuuar]

Right now it's not. Haskell code uses single global lock because when haskell thread is blocked on lock scheduler can run other threads on capability (OS thread). And attempting to acquire GIL will block capability.

One possible solution is to run python code in separate OS thread. Something like runAsyncPy :: Py a -> IO a and/or runAsyncPySTM :: Py a -> IO (STM a) which spawns new OS thread for running python code.

P.S. Out pf curiosity. What are you trying to use async python in haskell for?

[qwbarch]

That's what I figured, although I was trying to avoid having to spawn a new thread for each async python call.

P.S. Out pf curiosity. What are you trying to use async python in haskell for?

Mainly just trying to run a bunch of I/O bound functions from python libraries. I’d prefer to interface with those functions using inline-python and make most of the calls from haskell, rather than doing most logic from within python and then returning to haskell.

Thank you so much for the advice!

[Shimuuar]

I skimmed docs for asyncio.run. And it seems it does need whole OS thread for itself:

This function cannot be called when another asyncio event loop is running in the same thread.

[Shimuuar]

Now that #41 is merged there's support for running concurrent python. It's very experimental. @qwbarch I would love to know whether it works for you and whether you encountered any problems.

P.S. By Zeus, I got nerd-sniped.