parallel processing of data

[wdduncan]

Is there a way to leverage parallelism in Logica? Often I use parallelism by breaking the data into groups (using group by) and then processing each group on different cores. Can I do something similar in Logica?

This discussion may be related to #305.

[EvgSkv]

Okay, basically we are thinking of integrating with DuckDB. This would solve this and #305 at the same time.

Now you can load data to BigQuery and BQ will parallelize across hundreds or thousands of machines, but I am guessing you are looking for a more light weight solution.

As I mentioned in #305 DuckDB is expected to come no later than June, hoping for sooner.

[wdduncan]

Thanks. Looking forward to the new features!

[EvgSkv]

DuckDB support was added to Logica. You can use it with @Engine("duckdb"); imperative.
DuckDB parallelizes processing automatically, so whatever query you run, it should be using as many cores as useful. I did some quick checks and it indeed loads a lot of cores.
Let me know if you hit any issues!

[wdduncan]

Thanks @EvgSkv !