> so if you want to pass data between them, you must copy
Big binaries aren’t copied; they’re passed by reference to a shared ref-counted heap (where each process holds one reference to the binary on the shared heap, and the process’s heap can hold N references to the far reference pointer.)
In theory, more types could be made to do that.
In fact, I believe that handles passed back from NIFs actually reuse this infrastructure, presenting themselves as binaries such that sending the handle to another process is really just creating a new smart pointer in the new process-heap, holding the same raw pointer that the original NIF handle held.
And you can do a lot with that. Erlang’s new-ish `counters` module, where you get a handle to a mutable uint64 array, is essentially just a built in NIF that passes around a mutable NIF handle pointing to a uint64[] buffer, exactly as above.
There’s nothing stopping the Erlang runtime (or your own NIF code) from adding all sorts of other operations against native, mutable types, and exposing them as these sorts of abstract data structure handles. You could, for example, have a `matrix` module for doing matrix math; or a `data_frame` module that can participate in every operation a Pandas.DataFrame can (with all such operations implemented as native code); or a module exposing SHM buffers; or even a module exposing GPU driver primitives (e.g. vertex/shader/texture buffers).
Everything that the bytecode would do to such ADTs would be slow-ish, but the point would be to load stuff into them in your glue code, and then in your hot loop, just bump already-loaded ADTs together in fast, native ways.
See the sibling comments, passing binary refs that refer to Matrex blobs is pretty much what the Matrex library is doing. The refs returned are in a binary format (not sure exactly if it's just the C memory representation or based on Matlab's `.mat` format). It'd be possible to say utilize Elixir macros to build up an AST for common math formulas that could be passed to the native maths library for example which could be pretty fast.
Big binaries aren’t copied; they’re passed by reference to a shared ref-counted heap (where each process holds one reference to the binary on the shared heap, and the process’s heap can hold N references to the far reference pointer.)
In theory, more types could be made to do that.
In fact, I believe that handles passed back from NIFs actually reuse this infrastructure, presenting themselves as binaries such that sending the handle to another process is really just creating a new smart pointer in the new process-heap, holding the same raw pointer that the original NIF handle held.
And you can do a lot with that. Erlang’s new-ish `counters` module, where you get a handle to a mutable uint64 array, is essentially just a built in NIF that passes around a mutable NIF handle pointing to a uint64[] buffer, exactly as above.
There’s nothing stopping the Erlang runtime (or your own NIF code) from adding all sorts of other operations against native, mutable types, and exposing them as these sorts of abstract data structure handles. You could, for example, have a `matrix` module for doing matrix math; or a `data_frame` module that can participate in every operation a Pandas.DataFrame can (with all such operations implemented as native code); or a module exposing SHM buffers; or even a module exposing GPU driver primitives (e.g. vertex/shader/texture buffers).
Everything that the bytecode would do to such ADTs would be slow-ish, but the point would be to load stuff into them in your glue code, and then in your hot loop, just bump already-loaded ADTs together in fast, native ways.