Python and SysV Shared Memory

sgarland · 2024-10-12T13:04:10 1728738250

I was slightly confused until the end, when the author mentioned that all of this is unnecessary in 3.8, with the advent of multiprocessing.shared_memory.

3.8 really was a banger of a release. SHM, NamedTuples got a 2x speed boost for lookups, Lists got 12% faster, tons of new types, lru_cache as a decorator, math.dist() so people could stop installing NumPy just for that…

On SHM specifically, though, it really is quite useful for certain applications, and I wish it got more attention. It’s particularly nice for IPC with C shared libraries.

NewJazz · 2024-10-12T16:25:00 1728750300

That math dist thing reminds me of dateutil's relativedelta function. I really wish that were in the stdlib.

Galanwe · 2024-10-12T16:39:42 1728751182

I dont quite understand why the author bothers with creating ctypes bindings to the shm C API while using public shared memories.

You are better served by using a tmpfs (e.g. /dev/shm) and the regular python file API + mmap.

Also for reading and writing, the struct module (e.g. pack) is much more idiomatic than mapping a C style array.

emmelaich · 2024-10-12T18:45:23 1728758723

> At work-work the system uses, for historical reasons, a lot of SystemV shared memory ...

> I wanted to mess with the shared memory on such a system, from Python for convenience,

m3047 · 2024-10-12T18:44:39 1728758679

Fun! I call C funcs from python every few months. I rarely use ctypes, preferring to use buffers / bytearrays along with pack / unpack and a little declarative module for defining python analogs of c structs or maybe some numpy. I've never had a need to do shared memory in python (or in the last 20 years for that matter), but a long time ago I did a fair amount with it in DEC VMS.