Shared, streaming Python dict
Sychronized, streaming Python dictionary that uses shared memory as a backend
Warning: This is an early hack. There are only few unit tests and so on. Maybe not stable!
Features:
- Fast (compared to other shareing solutions)
- No running manager processes
- Works in spawn and fork context
- Safe locking between independent processes
- Tested with Python >= v3.9 on Linux and Windows
- Optional recursion for nested dicts
General Concept
UltraDict uses multiprocessing.shared_memory to synchronize a dict between multiple processes.
It does so by using a stream of updates in a shared memory buffer. This is efficient because only changes have to be serialized and transferred.
If the buffer is full, UltraDict will automatically do a full dump to a new shared
memory space, reset the streaming buffer and continue to