so I have been trying this for past few weekends - implementing a distributed queue in golang. although, this is an age-old problem, and state-of-art solutions exist - kafka, rabbit MQ, etc, but I still gave my thought experiment some time started off.
the initial experiment
my initial experiment was to write a queue shared within a process - and extensible to other memory backends. the implementation was a naive ring-buffer which multiple goroutines can enqueue/dequeue from without bumping into each other's offset.
objectives
had some pretty cool nuances, and set a few objectives -
-
queue should support multiple memory backends, could be process memory, shared memory, redis or any persistent disc.
-
the queue would be partitioned, and unlike other queues, multiple consumers could read from the same partition. this way, we could be writing to any partition and scaling them as-and-when needed.
-
interestingly each partition could be served on different backends while the queue would operate deterministically. so producer, and consumer both can set options as to what memory backend they want to use. I know this doesn't make much sense, but I felt that there are some use cases for this too.
-
processing should be blazing fast, with little memcopy/system call overheads, without being too rigid. wrote interesting performance tests for this.
-
queueing could be async/sync both - depending on per message level.
so, I wrote off a simple implementation with async and sync support, got some interesting learnings and once went into a deadlock, waiting for a semaphore signal when all locked goroutines went to sleep. will share more in the next post.