avatarSubarashi golang pooling 🍑

Go is great for setting up easy parallel jobs and processes, however, it is not easy when one starts confusing concurrency with parallelism and ending up endlessly fighting race conditions. komi is a generic pooling library that will satisfy your hunger.

Komi writes Go
Komi writes Go


Say you want to run a function foo(v) that performs some kind of work on parameter v, be it a database operation, a syscall, an IO operation, etc. (possibilities are endless!) Setting up a pool and sending jobs is as trivial as

pool := komi.New(komi.WorkSimple(foo))
defer pool.Close()
// other code...
pool.Submit(v) // will block if pool is full

Notice that pool.Close() will gracefully free all the resources and channels occupied by pool by waiting for final jobs to complete. pool.Close(true) will force pool closure.

But what if you want to collect outputs of work performed on v with foo(v) w?

pool := komi.New(komi.Work(foo))
defer pool.Close()
// collect outputs with pool.Outputs() channel
go func() {
	for output := range pool.Outputs() {
		// output is the result of `foo(v)`
// other code...
pool.Submit(v) // will block if pool is full

But what if you want to collect errors as well? Consider foo(v) error

pool := komi.New(komi.WorkSimpleWithErrors(foo))
defer pool.Close()
// collect errors with with pool.Errors() channel...
// other code...
pool.Submit(v) // will block if pool is full

Or with foo(v) (w, error)!

pool := komi.New(komi.WorkWithErrors(foo))
defer pool.Close()
// collect outputs with pool.Outputs() channel
// collect errors with pool.Errors() channel...
// other code...
pool.Submit(v) // will block if pool is full

So, depending on what function you give, any work type is handled by the pool on the fly! If work given doesn’t produce outputs, pool.Outputs() will return nil, similarly, if work given doesn’t produce errors, pool.Errors() will return nil.

Note: if work produces outputs or errors, those activated channels need to be consumed by the user, otherwise, when reaching size number of elements in either (if active), work will be blocked until the destination channel is consumed.


Unique feature of komi is that each pool can be connected with each other. Say you have two functions, where one opens file’s contents, openFile(filename string) (string, error), and the other counts the number of words, countWords(contents string) int.

Two pools can be created,

opener := komi.NewWithSettings(komi.WorkWithErrors(openFile), &komi.Settings{
	Name:     "Opener πŸ“‚ ",
	Laborers: 1,
	Size:     4,

counter := komi.NewWithSettings(komi.Work(countWords), &komi.Settings{
	Name:     "Counter πŸ“š ",
	Laborers: 10,
	Size:     20,

We can wire the outputs of opener to be automatically fed into counter with


So now, those two pools are "connected". We would call this relationship as opener being the dependent (child) pool and counter being the connected (parent) pool.

                Opener πŸ“‚                   Counter πŸ“š
  filenames  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  contents   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  word counts
   ───────>  β”‚  openFile   β”‚  ────────>  β”‚  countWords  β”‚  ────────>
   .Submit   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  .Connect   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  .Outputs
              pool: opener                 pool: counter

komi’s pools are smart enough that only by calling counter.Close(), it will issue a shutdown command back to opener and wait until it’s closed. This closing logic procedure will happen with any number of connected pools.

If you have pools 1,2,...,N-1,N connected in form 1->2->...->N-1->N, user can only call .Close() on pool N, as it would be responsible for sending a closure request to N-1, and so on until 2 sends the shutdown request to 1. When 1 is closed, the closure will resume on 2, up until N-1 and N, where the latter will return from the original Close() call.

Please note that none of the pools 1,2,...,N-1 in the above will honor user’s closure request, as it should come from their connected (parent) pool.


When the parent-most pool is closing, it will wait for all the child pools to complete their jobs. This behavior can be overwritten by calling .Close(true), which would skip any waiting of childrens’ queued jobs and skip any waiting of the parent-most pool’s queued jobs.


Pools support waiting (blocking) until the pool has no jobs waiting for completion with pool.Wait().

Some other quality of life operations are also provided,


You can tune the performance and behavior of the pool with komi.NewWithSetttings by providing *komi.Settings,


This is a brand new library I built for my static website generator, where it’s used extensively and in production. However, there are no guarantees provided for this library, that is, until something like v1.0 is out, in which case, I would promise to maintain backward compatibility.

Here is how Darkness πŸ₯¬ currently builds this website,

           Reading πŸ“š                      Parsing 🧹
   path  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   file handler   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
 ──────> β”‚ filesPool β”‚ ───────────────> β”‚  parserPool β”‚
         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
          log errors                          β”‚
                                              β”‚    parsed files
                                              β”‚  aka yunyun pages
   file  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  exported data  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  <───── β”‚ writerPool β”‚ <────────────── β”‚ exporterPool β”‚
         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                 β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
           Writing 🎸                     Exporting πŸ₯‚

Please use it with knowing your risks. However, if you use a tagged version or a commit hash in your go.mod, you should be fine.

Future work

Some future items in mind:


Please go to komi’s github page for more details and the code!

-> Komi - subarashi go pooling 🍑