Zack Scholl

zack.scholl@gmail.com

Worker pools in Golang

 / #golang 

A simple pattern to get started using worker pools with Golang.

In Go there is one boilerplate that I use more than any other - the worker pool. The worker pool that I like most is the one presented by Go By Example which I modify each time I use.

Since I like to have my own control over the worker pool I write the code each time. Each time it requires implementation of a worker function, and the job and result types. However, to make it easier, I’ve boiled it down to a simple copy and paste with only 5 steps of user input:

// step 1: specify the number of jobs
var numJobs = 1

// step 2: specify the job and result
type job struct {

}
type result struct {
	err error
}

jobs := make(chan job, numJobs)
results := make(chan result, numJobs)
runtime.GOMAXPROCS(runtime.NumCPU())
for i := 0; i < runtime.NumCPU(); i++ {
	go func(jobs <-chan job, results chan<- result) {
		for j := range jobs {
			// step 3: specify the work for the worker 
			var r result  				
			
			results <- r
		}
	}(jobs,results)
}

// step 4: send out jobs
for i:=0;i<numJobs;i++{
	jobs <- job{}
}
close(jobs)

// step 5: do something with results
for i := 0; i < numJobs; i++{
	r := <-results
	if r.err != nil {
		// do something with error
	}
}

The main change is that I’ve created a job and a result type which you can populate with anything you want exchanged between the worker and main thread. Also I use runtime to automatically utilize all the porcessors.

Hope that might work for you too!