Zack Scholl

zack.scholl@gmail.com

Worker pools in Golang

 / #golang 

A simple pattern to get started using worker pools with Golang.

In Go there is one boilerplate that I use more than any other - the worker pool. The worker pool that I like most is the one presented by Go By Example which I modify each time I use.

Since I like to have my own control over the worker pool I write the code each time. Each time it requires implementation of a worker function, and the job and result types. However, to make it easier, I’ve boiled it down to a simple copy and paste with only 5 steps of user input:

 1// step 1: specify the number of jobs
 2var numJobs = 1
 3
 4// step 2: specify the job and result
 5type job struct {
 6
 7}
 8type result struct {
 9	err error
10}
11
12jobs := make(chan job, numJobs)
13results := make(chan result, numJobs)
14runtime.GOMAXPROCS(runtime.NumCPU())
15for i := 0; i < runtime.NumCPU(); i++ {
16	go func(jobs <-chan job, results chan<- result) {
17		for j := range jobs {
18			// step 3: specify the work for the worker 
19			var r result  				
20			
21			results <- r
22		}
23	}(jobs,results)
24}
25
26// step 4: send out jobs
27for i:=0;i<numJobs;i++{
28	jobs <- job{}
29}
30close(jobs)
31
32// step 5: do something with results
33for i := 0; i < numJobs; i++{
34	r := <-results
35	if r.err != nil {
36		// do something with error
37	}
38}

The main change is that I’ve created a job and a result type which you can populate with anything you want exchanged between the worker and main thread. Also I use runtime to automatically utilize all the porcessors.

Hope that might work for you too!