WorkerPool - Simple Concurrent Processing

A lightweight, generic worker pool implementation for efficient concurrent processing in Go.

Features

• Generic Processing: Works with any data type using Go generics (1.18+)
• Adaptive Buffering: Configurable buffer scaling for optimal throughput
• Auto-Recovery: Graceful handling of edge cases and invalid parameters
• Zero Dependencies: Pure Go standard library implementation
• UNLICENSE: Public domain - use however you want

Installation

go get github.com/toxyl/workerpool

Usage

package main

import (
	"fmt"
	"github.com/toxyl/workerpool"
)

func main() {
	data := []int{1, 2, 3, 4, 5}
	
	workerpool.Process(
		4,    // numWorkers
		10,   // bufferScale
		data, // slice to process
		func(n int) {
			fmt.Printf("Processing %d\n", n)
		},
	)
}

Parameter	Description
numWorkers	Number of concurrent processors (<=0 defaults to 1)
bufferScale	Buffer multiplier: buffer size = numWorkers * bufferScale (min 0)
data	Slice of items to process
fn	Processing function (must be concurrency-safe)

Throughput Optimization

• Larger buffers reduce goroutine contention
• Higher worker counts improve CPU utilization
• Balance based on task characteristics and hardware
• Buffer memory allocation = numWorkers * bufferScale * sizeof(T)

Buffer Scale Recommendations

Duration	BufferScale	Use Case Example
< 1ms/item	50-100	Simple math operations
1-100ms/item	10-50	Network requests
> 100ms/item	1-5	Complex calculations
Memory-sensitive	1-3	Large payload processing

Examples

//  High Throughput Configuration
// Process 1M items with 100 workers and large buffer
workerpool.Process(100, 100, bigData, fastOperation)

// Memory-Constrained Processing
// Minimal buffer for large items
workerpool.Process(4, 1, hugePayloads, memoryIntensiveTask)

// Unbuffered Mode
// Synchronous processing (bufferScale = 0)
workerpool.Process(1, 0, sensitiveData, threadSafeOperation)