Bashbashparallelconcurrencyautomationproduction-ready
Parallel Task Execution
Shell patterns for running multiple tasks in parallel with job control, progress tracking, error collection, and result aggregation.
title: "Parallel Task Execution" description: "Shell patterns for running multiple tasks in parallel with job control, progress tracking, error collection, and result aggregation." category: "Bash" language: "bash" tags: ["bash", "parallel", "concurrency", "automation", "production-ready"] createdAt: "2025-01-20" updatedAt: "2025-02-10"
Overview
A robust parallel execution framework for shell scripts that limits concurrency, tracks progress, collects errors, and provides a summary report. Useful for deploying to multiple servers, batch processing files, or running tests in parallel.
Code
MAX_PARALLEL=8
for item in "${items[@]}"; do
run_with_limit "process $item" "$item"
done
wait_for_allNotes
set -euo pipefailensures failures are caught (exit on error, undefined vars, pipe failures)- The PID array tracks running background jobs;
kill -0tests if a process is alive - Temporary files with
trapcleanup prevent leftover files on script exit or interrupt MAX_PARALLELcan be tuned based on target system load (SSH connections, API rate limits)- For simpler parallelism without job limiting, consider
xargs -Por GNUparallel