Bashbashparallelconcurrencyautomationproduction-ready

Parallel Task Execution

Shell patterns for running multiple tasks in parallel with job control, progress tracking, error collection, and result aggregation.


title: "Parallel Task Execution" description: "Shell patterns for running multiple tasks in parallel with job control, progress tracking, error collection, and result aggregation." category: "Bash" language: "bash" tags: ["bash", "parallel", "concurrency", "automation", "production-ready"] createdAt: "2025-01-20" updatedAt: "2025-02-10"

Overview

A robust parallel execution framework for shell scripts that limits concurrency, tracks progress, collects errors, and provides a summary report. Useful for deploying to multiple servers, batch processing files, or running tests in parallel.

Code

MAX_PARALLEL=8
for item in "${items[@]}"; do
    run_with_limit "process $item" "$item"
done
wait_for_all

Notes

  • set -euo pipefail ensures failures are caught (exit on error, undefined vars, pipe failures)
  • The PID array tracks running background jobs; kill -0 tests if a process is alive
  • Temporary files with trap cleanup prevent leftover files on script exit or interrupt
  • MAX_PARALLEL can be tuned based on target system load (SSH connections, API rate limits)
  • For simpler parallelism without job limiting, consider xargs -P or GNU parallel