datamash

What it does

datamash performs numeric, textual, and statistical operations on tabular text input. Think: quick summaries and “group by” calculations without writing awk scripts or firing up a spreadsheet.

# Install (examples)
# Debian/Ubuntu:
sudo apt install datamash
# RHEL/Alma/Rocky (EPEL often):
sudo dnf install datamash

How it works (mechanical)

• Reads rows from stdin (or a file) and splits into fields (default: whitespace).
• Performs operations like sum, mean, min, max, count, median, etc.
• Grouping is done via -g / groupby (like SQL GROUP BY).
• Delimiters can be set with -t (e.g., CSV: -t,).
• Headers are supported with -H / --header-in/--header-out.

10 Practical Examples

# 1) Sum and mean of numbers 1..10
seq 10 | datamash sum 1 mean 1

# 2) Count lines (rows)
cat file.txt | datamash count 1

# 3) Group by column 1, sum column 2
# input: KEY VALUE
datamash -g 1 sum 2 < data.txt

# 4) Group by multiple columns, compute mean + sample stdev
datamash -g 1,2 mean 3 sstdev 3 < data.txt

# 5) CSV mode (comma delimiter): group by col 2, mean col 5
datamash -t, -g 2 mean 5 < data.csv

# 6) Use headers (refer to columns by name)
datamash -t, -H groupby Department mean Salary < employees.csv

# 7) Quick “top talkers”: sort then group
# Example: count occurrences of field 1
awk '{print $1}' access.log | sort | datamash -g 1 count 1 | sort -k2,2nr | head

# 8) Unique count per group
datamash -g 1 countunique 2 < data.txt

# 9) Collapse values per group into a single field (comma-separated)
datamash -g 1 collapse 2 < data.txt

# 10) Validate “looks like a table” (great for pipeline sanity checks)
datamash check < maybe_tabular.txt

Notes & Gotchas

• Sorting: grouping often assumes grouped rows; use --sort or sort input yourself.
• Delimiter pain: CSV with quoted commas is not “simple delimiter” territory — use a CSV-aware tool if needed.
• Headers: -H makes life easier, but requires clean header names.
• Numeric parsing: non-numeric junk in numeric fields will break math ops (clean first).

Historical Context

datamash is a GNU project tool built for quick command-line computation on plain text tables — a natural evolution of the Unix pipeline style: cut + sort + “math” without writing a full script.

Modern Equivalent

For heavier analytics or true CSV/JSON parsing, you might reach for:

• awk (custom), python/pandas (powerful),
• csvkit (CSV-aware),
• jq (JSON).

But for “fast, good-enough stats on tabular text,” datamash is hard to beat.

Related Commands

• awk — programmable text/data transforms
• sort, uniq — grouping primitives
• cut, column — field selection / display
• csvcut, csvstat (csvkit) — CSV-aware tooling