jq-json-processing

jq JSON Processing

Expert knowledge for processing, querying, and transforming JSON data using jq, the lightweight and flexible command-line JSON processor.

Core Expertise

JSON Operations

• Query and filter JSON with path expressions
• Transform JSON structure and shape
• Combine, merge, and split JSON data
• Validate JSON syntax and structure

Data Extraction

• Extract specific fields from JSON objects
• Filter arrays based on conditions
• Navigate nested JSON structures
• Handle null values and missing keys

Essential Commands

Basic Querying

# Pretty-print JSON
jq '.' file.json

# Extract specific field
jq '.fieldName' file.json

# Extract nested field
jq '.user.email' file.json

# Extract array element
jq '.[0]' file.json
jq '.items[2]' file.json

Array Operations

# Get array length
jq '.items | length' file.json

# Map over array
jq '.items | map(.name)' file.json

# Filter array
jq '.items[] | select(.active == true)' file.json
jq '.users[] | select(.age > 18)' file.json

# Sort array
jq '.items | sort_by(.name)' file.json
jq '.items | sort_by(.date) | reverse' file.json

# Get first/last elements
jq '.items | first' file.json
jq '.items | last' file.json

# Unique values
jq '.tags | unique' file.json

Object Operations

# Get all keys
jq 'keys' file.json
jq '.config | keys' file.json

# Get all values
jq '.[] | values' file.json

# Select specific fields
jq '{name, email}' file.json
jq '{name: .fullName, id: .userId}' file.json

# Add field
jq '. + {newField: "value"}' file.json

# Delete field
jq 'del(.password)' file.json

# Merge objects
jq '. * {updated: true}' file.json

Filtering and Conditions

# Select with conditions
jq 'select(.status == "active")' file.json
jq '.[] | select(.price < 100)' file.json

# Multiple conditions (AND)
jq '.[] | select(.active and .verified)' file.json
jq '.[] | select(.age > 18 and .country == "US")' file.json

# Multiple conditions (OR)
jq '.[] | select(.type == "admin" or .type == "moderator")' file.json

# Exists / has field
jq '.[] | select(has("email"))' file.json
jq 'select(.optional != null)' file.json

# Not condition
jq '.[] | select(.status != "deleted")' file.json

String Operations

# String interpolation
jq '"Hello, \(.name)"' file.json

# Convert to string
jq '.id | tostring' file.json

# String contains
jq '.[] | select(.email | contains("@gmail.com"))' file.json

# String starts/ends with
jq '.[] | select(.name | startswith("A"))' file.json
jq '.[] | select(.file | endswith(".json"))' file.json

# Split string
jq '.path | split("/")' file.json

# Join array to string
jq '.tags | join(", ")' file.json

# Lowercase/uppercase
jq '.name | ascii_downcase' file.json
jq '.name | ascii_upcase' file.json

Pipes and Composition

# Chain operations
jq '.items | map(.name) | sort | unique' file.json

# Multiple filters
jq '.users[] | select(.active) | select(.age > 18) | .email' file.json

# Group by
jq 'group_by(.category)' file.json
jq 'group_by(.status) | map({status: .[0].status, count: length})' file.json

Output Formatting

# Compact output (no pretty-print)
jq -c '.' file.json

# Raw output (no quotes for strings)
jq -r '.message' file.json
jq -r '.items[] | .name' file.json

# Output as tab-separated values
jq -r '.[] | [.name, .age, .email] | @tsv' file.json

# Output as CSV
jq -r '.[] | [.name, .age, .email] | @csv' file.json

# Output as JSON array on one line
jq -c '[.items[]]' file.json

Input/Output Options

# Read from stdin
cat file.json | jq '.items'
curl -s https://api.example.com/data | jq '.results'

# Multiple input files
jq -s '.' file1.json file2.json  # Slurp into array

# Write to file
jq '.filtered' input.json > output.json

# In-place edit (use sponge from moreutils)
jq '.updated = true' file.json | sponge file.json

Advanced Patterns

# Recursive descent
jq '.. | .email? // empty' file.json

# Reduce (sum, accumulate)
jq '[.items[].price] | add' file.json
jq 'reduce .items[] as $item (0; . + $item.price)' file.json

# Variable assignment
jq '.items[] | . as $item | $item.name + " - " + ($item.price | tostring)' file.json

# Conditional (if-then-else)
jq '.items[] | if .price > 100 then "expensive" else "affordable" end' file.json
jq 'if .error then .error else .data end' file.json

# Try-catch for error handling
jq '.items[] | try .field catch "not found"' file.json

# Flatten nested arrays
jq '.items | flatten' file.json
jq '.items | flatten(1)' file.json  # Flatten one level

Real-World Examples

# Extract all emails from nested structure
jq '.. | .email? // empty' users.json

# Get unique list of all tags across items
jq '[.items[].tags[]] | unique' data.json

# Count items by status
jq 'group_by(.status) | map({status: .[0].status, count: length})' items.json

# Transform API response to simple list
jq '.results[] | {id, name: .full_name, active: .is_active}' response.json

# Filter GitHub workflow runs (recent failures)
gh run list --json status,conclusion,name,createdAt | \
  jq '.[] | select(.conclusion == "failure") | {name, createdAt}'

# Extract package.json dependencies with versions
jq '.dependencies | to_entries | map("\(.key)@\(.value)")' package.json

# Merge two JSON files
jq -s '.[0] * .[1]' base.json override.json

# Create summary from log data
jq 'group_by(.level) | map({level: .[0].level, count: length, samples: [.[].message][:3]})' logs.json

Best Practices

Query Construction

• Start simple, build complexity incrementally
• Test filters on small datasets first
• Use -c flag for compact output in scripts
• Use -r flag for raw strings (no quotes)

Performance

• Use select() early in pipeline to reduce data
• Use targeted queries with specific paths for large files
• Stream large JSON with --stream flag
• Consider jq -c for faster processing

Error Handling

• Use ? operator for optional access: .field?
• Use // empty to filter out nulls/errors
• Use try-catch for graceful error handling
• Check for field existence with has("field")

Readability

• Break complex queries into multiple steps
• Use variables with as $var for clarity
• Add comments in shell scripts
• Format multi-line jq programs for readability

Common Patterns

API Response Processing

# GitHub API: Get PR titles and authors
gh pr list --json title,author,number | \
  jq -r '.[] | "#\(.number) - \(.title) by @\(.author.login)"'

# REST API: Extract and flatten pagination
curl -s "https://api.example.com/items" | \
  jq '.data.items[] | {id, name, status}'

Configuration Files

# Extract environment-specific config
jq '.environments.production' config.json

# Update configuration value
jq '.settings.timeout = 30' config.json > config.updated.json

# Merge base config with environment overrides
jq -s '.[0] * .[1]' base-config.json prod-config.json

Log Analysis

# Count errors by type
jq 'select(.level == "error") | .type' logs.json | sort | uniq -c

# Extract error messages with timestamps
jq -r 'select(.level == "error") | "\(.timestamp) - \(.message)"' logs.json

# Group by hour and count
jq -r '.timestamp | split("T")[1] | split(":")[0]' logs.json | sort | uniq -c

Data Transformation

# CSV to JSON (with headers)
jq -R -s 'split("\n") | .[1:] | map(split(",")) |
  map({name: .[0], age: .[1], email: .[2]})' data.csv

# JSON to CSV
jq -r '.[] | [.name, .age, .email] | @csv' data.json

# Flatten nested structure
jq '[.items[] | {id, name, category: .meta.category}]' nested.json

Troubleshooting

Invalid JSON

# Validate JSON syntax
jq empty file.json  # Returns exit code 0 if valid

# Find syntax errors
jq '.' file.json 2>&1 | grep "parse error"

Empty Results

# Debug: Print entire structure
jq '.' file.json

# Debug: Check field existence
jq 'keys' file.json
jq 'type' file.json  # Check if array, object, etc.

# Debug: Show all values
jq '.. | scalars' file.json

Type Errors

# Check field types
jq '.field | type' file.json

# Convert types safely
jq '.id | tonumber' file.json
jq '.count | tostring' file.json

# Handle mixed types
jq '.items[] | if type == "array" then .[] else . end' file.json

Performance Issues

# Stream large files
jq --stream '.' large-file.json

# Process line by line
cat large.json | jq -c '.[]' | while read -r line; do
  echo "$line" | jq '.field'
done

Integration with Other Tools

# With curl (API calls)
curl -s "https://api.github.com/users/octocat" | jq '.name, .bio'

# With gh CLI (GitHub operations)
gh api repos/owner/repo/issues | jq '.[] | {number, title, state}'

# With find (batch processing)
find . -name "package.json" -exec jq '.version' {} \;

# With xargs (parallel processing)
cat urls.txt | xargs -I {} curl -s {} | jq '.data'

# With yq (YAML to JSON conversion)
yq eval -o=json file.yaml | jq '.specific.field'

Quick Reference

Operators

• .field - Access field
• .[] - Iterate array/object
• | - Pipe (chain operations)
• , - Multiple outputs
• ? - Optional (suppress errors)
• // - Alternative operator (default value)

Functions

• keys, values - Object keys/values
• length - Array/object/string length
• map(), select() - Array operations
• sort, sort_by() - Sorting
• group_by() - Grouping
• unique - Remove duplicates
• add - Sum numbers or concatenate
• has() - Check field existence
• type - Get value type

Type Conversions

• tostring, tonumber - Convert types
• @csv, @tsv, @json - Format output
• split(), join() - String/array conversion

Installation

# macOS (Homebrew)
brew install jq

# Ubuntu/Debian
sudo apt-get install jq

# Verify installation
jq --version

Resources

• Official Manual: https://stedolan.github.io/jq/manual/
• jq Playground: https://jqplay.org/
• Tutorial: https://stedolan.github.io/jq/tutorial/