boto3-consistency

boto3 — consistent idioms

boto3 is stable, but generated code inherits the ecosystem's classic silent bugs: list calls without pagination (truncated at 1000 results, no error), sleep-polling where waiters exist, the feature-frozen resource interface in new code, string-matched exceptions, and hardcoded credentials. This skill pins the canonical client-first idiom set.

Canonical idioms — always X, never Y

Always	Never	Why
boto3.client("s3")	boto3.resource("s3") in new code	The resource interface is feature-frozen (no new APIs/services); clients cover everything and match the AWS API docs.
paginate every list/describe: for page in client.get_paginator("list_objects_v2").paginate(Bucket=b):	a single list_objects_v2 call	List APIs return one page (≤1000 items) plus a token; ignoring IsTruncated silently processes a fraction of the data.
client.get_waiter("object_exists").wait(...)	while True: describe...; time.sleep(5)	Waiters encode the correct polling intervals, terminal states, and failure states.
except ClientError as e: + e.response["Error"]["Code"] == "NoSuchKey" (or client.exceptions.NoSuchKey)	except Exception / matching str(e)	Error codes are the contract; message strings change.
explicit region/profile via boto3.session.Session(region_name=..., profile_name=...) or env config	hardcoded aws_access_key_id="AKIA..." literals	Credentials belong to the provider chain (env, profile, IAM role); literals leak.
create clients once (module/app scope) and reuse	a new client per function call	Client creation loads models/credentials — slow; clients are thread-safe (resources/sessions are not).
S3 files: client.upload_file / download_file / upload_fileobj	raw put_object/get_object for large files	Managed transfers do multipart, retries, and concurrency automatically (>5 GB requires multipart).
DynamoDB conditions via boto3.dynamodb.conditions.Key/Attr (resource) or exact AttributeValue maps (client) — one layer, consistently	mixing the two formats	The client wants {"S": "abc"} maps; the resource wants Python types. Mixing produces serialization errors or wrong queries.
Config(retries={"mode": "standard", "max_attempts": 5})	hand-rolled retry loops around every call	botocore retries throttling/transient errors correctly, with backoff.
Stubber/moto in tests	unittest.mock-ing boto3 internals	Stubber validates request parameters against the real service model.

House style:

import boto3
from botocore.config import Config
from botocore.exceptions import ClientError

session = boto3.session.Session()                  # region/creds from environment
s3 = session.client("s3", config=Config(retries={"mode": "standard", "max_attempts": 5}))

def iter_keys(bucket: str, prefix: str = ""):
    paginator = s3.get_paginator("list_objects_v2")
    for page in paginator.paginate(Bucket=bucket, Prefix=prefix):
        yield from (obj["Key"] for obj in page.get("Contents", []))

def get_json(bucket: str, key: str) -> dict | None:
    try:
        body = s3.get_object(Bucket=bucket, Key=key)["Body"].read()
    except ClientError as e:
        if e.response["Error"]["Code"] == "NoSuchKey":
            return None
        raise
    return json.loads(body)

Pitfalls that produce silently wrong results

• The 1000-item cliff: list_objects_v2, list_functions, describe_instances... every list API pages. Code without paginators works in dev (small accounts) and loses data in prod. Audit every list_*/describe_* call.
• Empty-result key absence: page["Contents"] raises KeyError on an empty bucket — page.get("Contents", []) always.
• DynamoDB scan vs query: query needs the partition key and is indexed; scan reads the whole table (cost + latency) — a scan with a FilterExpression still reads everything; filters apply after. Scans also paginate (LastEvaluatedKey).
• batch_write_item leaves UnprocessedItems under throttling — loop until empty (the resource's batch_writer() context manager does this for you).
• Throttling looks like failure: ProvisionedThroughputExceededException/ ThrottlingException are retryable — configure retry mode rather than catching ad hoc.
• Presigned URLs: generate_presigned_url succeeds even with credentials that can't access the object — the URL just 403s later; the signer's permissions and expiry are part of the contract. Region-mismatched clients produce redirect-broken URLs.
• Lambda invoke payloads are streams: json.loads(resp["Payload"].read()), and a 200 with "FunctionError" set is a failed function — check it.
• Credentials at import time: clients created at module import in Lambda/containers freeze the role/region of the first import context; fine normally, surprising in tests — inject sessions where flexibility matters.
• S3 "directories" don't exist: prefixes are key substrings; checking folder existence or "creating" folders is key-naming logic, not API calls.

Version notes

Target current boto3 1.x (the API surface is stable; service models update in lockstep with AWS). The resource interface (boto3.resource) entered maintenance mode in 2023 — existing resource code (especially DynamoDB's ergonomic Table) keeps working, and Table.batch_writer() remains the one resource feature worth reaching for; just don't start designs on resources. Async needs aioboto3/aiobotocore (separate packages), not async def wrappers around sync clients.

Workflow

1. One Session per process (or per assumed role), clients per service created once; region and credentials from the environment chain.
2. For every API call decide: pagination needed? waiter available? which error codes are expected (handle) vs unexpected (propagate)?
3. Configure retries on the client Config; add idempotency where the service supports it (client tokens) for mutating calls.
4. S3 via managed transfers; DynamoDB via one serialization layer; batch APIs with unprocessed-item loops.
5. When reviewing, hunt: unpaginated lists, sleep-polls, except Exception, hardcoded keys/regions, per-call client creation, scan-where-query, unchecked batch leftovers.

For the pagination/waiter catalogs, error-code handling table, DynamoDB layer comparison, S3 transfer reference, and testing setups, read references/boto3-patterns.md.