datascience

Data Science Standards

Conventions for data science projects.

Notebook Naming

Pattern: NN-initials-description.py (Jupytext percent format)

• Two-digit execution order: 01, 02, 03
• Author initials: kj for Konrad Jelen
• Brief description: data-exploration, train-yolov8m
• Examples: 01-kj-data-exploration.py, 04-kj-train-yolov8m.py

Sequential numbering within groupings. Archive obsolete to @archive/. Never delete. temp_ prefix for temporary notebooks excluded from Git.

File Format

Jupytext percent format (.py) = source of truth. # %% for code, # %% [markdown] for markdown. Add *.ipynb to .gitignore. Enables git diffs and code review.

Project Structure (cookiecutter-data-science)

data/raw/          # Original immutable datasets (never modify)
data/interim/      # Intermediate transformed data
data/processed/    # Final canonical datasets
data/external/     # Third-party data
notebooks/         # Jupytext notebooks
src/               # Reusable Python modules extracted from notebooks
models/            # Trained model artifacts
reports/           # Generated analysis and figures

PyTorch Model Artifacts

models/<model_name>/
  model.pt        # TorchScript (torch.jit.load, no class needed, for inference)
  checkpoint.pt   # State dict (needs class definition, for retraining)

Folder rolling: current → -1 → -2, up to 5 versions.

Code Standards

• Imports: never into __init__.py. Always explicit module imports
• Docstrings: Google format, type hints for params and returns
• DataFrames: purpose_df for DataFrames, purpose_lf for LazyFrames
• Rich output: from rich.jupyter import print
• Polars: lazy (pl.LazyFrame + collect()) for large datasets
• Prefer builtins: sklearn.model_selection.train_test_split over manual
• Plots: figsize=(12, 6), matplotlib + seaborn

EDA First

Start thorough: data types, missing values, basic statistics, visual exploration (histograms, scatter, box plots), testable hypotheses. High-dimensional: try UMAP/t-SNE.