correlation-analysis

Correlation Analysis

Find statistical relationships between metadata and content.

Example questions

• "Are my long voice notes more about work or personal?"
• "Do notes created late at night mention different topics than daytime ones?"
• "Is there a correlation between the source (phone vs desktop) and the topics I capture?"
• "Which entities co-occur with which categories?"

Procedure

1. Assemble feature matrix: one row per document with
   • Metadata columns: timestamp-derived (hour, weekday, month), source, author, length.
   • Content columns: topic assignments (topic-analysis), categories (categorize-corpus), entity presence flags (ner-extraction), parametric metrics (parametric-analysis), sentiment score.
2. Pick the right test by variable types:
   • Categorical × categorical → chi-square, Cramér's V.
   • Categorical × continuous → ANOVA / Kruskal-Wallis; effect size via eta-squared.
   • Continuous × continuous → Pearson (normal) or Spearman (monotonic).
   • Many categorical pairs → mutual information.
3. Correct for multiple testing: Bonferroni or Benjamini-Hochberg FDR. Running 100 correlations will surface 5 spurious ones at p<0.05.
4. Report effect size, not just p-values. In a 10k-doc corpus, trivially small effects hit significance. Require both statistical significance and a practical threshold (e.g. Cramér's V ≥ 0.1).
5. Co-occurrence matrices for entity×entity, entity×category, category×timebucket — useful visualizations.
6. Output:
   • correlations.csv: feature_a, feature_b, test, statistic, p_value, effect_size, p_adjusted
   • correlations-summary.md: top findings with plain-English sentences.

No LLM needed for the stats.

Optional: use an LLM to narrate the top findings into a readable report — one call, cheap.

Pitfalls

• Correlation ≠ causation. State relationships, don't imply cause.
• Confounders: topic-share over time often confounds with corpus volume change. Always check via trend-analysis normalization first.
• Timezones: if user travels, raw hour-of-day correlations are noise. Normalize to local time if possible.