fgcz-custom-analysis-register

fgcz-custom-analysis-register

Wrap a hand-written R Markdown analysis so it produces the same on-disk shape a SUSHI app would. The output drops cleanly into the SUSHI dataset graph: any downstream SUSHI app (ScSeurat, exploreSC, etc.) can chain off it, and the B-Fabric + SUSHI registrations (Steps 9–10) make it visible in the SUSHI UI and the B-Fabric audit trail.

A manually written .Rmd file needs to be adapated so that it fits the pattern of references/example-self-contained.Rmd.

• have a setup chunk to define absolute file paths
• analysis_definition chunk: specify the inputs and the parameters. It is important that the input dataset are explicity specified input_dataset.tsv. Those hold then links to the actual datafiles.
• have chunks doing the analysis and performing visualizations
• dataset_generation: final chunk that is executed at the end, generates all the files needed to run the reproducibly and to register the whole analysis in SUSHI and B-Fabric

The B-Fabric side is live via Ronald's register_custom_analysis.py (pending merge into btools main).

When to use

• You wrote an .Rmd, rendered it locally, and the user now wants the output delivered to gstore.
• The analysis consumes outputs from a SUSHI app (FeatureCounts, STAR, ScSeurat, CellRangerMulti…) and should be chainable from it in the SUSHI UI.
• You want the SUSHI/B-Fabric audit trail to show this analysis even though you didn't build a real SUSHI app.

Do NOT use for: - Building a real SUSHI app → use fgcz-sushi-app-dev.

• Just rendering an Rmd with retry/validation logic → use autonomous-render.
• Reading/writing B-Fabric workunits or datasets directly → use bfabric.

Inputs to gather before scaffolding

Confirm aloud with the user — echo back project + order before writing anything (the existing "confirm order numbers" rule applies here):

Field	Example	Notes
project	p40992	The pXXXXX directory under /srv/gstore/projects/.
order_id	o41017	The oXXXXX prefix matching the upstream order.
analysis_name	SC-FlashSeq_QC_Evaluation	Short, no spaces. Becomes part of the folder name and filenames.
timestamp	2026-05-15--12-00-00	Generate fresh with format(Sys.time(), "%Y-%m-%d--%H-%M-%S") — don't reuse a prior one.
Rmd path	~/git/<your-scripts>/pXXXXX_*/QC.Rmd	The source .Rmd.
Upstream SUSHI dataset path	/srv/gstore/projects/pXXXXX/oYYYYY_FeatureCounts_YYYY-MM-DD--...	Used in Step 4 to copy input_dataset.tsv for provenance. Without it, the SUSHI lineage tree has nothing to link back to.
Upstream SUSHI dataset ID	e.g. 109531	The numeric ID from production SUSHI's data_sets table (not the dev SUSHI ID, which is different — look it up via the URL fgcz-sushi.uzh.ch/data_set/pXXXXX/<id> or query the DB via the sushi-framework skill). Needed in Step 10 to set parent_id and build the lineage edge.

If the upstream SUSHI dataset path is unknown, ask the user; write it into the .Rmd file and make sure it is used — don't guess. Provenance without a real parent is worse than no provenance.

What this skill produces

/srv/gstore/projects/{project}/{order_id}_{analysis_name}_{timestamp}/
├── {analysis_name}.Rmd                # source (copied from your repo)
├── {analysis_name}.html               # rendered report
├── dataset.tsv                        # output dataset — SUSHI schema
├── parameters.tsv                     # analysis parameters
├── input_dataset.tsv                  # provenance — copied from upstream SUSHI dataset
├── *.qs2                              # any cached objects the Rmd writes
└── scripts/
    ├── {order_id}_run-{analysis}.sh   # vanilla bash launcher
    ├── {order_id}_run-{analysis}_o.log
    └── {order_id}_run-{analysis}_e.log

The folder name and the three TSVs are the exact contract every SUSHI app honours — that's what makes the output chainable.

Step-by-step recipe

Work locally in /srv/GT/analysis/{project}/Analyses_Paul/ (or a /tmp/ scratch dir for dry runs) Take an existing .Rmd fil; inspect it; Make a new folder {analysis_name}_scratch and place the improved versions of the .Rmd file in there. Never write into /srv/gstore/ directly.

Step 1 — Build the folder

Create the folder

Step 2 - Improve the .Rmd script

Work on the .Rmd file and improve it when needed.

Writing the parameters is is done via:

params <- list(
  analysis_name    = analysis_name,
  analysis_version = analysis_version
)

paramFrame <- ezFrame(Value = sapply(params, as.character))
ezWrite.table(paramFrame, file = "parameters.tsv", col.names = FALSE)

Keep this honest — add every parameter that materially affected the result (gene annotation version, thresholds, reference paths). Future-you will thank present-you.

Guardrails

• Confirm project + order_id aloud before scaffolding. Type them back. Cheap to do, expensive to get wrong.
• Execute the last chunk dataset_registration only after all editing is done that will create the scripts folder with the timestamp that serves identifies the version
• Always generate a fresh timestamp — never reuse one. Each run gets its own immutable folder.
• Always supply the web URL after g-req succeeds. CLAUDE.md mandates this.
• Don't write into /srv/gstore/ directly.
• Avoid using absolute paths parametrize the root folders instead

See also

• references/example-self-contained.Rmd — the initial version that exemplifies the general pattern.
• fgcz-sushi-app-dev — for the real SUSHI app build path (when this analysis becomes routine and deserves to be a button in the SUSHI UI).
• autonomous-render — for self-correcting Rmd rendering with retry logic.
• bfabric — for direct B-Fabric reads/writes once Step 9 lands.

Follow-ups (deliberately out of scope)

• Merge register_custom_analysis.py + src/ updates into main btools so users don't have to copy from /home/rdomi/.
• Build a production-SUSHI Python API (mirror of the dev one on fgcz-h-083:4071) so Step 10 can become an HTTP call instead of a raw MySQL insert. Until then, Step 10 is the canonical path.
• Add SBATCH wrapping for long renders. The synchronous bash launcher OOMs on heavy CellChat / multi-GB Seurat workloads.
• Add evals/evals.json with 2–3 test prompts once the workflow has stabilised.