iowarp-dev-setup

CLIO Core

A Comprehensive Platform for Context Management in Scientific Computing

Overview · Installation · Components · Quickstart · Documentation · Contributing

Overview

CLIO Core is a unified framework that integrates multiple high-performance components for context management, data transfer, and scientific computing. CLIO Core enables developers to build efficient data processing pipelines for HPC, storage systems, and near-data computing applications.

It provides:

• High-performance context management for computational contexts and data transformations.
• Heterogeneous-aware I/O with multi-tiered, dynamic buffering.
• A modular runtime with dynamically loadable processing modules.
• Shared-memory data structures that work across host, CUDA, and ROCm.
• Distributed-by-construction scaling from single node to clusters.

Architecture

┌──────────────────────────────────────────────────────────────┐
│                      Applications                            │
│          (Scientific Workflows, HPC, Storage Systems)        │
└──────────────────────────────────────────────────────────────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
┌───────────────┐   ┌──────────────────┐   ┌────────────────┐
│   Context     │   │    Context       │   │   Context      │
│  Exploration  │   │  Assimilation    │   │   Transfer     │
│    Engine     │   │     Engine       │   │    Engine      │
└───────────────┘   └──────────────────┘   └────────────────┘
        │                     │                     │
        └─────────────────────┼─────────────────────┘
                              │
                    ┌─────────────────┐
                    │  CLIO Runtime   │
                    │  (Module System)│
                    └─────────────────┘
                              │
                ┌─────────────────────────┐
                │  Context Transport      │
                │  Primitives             │
                │  (Shared Memory & IPC)  │
                └─────────────────────────┘

Installation

Pip (recommended)

The pip wheel ships a portable, self-contained build with all dependencies statically linked. No system installs are required beyond glibc and Python 3.10+.

pip install iowarp-core

The wheel includes the CLIO runtime, the clio_run CLI, the CTE, CAE, and CEE engines, and the clio_cee Python bindings. A default configuration is seeded at ~/.clio/clio.yaml on first import.

Switch to a source build below if you need any of:

• NVIDIA GPU (CUDA) or AMD GPU (ROCm) acceleration
• MPI for distributed multi-node deployment
• HDF5 / ADIOS2 adapters
• FUSE adapter
• Compression backends (LibPressio, Blosc, etc.)
• Custom ChiMods built against the C++ headers
• Sanitizer or debug builds

Source build (Conda)

git clone --recurse-submodules https://github.com/iowarp/clio-core.git
cd clio-core
bash install.sh release

release corresponds to a variant under installers/conda/variants/. Other variants (cuda, rocm, mpi, full, release-fuse, debug, ...) enable the corresponding features.

For a full bare-metal source build (without Conda) with the per-feature apt / dnf dependency list and the complete CLIO_*_ENABLE_* flag checklist, see INSTALL.md. Other methods (Docker, Spack) are documented in docs/getting-started/installation.

Components