palantir-reference-architecture

Palantir Reference Architecture

Overview

Production-ready architecture for Foundry-integrated applications. Covers the standard data pipeline pattern (ingest > clean > model > serve), Ontology design, external API integration, and multi-repo project layout.

Prerequisites

• Foundry enrollment with project access
• Understanding of Ontology concepts (object types, link types, actions)
• Familiarity with palantir-core-workflow-a (transforms) and palantir-core-workflow-b (Ontology)

Instructions

Step 1: Data Pipeline Architecture

┌─────────────┐     ┌──────────────┐     ┌─────────────┐     ┌───────────┐
│  Raw Layer   │────>│  Clean Layer │────>│ Model Layer │────>│ Ontology  │
│ (ingested)   │     │  (validated) │     │ (enriched)  │     │ (objects) │
└─────────────┘     └──────────────┘     └─────────────┘     └───────────┘
  ↑ Connectors        @transform_df       @transform_df       Object types
  ↑ REST sync          null checks         joins, aggs         Link types
  ↑ File upload        type casting        ML features         Actions

Step 2: Project Layout (Foundry)

Foundry Project: "Customer Analytics"
├── Datasets/
│   ├── raw/                    # Ingested from sources
│   │   ├── raw_orders          # REST connector → CRM
│   │   ├── raw_customers       # JDBC connector → DB
│   │   └── raw_products        # File upload (CSV/Parquet)
│   ├── clean/                  # Validated, typed
│   │   ├── clean_orders        # Nulls removed, dates parsed
│   │   ├── clean_customers     # Deduped, normalized
│   │   └── clean_products      # Schema enforced
│   └── model/                  # Enriched, analytics-ready
│       ├── order_enriched      # Joined with customer + product
│       ├── customer_360        # Aggregated customer view
│       └── daily_summary       # Time-series aggregation
├── Code Repositories/
│   ├── pipeline-ingestion/     # Connectors and raw → clean
│   ├── pipeline-analytics/     # Clean → model transforms
│   └── ontology-actions/       # Action implementations
└── Ontology/
    ├── Object Types: Customer, Order, Product
    ├── Link Types: Customer→Orders, Order→Products
    └── Actions: createOrder, updateCustomerSegment

Step 3: External API Integration Pattern

# External app consuming Foundry Ontology via Platform SDK
my-external-app/
├── src/
│   ├── foundry/
│   │   ├── client.py           # Singleton FoundryClient
│   │   ├── objects.py          # Object query helpers
│   │   ├── actions.py          # Action wrappers
│   │   └── cache.py            # TTL cache layer
│   ├── api/
│   │   ├── routes.py           # REST endpoints
│   │   └── webhooks.py         # Foundry event handlers
│   └── main.py
├── tests/
│   ├── conftest.py             # Mocked FoundryClient
│   ├── test_objects.py
│   └── test_actions.py
├── .env                        # FOUNDRY_HOSTNAME, credentials
└── requirements.txt

Step 4: Ontology Design Patterns

Pattern	When to Use	Example
Hub-and-spoke	Central entity with many relationships	Customer → Orders, Tickets, Payments
Event sourcing	Audit trail needed	OrderEvent (created, shipped, delivered)
Computed properties	Derived values	totalRevenue on Customer (sum of orders)
Composite actions	Multi-step mutations	processReturn: update order + create credit + notify

Step 5: Security Layers

┌──────────────────────────────────────────┐
│ Layer 1: Network (VPN/private link)       │
├──────────────────────────────────────────┤
│ Layer 2: OAuth2 (service user per app)    │
├──────────────────────────────────────────┤
│ Layer 3: Scopes (minimum per app)         │
├──────────────────────────────────────────┤
│ Layer 4: Project roles (Viewer/Editor)    │
├──────────────────────────────────────────┤
│ Layer 5: Marking (data classification)    │
└──────────────────────────────────────────┘

Output

• Standard 3-layer data pipeline (raw > clean > model)
• Ontology design with typed objects, links, and actions
• External app architecture with caching and webhooks
• Security model with 5 defense layers

Error Handling

Architecture Issue	Symptom	Fix
Circular dependencies	Builds fail	Restructure pipeline DAG
Missing clean layer	Bad data in model	Always validate between raw and model
Monolithic transforms	Slow builds	Split into focused transforms
No caching	API rate limits	Add TTL cache layer

Resources

• Foundry Documentation
• Ontology SDK Overview
• Transforms Guide

Next Steps

For data handling and compliance, see palantir-data-handling.