# oryxflow

> oryxflow is an open-source Python library and Claude Code plugin for faster, cheaper, more reliable AI data analysis: it makes your data-science pipeline reproducible and lineage-tracked, caching results and rerunning exactly what a parameter, data, or code change affects — local-first, no server or account.

oryxflow is a small, self-contained Python library for building data-science workflows: declare Task classes with parameters, dependencies (requires) and a run() that save()s output; the engine runs the DAG in dependency order, skips tasks whose output already exists, and reruns exactly what a parameter, data, or code change affects.

# Getting started

# oryxflow

**Faster, cheaper, and more trustworthy data analysis — for humans and AI coding agents.** oryxflow turns a data-science script into a pipeline that caches every step, reruns exactly what a change affects, and records how each result was made. It's a Python library with no server, no database, and no account: `pip install oryxflow` and you're done.

Working with an AI agent? The **[Claude Code plugin](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md)** teaches Claude Code to build your data analysis this way — so the agent reuses expensive results instead of burning your time and tokens redoing them, and never trains a model on stale data. You get analysis you can **trust**, not just analysis that runs. Using a different agent? The docs are [machine-readable too](https://docs.oryxflow.dev/docs/ai-ready/index.md) — point it at [`llms-full.txt`](https://docs.oryxflow.dev/llms-full.txt) and it has read all of oryxflow in one request.

## Why oryxflow

Four things you get on day one — the full argument is in **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)**:

- **You always get the right result.** Change a parameter, the data, or a task's code and oryxflow reruns exactly what that change affects. You can't accidentally evaluate a new model on stale features.
- **No file mess, no parameter bookkeeping.** You never name, path, or version an intermediate file again — no `features_v3_final.pkl`, no `to_pickle`/`read_pickle` plumbing, no spreadsheet of which run used which settings. Ask for a result by the task and parameters that made it.
- **Seconds instead of minutes.** Finished steps load from cache, so the 10-minute data pull runs once, not once per edit.
- **An answer to "is this stale?"** oryxflow records what ran, when, with which code and inputs, and why it recomputed — so staleness and provenance are queries, not guesses.

**Start small; it scales with you.** A first pass can be a plain script or a quick exploratory probe — you don't need a task graph on day one. As the work gains steps, cost, and parameter combinations (as it always does), the plugin's `/oryxflow:migrate` command lifts what you already wrote into cached tasks. No rewrite, and no cliff between "exploring" and "pipeline" — [migrate a messy project](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md).

## oryxflow in brief

You declare each step of your analysis as a **task**: what it depends on and what it produces. The engine runs them in dependency order, skips anything already computed, and hands you any result by name.

```
import oryxflow
import pandas as pd

class GetData(oryxflow.tasks.TaskPqPandas):        # output saved as parquet
    def run(self):
        self.save(pd.DataFrame({'x': range(10)}))

@oryxflow.requires(GetData)                        # declare the dependency
class ProcessData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                      # GetData's output, already loaded
        df['x2'] = df['x'] ** 2
        self.save(df)

flow = oryxflow.Workflow(ProcessData)
flow.run()                                         # runs GetData, then ProcessData
df = flow.outputLoad()                             # load the result by name
```

Run `flow.run()` again and nothing happens — both outputs already exist, so the engine skips them. That is the core payoff: re-running a pipeline only pays for what actually changed. Caching is how it works; **trust is what you get**. Next step: **[Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)** — a real pipeline in a few minutes.

## How oryxflow compares to Airflow, MLflow, DVC, and notebooks

oryxflow doesn't replace an experiment tracker or a production orchestrator — it fills the gap between an ad-hoc script and a heavyweight platform, and composes with both. What's distinctive is the combination of local-first simplicity, invalidation that notices a **code** change, and always-on lineage.

|                             | Local, zero-infra | Automatic caching & reruns | Reruns on a **code** change | Queryable lineage | Experiment dashboard | Production scheduling   |
| --------------------------- | ----------------- | -------------------------- | --------------------------- | ----------------- | -------------------- | ----------------------- |
| **oryxflow**                | ✅                | ✅                         | ✅ automatic                | ✅                | — (use a tracker)    | — (use an orchestrator) |
| Notebooks + pickle files    | ✅                | ❌ hand-rolled             | ❌                          | ❌                | ❌                   | ❌                      |
| MLflow / W&B                | partial           | ❌ (tracks, doesn't rerun) | ❌                          | logs runs         | ✅                   | ❌                      |
| Airflow / Prefect / Dagster | ❌ server/infra   | opt-in / configured        | ❌                          | run history       | partial              | ✅                      |
| DVC                         | ✅                | ✅ (file-hash stages)      | on declared file deps       | via Git           | ❌                   | ❌                      |

The per-tool detail is in [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/#how-oryxflow-compares-to-mlflow-airflow-and-dvc), and the head-to-heads are on the blog: [vs Airflow](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md), [vs MLflow](https://docs.oryxflow.dev/blog/comparisons/mlflow-alternatives/index.md), [vs DVC](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-dvc/index.md), [vs Dagster](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-dagster/index.md), [vs Prefect](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-prefect/index.md), and [the whole field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md).

## Where to next

- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)**

  The argument in full: trustworthy AI data analysis, no file mess, honest tool comparisons — and how it scales from a first EDA script upward.

- **[Installation](https://docs.oryxflow.dev/docs/installation/index.md)**

  Install oryxflow and its optional extras (cloud storage, export, dask).

- **[Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)**

  From nothing to a running, self-caching pipeline in a few minutes.

- **[Documentation](https://docs.oryxflow.dev/docs/index.md)**

  The full guide: tasks, workflows, parameters, I/O formats, and logging.

- **[Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md)**

  The official plugin makes AI-written data analysis trustworthy — it scaffolds the project, wires the DAG, and teaches the agent to use the cache correctly.

- **[Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md)**

  Automatic code invalidation, selective resets, and multi-experiment flows.

- **[Migrate a messy notebook project](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md)**

  Nine notebooks and a folder of `clean_v3.csv`? Restructure it so a wrong number stops being possible — by hand, or in one command.

- **[Blog](https://docs.oryxflow.dev/blog/index.md)**

  Reproducibility and trust, tool comparisons (vs Airflow, MLflow, DVC), and trustworthy AI-assisted data science.

## Frequently asked questions

**How do I stop rerunning my whole pipeline every time I change one step?** oryxflow caches each step's output and reruns only what a code, data, or parameter change affects, plus everything downstream of it. Change one feature and the untouched upstream steps load instantly from cache instead of recomputing. It's a local-first Python library: `pip install oryxflow`, declare each step as a task, and re-running only pays for what actually changed.

**How do I cache intermediate DataFrames in Python without brittle pickle files?** Declare each step as an oryxflow task that saves its DataFrame, and the engine caches it, addresses it by task identity instead of a hand-managed filename, and reloads it on the next run. You never wire up `to_pickle` / `read_pickle` paths or track which file is current — you ask for a result by the task that made it, and stale outputs rerun automatically when the code changes.

**Is there a lightweight alternative to Airflow or MLflow for a local data science project?** oryxflow is a local-first Python workflow library that sits between notebooks and heavyweight orchestrators — no server, scheduler, database, or account. Where Airflow orchestrates production DAGs and MLflow tracks experiments, oryxflow makes one analyst's pipeline reproducible and cached: it reruns only what changed and records what produced each result. Reach for it when a notebook has outgrown itself but Airflow or MLflow would be overkill.

**How do I run a parameter sweep without rerunning the upstream steps every time?** Parameters flow through the task graph, so oryxflow reruns only the tasks a given parameter actually changes and reuses the shared upstream cache across every combination in the sweep. Compare ten model configs and the data-loading and feature steps run once, not ten times. Each run is tagged by its parameters, so results stay reproducible and you can load any combination's output by name.

**Is there a Claude Code plugin to make AI-generated data analysis reproducible and trustworthy?** Yes — the oryxflow Claude Code plugin. It teaches your coding agent to build the analysis as a cached, reproducible pipeline: reusing expensive results, verifying its own reruns, and never training on stale intermediates. oryxflow guarantees a result was produced by the code and inputs it recorded — reproducible, not automatically *correct* — so you can check AI-written analysis instead of trusting it blindly. It ships as a skill plus slash commands, not an MCP server.

**When should I not use oryxflow?** You don't need a task graph for a first look at a dataset — a quick CSV load, a group-by, one plot is fine as a plain script. You don't have to choose upfront, though: start there and run `/oryxflow:migrate` when the work gains depth, cost, or parameter combinations, and what you already wrote becomes cached tasks. oryxflow earns its keep the moment a stale early step can silently corrupt everything below it, an expensive step makes the edit-run loop painful, or you're sweeping an experiment matrix — which is also where hand-managed scripts and AI coding agents go wrong. It isn't a production orchestrator (use Airflow or Prefect) or an experiment dashboard (use MLflow or Weights & Biases); it composes beside both.

## Learn more

- **Scaffold a real project:** [`/oryxflow:init-project`](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) in Claude Code
- **Why this matters:** [4 reasons your machine learning code is probably bad](https://docs.oryxflow.dev/blog/machine-learning/why-machine-learning-code-is-bad/index.md)
- **API details:** [API Reference](https://docs.oryxflow.dev/docs/reference/index.md)

# Documentation

Everything you need to build data-science pipelines with oryxflow. Declare each step as a **task**. The engine runs your steps in dependency order, skips what's already computed, and reruns exactly what a parameter, data, or code change affects — and you load any result by name. You get faster, cheaper, more **trustworthy** data analysis, reproducible and lineage-tracked by default, for humans and AI coding agents alike — with no filenames to invent and no note about which parameters produced which output.

New here? Read **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** for the positioning, then start with **[Installation](https://docs.oryxflow.dev/docs/installation/index.md)** and the **[Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)**. Already have a project that got out of hand? Go straight to **[Migrate a messy notebook project](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md)**.

- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)**

  What it's for and when *not* to use it — reproducibility, lineage, and trustworthy AI data analysis, plus honest comparisons.

- **[Installation](https://docs.oryxflow.dev/docs/installation/index.md)**

  Install oryxflow and its optional extras (cloud storage, export, dask).

- **[Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)**

  From nothing to a running, self-caching pipeline in a few minutes.

- **[Transition from scripts](https://docs.oryxflow.dev/docs/transition/index.md)**

  Turn an existing analysis script into cached tasks.

- **[Migrate a messy notebook project](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md)**

  Nine notebooks and a folder of `clean_v3.csv`? Restructure it so a wrong number stops being possible — by hand or in one command.

- **[Writing & managing tasks](https://docs.oryxflow.dev/docs/tasks/index.md)**

  Dependencies, inputs, outputs, and save formats.

- **[Workflows](https://docs.oryxflow.dev/docs/workflow/index.md)** & **[Running workflows](https://docs.oryxflow.dev/docs/run/index.md)**

  Wrap tasks in a flow; preview, run, and reset.

- **[Parameters](https://docs.oryxflow.dev/docs/advparam/index.md)**

  Parameter inheritance and how it drives selective reruns.

- **[Task I/O formats](https://docs.oryxflow.dev/docs/targets/index.md)**

  Parquet, pickle, CSV, in-memory cache, and cloud storage.

- **[Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md)**

  Automatic code invalidation, selective resets, multi-experiment flows.

- **[Experiment tracking](https://docs.oryxflow.dev/docs/experiment-tracking/index.md)**

  How oryxflow pairs with MLflow or Weights & Biases — different halves of the same project.

- **[Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md)**

  Make AI-written data analysis trustworthy: scaffold the project, wire the DAG, and teach the agent to use the cache correctly.

- **[Built for AI coding agents](https://docs.oryxflow.dev/docs/ai-ready/index.md)**

  `llms.txt` for one-request ingestion, core examples executed by the test suite, and a reference generated from docstrings that cover the whole public API.

- **[API Reference](https://docs.oryxflow.dev/docs/reference/index.md)**

  Every public symbol, generated from the source docstrings.

# Why oryxflow

**oryxflow makes data-science work faster, cheaper, and more trustworthy** — it turns an analysis script into a reproducible pipeline that records how every result was made and reruns only what actually changed. It's a pip-installable Python library with no server, no database, and no account: your code, your cache, your repo.

If you only remember one thing: oryxflow is the layer that makes an iterative analysis **trustworthy** — for you, your teammates, and the AI coding agent writing half the code.

## The problem: iterative analysis quietly stops being trustworthy

Almost every project starts as a script that works. Then it accumulates the failures that erode trust in the result long before anyone questions the math:

- **Stale intermediates.** You change a feature, forget to regenerate a cached file, and train on yesterday's data. Nothing errors. The number is just wrong.
- **A folder full of files you have to keep straight.** `features_v3.pkl`, `features_v3_final.pkl`, `features_v3_final_FIXED.pkl` — plus the mental note about which settings each one used, which nobody wrote down.
- **Lost lineage.** Six months (or six hours) later, no one can say which code and which inputs produced `model_final_v3.pkl`.
- **Wasted recomputation.** A one-line change downstream re-runs the 10-minute data pull, so you either wait or start hand-rolling `if os.path.exists(...)` caches that themselves go stale.
- **AI-generated code you can't fully trust.** Coding agents write plausible pandas and scikit-learn fast — but across a long session they lose track of what's already computed and whether it's still valid, and silently build on stale state.

None of these are math errors. They're **trust** errors — in the mechanics of the pipeline. And they get worse, not better, as an AI agent writes more of the code.

## What oryxflow gives you

- **No storage or parameter boilerplate.** You never name a file, build a path, or track which settings produced which output. `self.save(df)` puts it away, `flow.outputLoad()` gets it back, and oryxflow works out where it lives from the task and its parameters.
- **Reproducibility by default.** Every output is tied to the exact task, parameters, and code version that produced it. "Can I reproduce last week's result?" becomes yes, mechanically.
- **Lineage you can query.** oryxflow records what ran, when, with which parameters and code, and *why* it recomputed. "Is this stale? Was it built with current code?" are queries, not guesses.
- **Reruns exactly what changed.** Change a parameter, a data input, or a task's code and exactly the affected outputs rebuild — you can't accidentally evaluate a new model on old features.
- **Speed and cost savings.** Completed steps load from cache instead of recomputing, so the edit–run loop drops from minutes to seconds. An AI agent stops paying — in time and tokens — to redo expensive work it already did.
- **AI-agent reliability.** The same cache and lineage log become an agent's memory across sessions. The companion [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) ships these disciplines as an auto-activating skill, so the agent uses the cache correctly instead of trusting stale state.

Caching is the *engine*. Trust — reproducible, lineage-tracked reruns that update exactly what changed — is the *product*.

## No file paths, no parameter bookkeeping

The most immediate change is how much clerical work disappears. Storage and parameter tracking are the same job, and oryxflow does both for you.

Before, every intermediate is a filename you invent and a decision you have to remember:

```
# was the 60-day window the one that produced this? and is it still current?
features = build_features(df, window=60)
features.to_pickle('data/features_w60_v3_final.pkl')
...
features = pd.read_pickle('data/features_w60_v3_final.pkl')   # ...probably the right file
```

After, the parameter *is* the identity of the result, and the file is oryxflow's problem:

```
class Features(oryxflow.tasks.TaskPqPandas):
    window = oryxflow.IntParameter(default=60)

    def run(self):
        self.save(build_features(self.inputLoad(), window=self.window))
```

That one change buys you three things:

- **You always get the right result.** Ask for `window=60` and you get the output built with `window=60` and the current code — not whichever file was written last. There is no way to point at the wrong pickle, because you never point at a pickle.
- **No file mess.** No `_v3_final_FIXED` suffixes, no path constants threaded through modules, no cleanup of an intermediates folder nobody understands. Change a parameter and you get a *new* cached result rather than an overwritten one, so a ten-way sweep needs zero filing.
- **Nothing to keep in your head — or the agent's.** "Which settings made this?" is recorded, not remembered. That matters most when an AI coding agent is writing the code, since it's exactly the state an agent silently loses over a long session.

## When to use oryxflow — and how to start small

Reach for it when the work has a **shape worth keeping** — it will be rerun, depended on, or swept over parameters:

- Feature-engineering pipelines with expensive intermediate steps.
- Model training and evaluation you iterate on repeatedly.
- Parameter sweeps and experiment matrices (model × features × window).
- Research code that must be reproduced, compared, and handed off.
- Any of the above written with an AI coding agent, where mechanical trust matters most.

**But you don't have to know that on day one, and you don't have to start there.** Nobody begins a project by declaring a DAG — you begin by looking at a dataset. So start with plain exploratory scripts: the [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) gives that stage a home (read-only probes under `eda/`, each documenting the question it answers and writing findings into the project's data doc), and when a probe turns out to be load-bearing, `/oryxflow:migrate` lifts what you already wrote into cached tasks. Simple scripts at the start, any complexity later, with no rewrite and no cliff in between — which matters because analyses only ever get more complicated.

## What oryxflow is *not*

Being honest about fit is part of being trustworthy. Two jobs are somebody else's, and in both cases oryxflow is designed to sit *beside* the other tool, not argue with it:

- **Production orchestration.** If you need cron-style scheduling, retries across a cluster, and SLAs, use [Airflow, Prefect, or Dagster](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md). oryxflow is built for the research loop, not production ops.
- **Experiment dashboards.** If you want a searchable UI charting every run's metrics, that's an experiment tracker's job (MLflow, Weights & Biases). **Experiment tracking and oryxflow are complementary, and you should expect to use both** — see below.

## How oryxflow compares to MLflow, Airflow, and DVC

oryxflow doesn't replace trackers or orchestrators; it fills the gap between an ad-hoc script and a heavyweight platform. What's distinctive is the **combination** of local-first simplicity, automatic *code-aware* invalidation, and always-on lineage.

|                             | Local, zero-infra | Automatic caching & reruns | Reruns on a **code** change | Queryable lineage | Experiment dashboard | Production scheduling   |
| --------------------------- | ----------------- | -------------------------- | --------------------------- | ----------------- | -------------------- | ----------------------- |
| **oryxflow**                | ✅                | ✅                         | ✅ automatic                | ✅                | — (use a tracker)    | — (use an orchestrator) |
| Notebooks + pickle files    | ✅                | ❌ hand-rolled             | ❌                          | ❌                | ❌                   | ❌                      |
| MLflow / W&B                | partial           | ❌ (tracks, doesn't rerun) | ❌                          | logs runs         | ✅                   | ❌                      |
| Airflow / Prefect / Dagster | ❌ server/infra   | opt-in / configured        | ❌                          | run history       | partial              | ✅                      |
| DVC                         | ✅                | ✅ (file-hash stages)      | on declared file deps       | via Git           | ❌                   | ❌                      |

A few honest specifics:

- **vs notebooks + pickle files** — oryxflow gives you the caching, dependency order, and reproducibility you were hand-rolling, without the stale-`.pkl` graveyard.
- **vs MLflow / W&B** — complementary, not competing. Trackers answer "which run scored 0.91?"; oryxflow answers "which steps do I actually need to rerun to reproduce it, and are they stale?" Keep logging to your tracker *inside* oryxflow tasks. See [MLflow or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md)
- **vs Airflow / Prefect / Dagster** — a different job. Those run scheduled production pipelines on real infrastructure; oryxflow is a `pip install` for the local research loop. See [oryxflow vs Airflow](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md).
- **vs DVC** — both cache pipelines. DVC hashes files and YAML-declared stages; oryxflow keeps identity in native Python — a parameter change is a new cached identity automatically, and a code edit reruns the affected tasks on its own, no config files to maintain.

### Experiment tracking is complementary — expect to use both

This is the comparison people most often read as either/or, so to be explicit: **an experiment tracker and oryxflow do different halves of the same project, and using both is the normal setup.**

A tracker is a *record of results*: it collects metrics, params, and artifacts from runs you already did, and gives you a UI to sort and chart them. oryxflow is the *machinery that produces those runs*: it decides which steps have to execute, reuses the ones that don't, and guarantees the features your model just scored on were built by current code. A tracker can't tell you a logged run was trained on a stale intermediate; oryxflow can't draw you a leaderboard. In practice the tracker call lives *inside* an oryxflow task, so every logged run is also a cached, reproducible one.

Full treatment, with the integration pattern and when one tool alone is enough: **[Experiment tracking with oryxflow](https://docs.oryxflow.dev/docs/experiment-tracking/index.md)**.

## What it looks like

```
import oryxflow
import pandas as pd

class GetData(oryxflow.tasks.TaskPqPandas):        # output saved as parquet — no file paths
    def run(self):
        self.save(pd.DataFrame({'x': range(10)}))

@oryxflow.requires(GetData)                         # declare the dependency
class ProcessData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                       # GetData's output, already loaded
        df['x2'] = df['x'] ** 2
        self.save(df)

flow = oryxflow.Workflow(ProcessData)
flow.run()                                          # runs GetData, then ProcessData
df = flow.outputLoad()                              # load the result by name
```

Run `flow.run()` again and nothing recomputes — both outputs already exist. Edit `ProcessData`'s code and only it (and anything downstream) reruns, automatically. The record of what ran and why is written to a lineage log you can query later.

## Frequently asked questions

**Is there a lightweight alternative to Airflow for a data science project?** Yes — oryxflow is a local-first Python library built for the research loop rather than production ops: `pip install oryxflow`, no server, scheduler, database, or account. Airflow's job is running scheduled pipelines on real infrastructure with retries and alerting; oryxflow's job is making one analyst's pipeline fast and trustworthy to iterate on, caching each step and rerunning exactly what a code, data, or parameter change affects. If you're reaching for Airflow only to get dependency order and caching on your laptop, oryxflow is the smaller tool that does that part.

**Is oryxflow an MLflow alternative?** Not a replacement — a complement, and they answer different questions. MLflow tracks and charts experiment runs, so it tells you which run scored 0.91; oryxflow decides what actually has to rerun to reproduce that run and whether its inputs are stale. Keep logging metrics to MLflow or Weights & Biases from inside your oryxflow tasks. If what you wanted from a tracker was really caching and reproducible reruns rather than a dashboard, oryxflow covers that on its own with no server or account.

**How do I make my data science workflow reproducible?** Declare each step of your analysis as an oryxflow task that saves its output, and the pipeline becomes reproducible by construction: every result is addressed by the code and parameters that produced it, each run records what ran and why, and re-running regenerates any result from the recorded inputs. Reproducibility stops depending on remembering which cell you ran in what order.

**How does oryxflow know when to rerun a task?** From the task's parameters and its code. Change a parameter, a data input, or the code and oryxflow reruns exactly the affected outputs — cosmetic edits like comments or formatting don't trigger a rerun.

**Do I have to manage file names and paths for cached results?** No — that bookkeeping is the thing oryxflow takes over. A task saves its output with `self.save(df)` and you read it back with `flow.outputLoad()`; oryxflow derives the storage location from the task and its parameters, so there is no `features_v3_final.pkl` to name, no path constants to thread through your code, and no note about which run used which settings. Change a parameter and you get a separate cached result automatically instead of overwriting the old one, which is why you can sweep an experiment matrix without curating a folder of files by hand.

**Can I use oryxflow for exploratory data analysis?** Yes, and that's the recommended way in: start with plain exploratory scripts, then let the work grow into a pipeline. The Claude Code plugin gives exploration a home — read-only probes under `eda/`, with findings written up as you go — and when a probe turns out to be load-bearing, `/oryxflow:migrate` lifts what you already wrote into cached tasks. You get a first look at a dataset without ceremony and a reproducible pipeline once the analysis earns one, with no rewrite in between.

## Takeaway

- oryxflow makes iterative data analysis **trustworthy**: the right steps rebuild automatically, so you always get the result your current code and parameters imply — for humans and AI agents.
- **No storage or parameter boilerplate.** No filenames to invent, no paths to thread through, no record of which settings produced which output.
- It's **local-first and zero-infrastructure**: `pip install oryxflow`, no server or account.
- **Start small.** A first EDA script is fine; `/oryxflow:migrate` grows it into a pipeline when the work earns one.
- It **composes** with the tools you already use — trackers for dashboards, orchestrators for production.

Ready to build?

```
pip install oryxflow
```

- **[Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)** — nothing to a running, self-caching pipeline in minutes.
- **[Transition from scripts](https://docs.oryxflow.dev/docs/transition/index.md)** — convert an existing analysis.
- **[Migrate a messy notebook project](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md)** — take a project that's already out of control and make it scalable.
- **[Experiment tracking with oryxflow](https://docs.oryxflow.dev/docs/experiment-tracking/index.md)** — how it pairs with MLflow or Weights & Biases.
- **[Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md)** — let an AI agent scaffold and wire it, the trustworthy way.

# Installation

Install the latest release from PyPI:

```
pip install oryxflow
```

To upgrade later:

```
pip install oryxflow -U
```

Note

oryxflow is **Python 3 only**. If Python 3 is not your default, use `pip3 install oryxflow`. Behind an enterprise firewall you can clone the repository and run `pip install .` from its root instead.

## Development version

To install the latest unreleased code straight from GitHub:

```
pip install git+https://github.com/oryxintel/oryxflow.git
```

Or upgrade in place without touching dependencies:

```
pip install git+https://github.com/oryxintel/oryxflow.git -U --no-deps
```

## Optional extras

The core install keeps dependencies light. Enable the pieces you need with pip *extras*:

| Extra                | Install                              | Adds                                                             |
| -------------------- | ------------------------------------ | ---------------------------------------------------------------- |
| Cloud storage (GCS)  | `pip install "oryxflow[gcs]"`        | Read/write task outputs on Google Cloud Storage                  |
| Cloud storage (S3)   | `pip install "oryxflow[s3]"`         | Read/write task outputs on Amazon S3                             |
| Cloud storage (base) | `pip install "oryxflow[cloud-base]"` | fsspec/`universal_pathlib` backend without a specific provider   |
| Flow export          | `pip install "oryxflow[export]"`     | `FlowExport` / `FlowImport` standalone task files (needs Jinja2) |
| Dask                 | `pip install "oryxflow[dask]"`       | Dask DataFrame task types                                        |

After installing a cloud extra, point the engine at remote storage with `oryxflow.enable_gcs(bucket, prefix=None)` or the generic `oryxflow.enable_cloud_storage(protocol, bucket, prefix=None)`. See [Collaborate & share flows](https://docs.oryxflow.dev/docs/collaborate/index.md) for the full workflow.

## Build with an AI assistant

If you use [Claude Code](https://claude.com/claude-code), the official plugin can scaffold your project and wire task dependencies for you:

```
/plugin install oryxflow@oryxflow
```

See [Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md) for what it sets up.

## Next steps

- [Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md) — build your first self-caching pipeline.
- [Transition from scripts](https://docs.oryxflow.dev/docs/transition/index.md) — convert an existing analysis script.

# Quickstart

oryxflow turns a data-science script into a pipeline of **tasks**. You declare each step as a task: what it *depends on* and what it *produces*. The engine runs them in the right order, skips anything already computed, and lets you load any result by name. No manual file paths, no re-running the slow steps to test a fast one.

This page gets you from nothing to a running pipeline. For installation, follow the [GitHub instructions](https://github.com/oryxintel/oryxflow#installation) (`pip install oryxflow`).

## The idea in one minute

A **task** is one step of your analysis, written as a class:

- it inherits from a task type that decides the output format (parquet, pickle, in-memory, …), so you never write save/load code;
- `run()` does the work and calls `self.save(...)` to store the result;
- `@oryxflow.requires(...)` declares which other task(s) it depends on;
- inside `run()`, `self.inputLoad()` hands you the dependency's output, already loaded.

You then wrap the final task in a `Workflow` and call `run()`. The engine walks the dependencies, runs only what's missing, and caches every output.

## Your first pipeline

Two tasks: one produces data, the next transforms it.

```
import oryxflow
import pandas as pd

class GetData(oryxflow.tasks.TaskPqPandas):     # output saved as parquet
    def run(self):
        df = pd.DataFrame({'x': range(10)})
        self.save(df)                           # cache this task's output

@oryxflow.requires(GetData)                     # declare the dependency on GetData
class ProcessData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                   # GetData's output, already loaded
        df['x2'] = df['x'] ** 2
        self.save(df)
```

Run it by asking for the *final* task — upstream dependencies run automatically:

```
flow = oryxflow.Workflow(ProcessData)

flow.preview()          # show what will run, without running it
flow.run()              # runs GetData, then ProcessData

df = flow.outputLoad()  # load ProcessData's result by referencing the flow
print(df.head())
```

Run `flow.run()` **again** and nothing happens — both outputs already exist, so the engine skips them. That is the core payoff: re-running a pipeline only pays for what actually changed.

Nothing to configure first: outputs go to `data/`, one file per task and parameter set, named for you. To keep them somewhere else, call `oryxflow.set_dir('other/')` before you run.

## Change a parameter, rerun only what depends on it

Parameters are how you try different settings without renaming files by hand. Give a task a parameter and oryxflow caches a **separate output per value**, so you can switch between them instantly, and changing one reruns exactly the tasks that depend on it.

```
@oryxflow.requires(GetData)
class ProcessData(oryxflow.tasks.TaskPqPandas):
    power = oryxflow.IntParameter(default=2)    # a knob to experiment with
    def run(self):
        df = self.inputLoad()
        df['x_pow'] = df['x'] ** self.power
        self.save(df)

# run with a non-default parameter
flow = oryxflow.Workflow(ProcessData, {'power': 3})
flow.run()              # GetData is already complete and is skipped; only ProcessData runs
df = flow.outputLoad()
```

`GetData` doesn't re-run — it has no dependence on `power`, so its cached output is reused. Only `ProcessData` recomputes. (Edit a task's *code* and it reruns automatically too; see [Automatic code invalidation](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation).)

## A realistic ML workflow

The same three ideas — depend, produce, load by name — scale to a real pipeline: get data, preprocess it, train a model, and compare two models. This needs `scikit-learn` installed.

```
import oryxflow
import pandas as pd
import sklearn.datasets, sklearn.preprocessing
import sklearn.linear_model, sklearn.ensemble

class GetDiabetes(oryxflow.tasks.TaskPqPandas):
    def run(self):
        ds = sklearn.datasets.load_diabetes()
        df = pd.DataFrame(ds.data, columns=ds.feature_names)
        df['y'] = ds.target
        self.save(df)

@oryxflow.requires(GetDiabetes)                 # inherits GetDiabetes's params, wires the dependency
class ModelData(oryxflow.tasks.TaskPqPandas):
    do_preprocess = oryxflow.BoolParameter(default=True)   # preprocessing on/off
    def run(self):
        df = self.inputLoad()
        if self.do_preprocess:
            df.iloc[:, :-1] = sklearn.preprocessing.scale(df.iloc[:, :-1])
        self.save(df)

@oryxflow.requires(ModelData)                   # parameters flow upstream automatically
class ModelTrain(oryxflow.tasks.TaskPickle):    # a model object → saved as pickle
    model = oryxflow.Parameter(default='ols')   # which model to train
    def run(self):
        df = self.inputLoad()
        X, y = df.drop(columns='y'), df['y']
        if self.model == 'ols':
            m = sklearn.linear_model.LinearRegression()
        elif self.model == 'gbm':
            m = sklearn.ensemble.GradientBoostingRegressor()
        else:
            raise ValueError('invalid model selection')
        m.fit(X, y)
        self.save(m)
        self.saveMeta({'score': m.score(X, y)})   # save a small metadata sidecar
```

Compare two models by running both as separate *flows*. `WorkflowMulti` maps a name to each flow's parameters:

```
flow = oryxflow.WorkflowMulti(ModelTrain, {
    'ols': {'do_preprocess': True,  'model': 'ols'},
    'gbm': {'do_preprocess': False, 'model': 'gbm'},
})
flow.run()      # GetDiabetes runs once and is shared — the 'gbm' flow reuses it, it doesn't refetch

print(flow.outputLoadMeta())          # scores from the metadata sidecars
# {'ols': {'score': 0.52}, 'gbm': {'score': 0.80}}

models = flow.outputLoad(ModelTrain)  # {'ols': <fitted model>, 'gbm': <fitted model>}
```

Notice that `GetDiabetes` runs only once even though two models are trained — the engine sees its output is already complete and shares it across both flows. That is the whole point: expensive, shared steps are computed once and reused.

Tip

The fastest way to a fully-structured project is the Claude Code plugin: it sets up the project layout and wires tasks for you. See [Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md).

## Next steps

- [Transition to oryxflow](https://docs.oryxflow.dev/docs/transition/index.md) — turn an existing script into tasks.
- [Writing and Managing Tasks](https://docs.oryxflow.dev/docs/tasks/index.md) — dependencies, inputs, outputs, and save formats.
- [Advanced: Parameters](https://docs.oryxflow.dev/docs/advparam/index.md) — parameter inheritance and how it drives reruns.
- [Running Workflows](https://docs.oryxflow.dev/docs/run/index.md) — previewing, running, and resetting tasks.
- [Managing Complex Workflows](https://docs.oryxflow.dev/docs/managing-workflows/#managing-complex-workflows) — caching expensive granular work and resetting it selectively as you iterate.

The ML example is in the repo as a runnable script, [example-ml-compare.py](https://github.com/oryxintel/oryxflow/blob/main/docs/example-ml-compare.py), and as a [notebook](https://github.com/oryxintel/oryxflow/blob/main/docs/example.ipynb). To scaffold a full real-life project layout — tasks, parameters, flow, config, and the supporting folders — run [`/oryxflow:init-project`](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) in Claude Code.
# Guide

# Transition to oryxflow

Most data-science code starts as a script: a chain of functions that read a file, transform it, and write the next file, wired together by hand at the bottom. It works until it doesn't — you change one step and have to remember which downstream files are now stale, you re-run the whole thing (including the slow data pull) just to test a small change, and six months later you can't tell which parameters produced which output.

oryxflow turns that script into a pipeline of **tasks** and takes over the bookkeeping. You get three things you were doing in your head before:

- **No wasted recomputation** — a task that has already produced its output is skipped, so re-running the pipeline only runs what actually changed (a small edit no longer re-pulls the raw data).
- **Reproducibility** — every output is tied to the task and parameters that produced it, so you always know how a result was made and can reproduce it exactly.
- **Automatic parameter management** — change a parameter and oryxflow reruns exactly the tasks that depend on it, and keeps the outputs for each parameter set side by side.
- **Provenance you can query** — oryxflow records what ran, when, and why, and (when you set a `code_version`) catches the classic trap of editing a task's code while the cache serves the old output. "Is this stale?" and "did I already run this?" stop being things you track in your head.

## Current Workflow Using Functions

Your code currently probably looks like the example below. How do you turn it into a oryxflow workflow?

```
import pandas as pd

def get_data():
    data = pd.read_csv('rawdata.csv')
    data = clean(data)
    data.to_pickle('data.pkl')

def preprocess(data):
    data = scale(data)
    return data

# execute workflow
get_data()
df_train = pd.read_pickle('data.pkl')
do_preprocess = True
if do_preprocess:
    df_train = preprocess(df_train)
```

## Workflow Using oryxflow Tasks

In a oryxflow workflow, you define your own task classes and then execute the workflow by running the final downstream task which will automatically run required upstream dependencies.

The function-based workflow example will transform to this:

```
import oryxflow
import pandas as pd

class TaskGetData(oryxflow.tasks.TaskPqPandas):

    # no dependency

    def run(self): # from `def get_data()`
        data = pd.read_csv('rawdata.csv')
        data = clean(data)
        self.save(data) # save output data

class TaskProcess(oryxflow.tasks.TaskPqPandas):
    do_preprocess = oryxflow.BoolParameter(default=True) # optional parameter

    def requires(self):
        return TaskGetData() # define dependency

    def run(self): 
        data = self.inputLoad() # load input data
        if self.do_preprocess:
            data = scale(data) # # from `def preprocess(data)`
        self.save(data) # save output data

flow = oryxflow.Workflow(TaskProcess)
flow.run() # execute task with dependencies
data = flow.outputLoad() # load output data
```

Learn more about [Writing and Managing Tasks](https://docs.oryxflow.dev/docs/tasks/index.md) and [Running Workflows](https://docs.oryxflow.dev/docs/run/index.md).

Tip

The Claude Code plugin automates this transition: describe your existing script in plain language and it creates the task classes and wires the `@oryxflow.requires` dependencies. See [Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md).

## Design Pattern Templates for Machine Learning Workflows

For a larger real-life project layout, run [`/oryxflow:init-project`](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) in Claude Code — it scaffolds the tasks, parameters, flow, config, and supporting folders for you. Already have a messy project rather than a blank one? See [Migrate a messy notebook project](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md).

# Migrate a messy notebook project into a pipeline

**This is the page for the project that got away from you.** Nine notebooks and four scripts that all read the same folder. `clean_v3.csv` sitting next to `clean_v3_final.csv`. A threshold typed into a cell six weeks ago that you can no longer connect to any result. And one headline number you would rather not have to defend in a meeting.

The reason to restructure it is **not** tidiness. It is that in a project shaped like that, a wrong answer is always available and nothing warns you: the cell you didn't rerun, the intermediate that never refreshed, the parameter you changed after the chart was made. Migrating to an oryxflow pipeline removes that whole class of failure — **the wrong number stops being possible, and your AI coding agent stops guessing.**

You do not rewrite the project to get there. You migrate it one task at a time, keeping a working pipeline at every step, with the original code left in place as the specification.

## What actually goes wrong

None of these show up as errors. That's the problem — they produce output, confidently.

- **The run order lives in your memory.** A notebook's correctness depends on which cells ran, in what order, against which variables still in the kernel. Reopen it in a month, hit Run All, and you get a different number with no explanation.
- **Intermediates outlive the code that made them.** `clean_v3.csv` was written by a cell you have since edited or deleted. Everything downstream still reads it happily. There is no way to ask the file which code produced it.
- **Magic constants can't be mapped to results.** A cutoff of `0.15` is in a cell somewhere, and the chart you sent last week used `0.20`. Both charts exist; only one is labeled, and neither records what it was built with.
- **A stale step answers as confidently as a fresh one.** You fix the cleaning logic, forget that the feature file was built before the fix, and the model trains on the old features. Every cell runs green. The result is simply wrong.

### An AI coding agent makes this worse, not better

Handing a messy project to a coding agent accelerates the failure rather than fixing it. The agent **cannot see which cell ran**, which intermediate is current, or which constant produced the number sitting in a notebook's saved output — so it reconstructs that state by inference, and then writes fast, plausible code on top of whatever it inferred. Being wrong is cheap and silent; being confidently wrong is the default.

So the structure that helps a human helps an agent more, because it turns invisible state into something readable: the run order becomes a decorator, the intermediates become a cache keyed by code and parameters, and "is this current?" becomes a question with an answer instead of a guess. That's the subject of [Trustworthy AI data analysis](https://docs.oryxflow.dev/docs/claude-plugin/trust/index.md).

## What you migrate toward

Each piece of the target structure removes a specific class of error — that's the whole point of the exercise, and it's worth reading as a mapping rather than as a style guide:

| In the messy project              | What replaces it                                                 | The error it removes                                                      |
| --------------------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------------- |
| Run order remembered by hand      | `@oryxflow.requires(Upstream)` on each task                      | Out-of-order execution: the engine always runs dependencies first         |
| Magic constants in cells          | Parameters declared on the task (`oryxflow.FloatParameter(...)`) | "Which value produced this?" — the value is part of the output's identity |
| `to_csv('clean_v3.csv')` handoffs | `self.save(...)` and `self.inputLoad()`                          | Reading an orphaned or stale file as if it were current                   |
| "Run these cells, in this order"  | `python run.py`                                                  | A result nobody else can reproduce                                        |
| Numbers printed to scrollback     | A saved output you reload by naming its task                     | The headline figure that can't be found again                             |

Concretely, that lands as a small, boring project layout: task definitions in `tasks.py`, parameters in `flow_params.py`, environment and source paths in `cfg.py`, the workflow instance in `flow.py` (imported everywhere as `from flow import flow`), execution in `run.py`, and analysis in `visualize.py` or a report notebook. Exploration stays out of the pipeline entirely, under `eda/`. See [data-science project structure](https://docs.oryxflow.dev/docs/claude-plugin/project-structure/index.md) for why that layout is load-bearing rather than decorative.

## Two ways to do it

### 1. The one-command path with Claude Code

If you use [Claude Code](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md), install the oryxflow plugin and let it drive the restructuring:

```
/plugin marketplace add https://github.com/oryxintel/oryxflow-claude-plugin.git
/plugin install oryxflow@oryxflow
```

Order matters — there are two commands, and migrate builds *into* what init-project creates:

1. **`/oryxflow:init-project`** — scaffolds the runnable project structure if the directory doesn't have one yet. It never overwrites files you already have. Skip it only if `tasks.py` and `flow.py` already exist.
1. **`/oryxflow:migrate`** — reads your notebooks and scripts (it reads them, it does not run them), builds the step-to-task map, and shows you the plan.

It's **plan-then-apply**, deliberately. You see the proposed task map — each source step, the task name it becomes, what it saves, what it depends on — plus which constants become parameters, where the plots and probes and helpers land, and anything that *doesn't* decompose cleanly (an interactive cell needing a human in the loop, two concerns tangled in one block) flagged for you to resolve rather than silently split. Nothing is written until you say go. Then it builds up one task at a time, running the flow after each, so a break surfaces at the task that caused it instead of five tasks later.

One rule worth knowing before you start: **it never deletes your source.** The original notebooks and scripts are the spec it migrates *from* and the oracle it checks results against, so they stay exactly where they are. See [plugin commands](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) for the full command set.

### 2. By hand, incrementally

The same migration done manually is a short loop: pick one step, make it a task, run it, move on. The mechanics of turning functions into task classes — the `run()` body, `requires()`, parameters — are covered in the [transition guide](https://docs.oryxflow.dev/docs/transition/index.md); the [notebook-to-pipeline walkthrough](https://docs.oryxflow.dev/blog/reproducibility/notebook-to-reproducible-pipeline/index.md) tells the same story as a narrative if you'd rather read it start to finish.

Start with the step that hurts most, which is almost always the slow load. Wrap it in a task, run it once, and feed the rest of your existing cells from `flow.outputLoad()`. You've gained caching and lineage for the expensive part while rewriting almost nothing — and you can stop there for a week if you want.

## How to cut a notebook into tasks

The hard part of migrating isn't the syntax, it's deciding where one task ends and the next begins. Use the seams that are already in the code:

- **Every block that consumes inputs and produces an intermediate is a candidate task.** A loaded frame, a cleaned frame, a feature matrix, a fitted model, an evaluation table — each of those is one task.
- **Every `read_csv` / `read_excel` / database pull at the top is a source task.** No `requires`, and its path comes from `cfg.py`, not a string typed into the body.
- **Every `to_csv(...)` followed later by a `read_csv(...)` is one dependency edge.** That pair becomes `self.save(...)` upstream and `self.inputLoad()` downstream, and the intermediate file simply goes away — oryxflow owns where data lands.
- **Every magic constant becomes a parameter or a config value.** Thresholds, date ranges, model choices belong in `flow_params.py`; the source data directory and environment belong in `cfg.py`.
- **Name each task for the OUTPUT it produces, never the verb.** `SalesClean`, `SalesFeatures`, `MarginModel` — not `GetData`, `Process`, `RunModel`. The output is what downstream code asks for and what the cache is keyed on, so the name reads as a noun. Order the words broad to narrow so related tasks share a leading token and cluster together.

### Before: one linear script

Here's a typical notebook flattened into the script it really is. Three problems are load-bearing: the CSV handoff in the middle, the two constants, and the fact that correctness depends on running it top to bottom without ever editing a cell.

```
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

df = pd.read_csv('data/sales_raw.csv', parse_dates=['order_date'])   # slow, re-read every time
df = df[df['amount'] > 0]                                            # magic constant
df['region'] = df['region'].str.strip().str.lower()
df.to_csv('data/clean_v3.csv', index=False)                          # hand-managed handoff

df = pd.read_csv('data/clean_v3.csv', parse_dates=['order_date'])    # ...is this file current?
df['amount_log'] = np.log1p(df['amount'])
df['days_since'] = (pd.Timestamp('2026-01-01') - df['order_date']).dt.days   # magic date

X = df[['amount_log', 'days_since']]
model = LinearRegression().fit(X, df['margin'])
print(model.score(X, df['margin']))                                  # the number, in scrollback
```

### After: four tasks

The same work, cut at its seams. Each task is named for what it produces, declares what it needs, and ends in `self.save(...)`. The two constants are now parameters, so the cached output of each task is tied to the values that made it.

```
from datetime import date
import numpy as np
import pandas as pd
import oryxflow
from sklearn.linear_model import LinearRegression


class DataSales(oryxflow.tasks.TaskPqPandas):
    """Raw order rows, exactly as delivered by the source export.

    Out: one row per order; order_date parsed to datetime.
    """
    def run(self):
        self.save(pd.read_csv('data/sales_raw.csv', parse_dates=['order_date']))


@oryxflow.requires(DataSales)
class SalesClean(oryxflow.tasks.TaskPqPandas):
    """Order rows with non-positive amounts dropped and region normalized.

    In:  raw orders (from DataSales).
    Out: same columns; region lowercased and stripped.
    """
    amount_min = oryxflow.FloatParameter(default=0.0)

    def run(self):
        df = self.inputLoad()
        df = df[df['amount'] > self.amount_min]
        df['region'] = df['region'].str.strip().str.lower()
        self.save(df)


@oryxflow.requires(SalesClean)
class SalesFeatures(oryxflow.tasks.TaskPqPandas):
    """Model-ready features, one row per order.

    In:  cleaned orders (from SalesClean).
    Out: amount_log, days_since, margin. days_since is measured from asof_date.
    """
    asof_date = oryxflow.DateParameter(default=date(2026, 1, 1))

    def run(self):
        df = self.inputLoad()
        df['amount_log'] = np.log1p(df['amount'])
        df['days_since'] = (pd.Timestamp(self.asof_date) - df['order_date']).dt.days
        self.save(df[['amount_log', 'days_since', 'margin']])


@oryxflow.requires(SalesFeatures)
class MarginModel(oryxflow.tasks.TaskPickle):
    """Fitted linear model predicting margin from the order features.

    In:  feature matrix (from SalesFeatures).
    Out: the fitted estimator (pickled).
    """
    def run(self):
        df = self.inputLoad()
        X, y = df[['amount_log', 'days_since']], df['margin']
        self.save(LinearRegression().fit(X, y))
```

Pick the task type by what the step saves: `TaskPqPandas` for a DataFrame (the default and the fastest), `TaskPickle` for a model or any other Python object, `TaskJson` for a small dict. A step that genuinely produces two outputs declares them — `persists = ['train', 'valid']` — saves a dict with those keys, and downstream you read one with `self.inputLoad(keys='train')`. More task types and patterns are in [writing tasks](https://docs.oryxflow.dev/docs/tasks/index.md).

Running it is one object, and it's the object every other file in the project imports:

```
flow = oryxflow.Workflow(MarginModel, {'amount_min': 0.0})

flow.preview()                            # print the execution plan, run nothing
flow.run()                                # runs DataSales -> SalesClean -> ... in order

model = flow.outputLoad()                 # the final task's output
df_features = flow.outputLoad(SalesFeatures)   # any intermediate, by name
```

Notice what disappeared: you never call `DataSales` yourself, you never name an intermediate file, and the parameters you swept are recorded rather than remembered. Change `amount_min` and you get a *new* cached result alongside the old one, not an overwritten file. See [running workflows](https://docs.oryxflow.dev/docs/run/index.md) and [workflow parameters](https://docs.oryxflow.dev/docs/advparam/index.md).

## Migrate incrementally, and let exploration stay exploration

Two habits keep a migration from turning into a rewrite.

**Convert one step at a time, in dependency order.** Extract the config and parameters first, then add the root loader and run it, then add each downstream task and run again. A pipeline that runs after every step means a failure points at the task you just wrote. Re-run after an edit and oryxflow reruns the edited task and everything downstream on its own — see [automatic code invalidation](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation).

**Exploration doesn't have to become tasks at all.** Checking a schema, eyeballing a distribution, testing a parsing guess — that work is read-only, produces no pipeline artifact, and belongs in a plain script under `eda/<subject>/<name>.py`, each stating the question it answers. Promote a probe into a task only when it turns out to be load-bearing, meaning something downstream now depends on its result. That's what makes the structure scale in both directions: simple scripts on day one, any complexity later, and no cliff in between where you have to stop and rearchitect.

## How you know the migration is correct

A migration that runs is not a migration that's right. Four checks, in order:

1. **Keep the original as the spec.** Don't delete or overwrite the notebooks and scripts you migrated from. They are the only record of what the pipeline is supposed to do, and the oracle for the next check. Move them under a `legacy/` folder if they're in the way.
1. **`flow.preview()` before you run anything.** It prints the task tree — what will run, what's already cached — so you can confirm the graph you built is the graph you meant. If a dependency is missing, you see it here, not in a wrong number later.
1. **Compare a headline number.** Run the flow, then `flow.outputLoad()` the result and check one real figure against the same figure in the original. This is the step people skip, and it's the only one that catches silently changed behavior.
1. **Run it twice.** The second `flow.run()` should do nothing at all — every task a cache hit. That's the proof your caching is wired correctly; `print(result.summary())` on the returned result spells out how many tasks ran versus were skipped, and `result.ran` lists exactly which recomputed.

```
result = flow.run()
print(result.summary())        # e.g. "4 ran successfully" the first time, 0 the second
print(result.ran)              # which tasks actually recomputed
```

From then on, `python run.py` reproduces the whole analysis from raw data, and that command is the answer to "how was this made?"

## What not to migrate

Migrating everything is its own kind of mess. Leave two things alone:

- **Genuinely throwaway code.** If you're poking at a new dataset for twenty minutes and will never run any of it twice, task scaffolding is pure overhead. A good rule: migrate a step the second time you wait for it to recompute something that didn't change.
- **Production scheduling.** oryxflow is built for the research loop, not for cron, retries, alerting, and SLAs across a fleet. If you need those, an orchestrator does that job and oryxflow sits beside it, handling the pipeline's caching and lineage.

Also not this page: if your project is *already* pipeline-shaped but imports the old `d6tflow` package, that's a package rename rather than a restructuring — see [migrating from d6tflow](https://docs.oryxflow.dev/blog/guides/migrate-from-d6tflow-to-oryxflow/index.md).

## Frequently asked questions

**How do I restructure a messy data-science project into a pipeline?** Read the notebook or script first and cut it at its seams: every block that consumes some inputs and produces an intermediate — a loaded frame, a cleaned frame, a feature matrix, a fitted model — becomes one task, named for the intermediate it produces. Wire the tasks with `@oryxflow.requires`, end each `run()` with `self.save()`, and hoist the magic constants into parameters. Then build the tasks up in dependency order, running the flow after each one, so a break surfaces at the task that caused it instead of five tasks later.

**Do I have to convert the whole notebook at once?** No, and you shouldn't. Convert the step that hurts most first — usually the slow data load — run it once so its result caches, and let the rest of your cells keep working, now fed by `flow.outputLoad()`. Add the next step only when it earns it. You have a working pipeline after every single step, so there is never a half-rewritten state to recover from.

**How do I know the migration didn't change my numbers?** Keep the original notebook or script in place and treat it as the specification: it is the only record of what the pipeline is supposed to do, and it is the oracle you check against. Once the migrated flow runs end to end, compare a headline number from `flow.outputLoad()` against the same number in the original. A migration that runs but yields different numbers has silently changed behavior, so diagnose the difference rather than declaring success because the run finished without an error.

**Does exploratory analysis have to become oryxflow tasks?** No. Read-only probes — checking a schema, eyeballing a distribution, testing a parsing guess — stay plain scripts under `eda/<subject>/<name>.py`, each stating the question it answers, and they write no pipeline output. Promote a probe into a task only when it turns out to be load-bearing, meaning something downstream depends on its result. That way there is no cliff between exploring and having a pipeline.

**Why does an AI coding agent make a messy notebook project worse instead of better?** Because the mess is invisible state, and an agent cannot see it. It cannot tell which cells you actually ran, which intermediate file is current, or which constant produced the number in the notebook's output, so it re-derives that state by guessing — and writes plausible code on top of a wrong guess. Moving the run order into `@oryxflow.requires` and the intermediates into a cache makes that state readable instead of guessable, which is why the same structure that helps a human helps an agent more.

## Takeaway

- The payoff isn't tidier code, it's that **a stale intermediate or an unrecorded parameter can no longer hand you a confident wrong number** — and an AI coding agent stops guessing at state it can't see.
- Cut at the seams already in your code: each block that produces an intermediate becomes one task, named for that intermediate; the `to_csv` / `read_csv` handoffs become `self.save()` / `self.inputLoad()`; the magic constants become parameters.
- Migrate **incrementally**, keep the original as the spec, and verify by comparing one headline number — then confirm a second run does nothing.
- With Claude Code, `/oryxflow:init-project` then `/oryxflow:migrate` does the mapping and builds it up one task at a time, showing you the plan before it writes anything.

Next: the [transition guide](https://docs.oryxflow.dev/docs/transition/index.md) for the hands-on conversion mechanics, the [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) for the automated path, [writing tasks](https://docs.oryxflow.dev/docs/tasks/index.md) for the task types and patterns you'll reach for, and [why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md) for where this fits next to trackers and orchestrators.

# Writing and Managing Tasks

## What are tasks?

A task is one step of your analysis — load the data, clean it, train the model — packaged so oryxflow can manage it for you. Instead of a loose function plus a hand-written line to read its input file and another to write its output, you declare what a task **depends on** and what it **produces**, and the engine handles the rest: it runs upstream steps first, skips anything already computed, and hands each task its inputs already loaded. Tasks are the main object you will be interacting with. They let you:

- define input dependency tasks — so you declare the pipeline order once instead of re-wiring it every run
- process data
- load input data from upstream tasks — already loaded, no manual file paths
- save output data for downstream tasks — so results are cached and reused, not recomputed
- run tasks — the engine runs only what's missing
- load output data — fetch any result by referencing the task that made it

You write your own tasks by inheriting from one of the predefined oryxflow task formats, for example pandas dataframes saved to parquet. Picking the parent class is how you choose the output format (parquet, CSV, pickle, in-memory, ...) without writing any save/load code yourself — see [Targets](https://docs.oryxflow.dev/docs/targets/index.md).

```
class YourTask(oryxflow.tasks.TaskPqPandas):
```

Additional details on how to write tasks is below. To run tasks see [Running Workflows](https://docs.oryxflow.dev/docs/run/index.md).

## Define Upstream Dependency Tasks

You can define input dependencies by using a @oryxflow.requires decorator which takes input tasks. You can have no, one or multiple input tasks. This may be required when the decorator shortcut does not work.

Tip

The [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) writes this wiring for you - ask it to "add a task that takes `<Upstream>`'s output" and it emits the task class with the correct `@oryxflow.requires` decorator.

```
# no dependency
class TaskSingleInput(oryxflow.tasks.TaskPqPandas):
    #[...]

# single dependency
@oryxflow.requires(TaskSingleOutput)
class TaskSingleInput(oryxflow.tasks.TaskPqPandas):
    #[...]

# multiple dependencies
@oryxflow.requires({'input1':TaskSingleOutput1, 'input2':TaskSingleOutput2})
class TaskMultipleInput(oryxflow.tasks.TaskPqPandas):
    #[...]
```

## Process Data

You process data by writing a `run()` function. This function will take input data, process it and save output data.

```
class YourTask(oryxflow.tasks.TaskPqPandas):

    def run(self):
        # load input data
        # process data
        # save data
```

## Load Input Data

Input data from upstream dependency tasks can be easily loaded in `run()`

```
# no dependency
class TaskNoInput(oryxflow.tasks.TaskPqPandas):

    def run(self):
        data = pd.read_csv(oryxflow.settings.dirpath/'file.csv') # data/file.csv

# single dependency, single output
@oryxflow.requires(TaskSingleOutput)
class TaskSingleInput(oryxflow.tasks.TaskPqPandas):
    def run(self):
        data = self.inputLoad()

# single dependency, multiple outputs
@oryxflow.requires(TaskMultipleOutput)
class TaskSingleInput(oryxflow.tasks.TaskPqPandas):
    def run(self):
        data1, data2 = self.inputLoad()  # load all outputs
        # or load just one specific output by its persists name
        data1 = self.inputLoad(keys='output1')
        # equivalent lower-level spelling
        data1 = self.input()['output1'].load()

# multiple dependencies, single output
# prefer the named-dict form: you select deps by meaningful name, not by position
@oryxflow.requires({'input1':TaskSingleOutput1, 'input2':TaskSingleOutput2})
class TaskMultipleInput(oryxflow.tasks.TaskPqPandas):
    def run(self):
        data1 = self.inputLoad()['input1']
        data2 = self.inputLoad()['input2']
        # or
        data1 = self.inputLoad(task='input1')
        data2 = self.inputLoad(task='input2')

# multiple dependencies, multiple outputs
@oryxflow.requires({'input1':TaskMultipleOutput1, 'input2':TaskMultipleOutput2})
class TaskMultipleInput(oryxflow.tasks.TaskPqPandas):
    def run(self):
        data = self.inputLoad(as_dict=True)
        data1a = data['input1']['output1']
        data1b = data['input1']['output2']
        data2a = data['input2']['output1']
        data2b = data['input2']['output2']
        # or
        data1a, data1b = self.inputLoad()["input1"]
        data2a, data2b = self.inputLoad()["input2"]
        # or
        data1a, data1b = self.inputLoad(task='input1')
        data2a, data2b = self.inputLoad(task='input2')

# multiple dependencies (positional, without a dictionary), multiple outputs
# works, but the named-dict form above is preferred — here deps are selected by
# integer position (0, 1) instead of by name
@oryxflow.requires(TaskMultipleOutput1, TaskMultipleOutput2)
class TaskMultipleInput(oryxflow.tasks.TaskPqPandas):
    def run(self):
        data = self.inputLoad(as_dict=True)
        data1a = data[0]['output1']
        data1b = data[0]['output2']
        data2a = data[1]['output1']
        data2b = data[1]['output2']
        # or
        data1a, data1b = self.inputLoad()[0]
        data2a, data2b = self.inputLoad()[1]
        # or
        data1a, data1b = self.inputLoad(task=0)
        data2a, data2b = self.inputLoad(task=1)
```

### Load External Files

You probably want to load external data which is not the output of a task. There are a few options.

```
class TaskExternalData(oryxflow.tasks.TaskPqPandas):

    def run(self):

        import pandas as pd
        # read from oryxflow data folder
        data = pd.read_parquet(oryxflow.settings.dirpath/'file.pq')

        # totally manual
        data = pd.read_parquet('/some/folder/file.pq')

        # multiple files
        data = pd.concat([pd.read_csv(f) for f in glob.glob('*.csv')], ignore_index=True)
```

For more advanced options see [Sharing Workflows and Outputs](https://docs.oryxflow.dev/docs/collaborate/index.md)

### Dynamic Inputs

See [Dynamic Tasks](https://docs.oryxflow.dev/docs/advtasksdyn/index.md)

## Save Output Data

Saving output data is quick and convenient. You can save a single or multiple outputs.

```
# quick save one output
class TaskSingleOutput(oryxflow.tasks.TaskPqPandas):

    def run(self):
        self.save(data_output)

# save more than one output
class TaskMultipleOutput(oryxflow.tasks.TaskPqPandas):
    persists=['output1','output2'] # declare what you will save

    def run(self):
        self.save({'output1':data1, 'output2':data2}) # needs to match persists
```

`persist` (singular) is a backwards-compatible alias for `persists`; prefer `persists`.

When you have multiple outputs and don't declare `persists` you will get `raise ValueError('Save dictionary needs to consistent with Task.persist')`

### What can I save?

Anything — you are not limited to dataframes. `self.save()` accepts whatever the task's parent class handles, so if your result is a dict, save it as JSON; if it's a trained model, save it as pickle. There's no need to force a non-tabular result into a dataframe.

```
# a dict saved as JSON, and loaded back as a dict
class TaskConfig(oryxflow.tasks.TaskJson):

    def run(self):
        self.save({'features': ['x1', 'x2'], 'nrows': 100})

# any python object saved as pickle
class TaskModel(oryxflow.tasks.TaskPickle):

    def run(self):
        self.save(model)
```

See [Targets](https://docs.oryxflow.dev/docs/targets/#which-task-class-should-i-use) for the full list of formats and which class saves what.

### Where Is Output Data Saved?

Output data by default is saved in `data/`, you can check with

```
oryxflow.settings.dirpath # folder where workflow output is saved
TaskTrain().output().path # file where task output is saved
```

You can change where data is saved using `oryxflow.set_dir('data/')`. See advanced options for [Sharing Workflows and Outputs](https://docs.oryxflow.dev/docs/collaborate/index.md) Global Data Path can be also changed by including the `path` parameter to the Workflow.

### Changing Task Output Formats

See [Targets](https://docs.oryxflow.dev/docs/targets/index.md)

## Running tasks

See [Running Workflows](https://docs.oryxflow.dev/docs/run/index.md)

## Load Output Data

Once a workflow is run and the task is complete, you can easily load its output data by referencing the task.

```
data = flow.outputLoad() # load default task output
data = flow.outputLoad(as_dict=True) # useful for multi output
data2 = flow.outputLoad(TaskMultipleOutput, as_dict=True) # load another task output
data2['data1']
data2['data2']
```

**Before you load output data you need to run the workflow**. See [run the workflow](https://docs.oryxflow.dev/docs/run/index.md). If a task has not been run, it will show

```
raise RuntimeError('Target does not exist, make sure task is complete')
RuntimeError: Target does not exist, make sure task is complete
```

### Which load method: `output().load()` vs `outputLoad()` vs `outputLoadConcat()`

There are three ways to get at a task's output, lowest- to highest-level. **Prefer the highest one that fits** — most code should just use `outputLoad()`.

- `task.output().load()` — **lowest level.** `output()` returns the *target object* (or a dict of targets for a multi-`persists` task), and `.load()` reads it. Reach for this only when you want the target itself — its `.path`, or a deliberate `.load()`. You index the target yourself, e.g. `task.output()['train'].load()`.
- `task.outputLoad()` / `flow.outputLoad()` — **the default; use this to fetch results.** Returns the *data* directly: a single object for a single-`persists` task, a list (or a dict with `as_dict=True`) for multiple outputs, and a `{flow: data}` dict for `WorkflowMulti`. It also checks the task is complete for you.
- `flow.outputLoadConcat()` — **narrow and opt-in;** `WorkflowMulti` only. Row-stacks every flow's output into **one** DataFrame, tagging each flow's rows with its params. Use it *only* when every flow's output is a schema-compatible DataFrame you want combined (the iterate-and-aggregate case). It is a separate method on purpose: it collapses the per-flow `{flow: data}` dict into a single frame — a different operation from `outputLoad`, and one that is meaningless for non-DataFrame outputs like models. See [advtasksdyn](https://docs.oryxflow.dev/docs/advtasksdyn/index.md).

The same three tiers exist on the **input** side inside `run()`: `self.input().load()` (the target), `self.inputLoad()` (the data — the default), and `self.inputLoadConcat()` (stack a task's dependencies into one frame).

### Loading Output Data with Parameters

If you are [using parameters](https://docs.oryxflow.dev/docs/advparam/index.md) this is how you load outputs. Make sure you run the task with that parameter first.

```
params = {'default_params':{}, 'use_params':{'preprocess':True}}
flow = oryxflow.WorkflowMulti(TaskSingleOutput, params)
data = flow.outputLoad() # load default task output
data['default_params']
data['use_params']

# multi output
data2 = flow.outputLoad(TaskMultipleOutput, as_dict=True) # load another task output
data2['default_params']['data1']
data2['default_params']['data2']
data2['use_params']['data1']
data2['use_params']['data2']
```

## Putting it all together

See the full [ML example](https://docs.oryxflow.dev/docs/example-ml/index.md).

To scaffold a real-life project layout, run [`/oryxflow:init-project`](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) in Claude Code.

## Advanced: task attribute overrides

`persist`: data items to save, see above `external`: do check dependencies, good for sharing tasks without providing code `code_version`: bump (str or int) when this task's logic changes so it and everything downstream recompute; see [Automatic code invalidation](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation) `keep_versions`: with `code_version` set, keep old versions at readable `.../<Task>/v<version>/` paths `target_dir`: specify directory `target_ext`: specify extension `save_attrib`: include taskid in filename

# Workflow

Workflow object is used to orchestrate tasks and define a task pipeline

## Define a workflow object

Workflow object can be defined by passing the parameters and the default task for the pipeline. Both the arguments are optional.

```
flow = Workflow(task=Task1, params = params)
```

### Defining the flow with just params

To define a workflow object with just parameters:

```
flow = Workflow(params = params)
```

### Previewing the flow

The pipeline can be previewed for the defined flow and passing the task. If nothing is passed, default task used during the initiation of the flow object is used

```
flow.preview(Task1)
```

### Running the flow

A list of tasks can run for the defined parameters of the flow. Other arguments that can be passed during the flow are: forced, forced_all,`forced_all_upstream`, confirm, workers, abort, execution_summary. Any additional named arguments can also be passed for the task objects.

```
flow.run(Task1)
```

### Getting the output load for the flow

To get the outputload for a specific task, after running it:

```
flow.run(Task1)
flow.outputLoad(Task1)
```

### Getting the output load for the flow including upstream tasks

To get the outputload for a specific task along with its upstream tasks, after running it:

```
flow.run(Task1)
flow.outputLoadAll(Task1)
```

### Reset task for the flow

To reset the task for the flow:

```
flow.reset(Task1)
```

### Reset downstream tasks for the flow

To reset the task for the flow:

```
flow.reset_downstream(Task1)
```

### Setting the default task for the flow

To set the default task for the flow:

```
flow.set_default(Task1)
```

### Getting the task the for flow

A task object can be retrieved by calling the get_task method

```
flow.get_task(Task1)
```

## Define a multi experiment workflow object

A multi experiment workflow can be defined with multiple flows and separate parameters for each flow and a default task. It is mandatory to define the flows and parameters for each of the flows.

```
flow2 = oryxflow.WorkflowMulti(params = {'experiment1': {'do_preprocess': False}, 'experiment2': {'do_preprocess': True}}, task=Task1)
```

### Defining the flow with just params

To define a workflow object with just parameters:

```
flow = WorkflowMulti(params = params)
```

### Constructing the params grid

`params` maps a **flow name** to that flow's `{param: value}` dict, e.g. `{'experiment1': {'do_preprocess': False}, 'experiment2': {'do_preprocess': True}}`. You rarely write that by hand — pass a compact spec and let `WorkflowMulti` expand it.

**One param, many values** — pass a single-key dict whose value is a list; you get one flow per value:

```
oryxflow.WorkflowMulti(CountryTask, params={'country': ['US', 'UK']})
# -> flows {0: {'country': 'US'}, 1: {'country': 'UK'}}
```

**Several params (cartesian product)** — a multi-key dict of lists expands to every combination, with descriptive string flow names:

```
oryxflow.WorkflowMulti(TaskTrain, params={'model': ['ols', 'gbm'], 'scale': [False, True]})
# -> flows 'model_ols_scale_False', 'model_ols_scale_True', 'model_gbm_scale_False', ...
```

**Explicit list of param sets** — when the combinations are not a full grid, pass a list (flows are keyed by position):

```
oryxflow.WorkflowMulti(Task1, params=[{'param1': 1}, {'param1': 2}])
# -> flows {0: {'param1': 1}, 1: {'param1': 2}}
```

For finer control, build the `params` dict yourself with the helpers in `oryxflow.utils` (all take an optional `params_base` merged into every flow):

- `params_generator_single({'a': [1, 2, 3]})` — one flow per value of a single param.
- `params_generator_dictlist({'p1': ['a', 'b'], 'p2': ['c', 'd']})` — cartesian product, integer-keyed.
- `params_generator_df(df)` — one flow per row of a DataFrame (each row's columns become that flow's params); handy when your grid comes from a table.

```
params_all = oryxflow.utils.params_generator_single({'country': ['US', 'UK']}, {'env': 'prod'})
flow = oryxflow.WorkflowMulti(CountryTask, params=params_all)
```

This is the **top-level** grid — the flows to run. It is distinct from any nested enumeration you index inside a task's `requires()` (see "Hierarchical iterate-and-aggregate" in [advtasksdyn](https://docs.oryxflow.dev/docs/advtasksdyn/index.md)), which is your own domain data, not a `WorkflowMulti` grid.

Tip

Projects scaffolded by the [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) keep this wiring in two conventional homes — the grid in `flow_params.py`, the flow objects in `flow.py` — so experiments stay organized as the grid grows, and the AI extends them in the right place.

### Operations on multi experiment workflow

All the operations like run, preview, outputLoad, outputLoadAll, reset, reset_downstream , get_task can be called for the multi flow object. Each of these functions takes an extra `flow` argument selecting which flow to act on. If `flow` is not passed, the operation is applied to every flow.

```
flow2.run(Task1, flow = "experiment1")
flow2.preview(Task1, flow = "experiment2")
flow2.get_task(Task1, flow = "experiment1")
flow2.outputLoad(Task1, flow = "experiment1")
flow2.outputLoadAll(Task1, flow = "experiment1")
flow2.reset(Task1, flow = "experiment1")
flow2.reset_downstream(Task1, flow = "experiment1")
```

### Concatenate outputs across flows

To load a task's output for every flow and stack them into a single DataFrame (each flow's rows tagged with that flow's params), use `outputLoadConcat`:

```
flow = oryxflow.WorkflowMulti(CountryTask, params={'country': ['US', 'UK']})
flow.run()
dfall = flow.outputLoadConcat(CountryTask)   # one frame, 'country' column tags each flow
```

The one-liner runIterConcat builds the WorkflowMulti, runs it and returns the concatenated frame in one call:

```
dfall = oryxflow.runIterConcat(CountryTask, params={'country': ['US', 'UK']})
```

# Running Tasks and Managing Workflows

A workflow object is used to orchestrate tasks and define a task pipeline.

## Define a workflow object

Workflow object can be defined by passing the default task and the parameters for the pipeline. Both the arguments are optional.

```
flow = oryxflow.Workflow(Task1, params)
flow = oryxflow.Workflow(Task1) # use default params
```

Note you want to pass the task definition, not an instantiated task.

```
import tasks
flow = oryxflow.Workflow(tasks.Task1) # yes
flow = oryxflow.Workflow(tasks.Task1()) # no
```

## Previewing Task Execution Status

Running a task will automatically run all the upstream dependencies. Before running a workflow, you can preview which tasks will be run.

```
flow.preview() # default task
flow.preview(TaskTrain) # single task
flow.preview([TaskPreprocess,TaskTrain]) # multiple tasks
```

## Running Multiple Tasks as Workflows

To run all tasks in a workflow, run the downstream task you want to complete. It will check if all the upstream dependencies are complete and if not it will run them intelligently for you.

```
flow.run() # default task
flow.run(TaskTrain) # single task
flow.run([TaskPreprocess,TaskTrain]) # multiple tasks
```

If your tasks are already complete, they will not rerun. To force rerunning of all tasks but there are better alternatives, see below.

```
flow.run(forced_all_upstream=True, confirm=False) # use flow.reset() instead
```

## Run and Load in One Call

For quick scripts and notebooks, `oryxflow.runLoad` builds a workflow, runs the task (with all upstream dependencies) and returns its loaded output in a single call - saving you from creating a `Workflow` object just to fetch one result.

```
# equivalent to: oryxflow.Workflow(TaskTrain, params).run() then outputLoad()
model = oryxflow.runLoad(TaskTrain, params={'do_preprocess': True})

# reset=True forces a rerun first (for a data/input change or a suspect cache;
# a *code* change needs nothing — it invalidates automatically; see "Handling Code Change")
df = oryxflow.runLoad(TaskPreprocess, params={'do_preprocess': True}, reset=True)

# runIt runs without loading the output (same as runLoad(..., load=False))
oryxflow.runIt(TaskTrain)
```

## How is a task marked complete?

This is the mechanism behind "don't recompute what's already done" — the thing that lets you re-run a pipeline freely and only pay for what changed. Tasks are complete when task output exists. This is typically the existance of a file, database table or cache. See [Task I/O Formats](https://docs.oryxflow.dev/docs/targets/index.md) how task output is stored to understand what needs to exist for a task to be complete.

```
flow.get_task().complete() # status
flow.get_task().output().path # where is output saved?
flow.get_task().output()['output1'].path # multiple outputs
```

Every task also carries one more completeness condition: its code must still match the version that produced its output. By default this is tracked automatically — comparing what the task's code *does*, not how it's written — so editing a task's logic (or a helper it imports) makes it incomplete even though its output file is still on disk: "the output exists" never silently masks a code change. A task that declares an explicit `code_version` is pinned instead: only bumping the token marks it changed (edits without a bump produce a staleness warning). Be honest about the limit: code-change detection sees your task code and the project-local modules it imports, but **not** data-file contents or external APIs — a cache hit is not proof of freshness for those (reset is the verb there). See [Code changes: handled automatically](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation) for the full model.

### Task Completion with Parameters

If a task has parameters, it needs to be run separately for each parameter to be complete when using different parameter settings. The oryxflow.WorkflowMulti helps you do that

```
flow = oryxflow.WorkflowMulti(Task1, {'flow1':{'preprocess':False},'flow2':{'preprocess':True}})
flow.run() # will run all flow with all parameters
```

### Disable Dependency Checks

By default, for a task to be complete, it checks if all dependencies are complete also, not just the task itself. To check if just the task is complete without checking dependencies, set `oryxflow.settings.check_dependencies=False`

```
flow.reset(TaskGetData, confirm=False)
oryxflow.settings.check_dependencies=True # default
flow.preview() # TaskGetData is pending so all tasks are pending
'''
+--[TaskTrain-{'do_preprocess': 'True'} (PENDING)]
   +--[TaskPreprocess-{'do_preprocess': 'True'} (PENDING)]
      +--[TaskGetData-{} (PENDING)]
'''
oryxflow.settings.check_dependencies=False # deactivate dependency checks
flow.preview()
+--[TaskTrain-{'do_preprocess': 'True'} (COMPLETE)]
   +--[TaskPreprocess-{'do_preprocess': 'True'} (COMPLETE)]
      +--[TaskGetData-{} (PENDING)]
oryxflow.settings.check_dependencies=True # set to default
```

## Debugging Failures

If a task fails, oryxflow raises a `RuntimeError` chained to the original error that caused the failure (`... the direct cause of the following exception ...`). Read the FIRST traceback -- the line in your task's `run()` is the real cause. Example:

```
File "tasks.py", line 37, in run     <== the real error is here
    1/0
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:
...
RuntimeError: Exception found running flow, check trace
```

Tips:

- The first traceback (the `ZeroDivisionError` above) points at your bug; the trailing `RuntimeError` is just oryxflow reporting that the flow aborted.
- Set a breakpoint in the task's `run()` and step through it.
- Run a single task in isolation to debug it directly: `TaskTrain().run()` (note: this skips dependency resolution -- make sure upstream outputs already exist).
- Turn on engine logging to see which task failed and timing: `oryxflow.enable_logging()` (see "Logging" below).
- Every failure is also recorded durably in the event stream: `oryxflow.events.status()` returns recent failures (with the error and a bounded traceback) even after the script has exited, so a post-mortem doesn't depend on still having the run's stdout. See [Managing Complex Workflows](https://docs.oryxflow.dev/docs/managing-workflows/#managing-complex-workflows).

## Rerun Tasks When You Make Changes

You have several options to force tasks to reset and rerun. See sections below on how to handle parameter, data and code changes.

Tip

Editing a task's code with unchanged parameters is handled for you: the task plus everything downstream recompute on the next run — no reset, no version to bump (see [Code changes: handled automatically](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation)). Only tasks pinned with an explicit `code_version` need a bump in the same edit; the staleness advisory warns if you forget. Resets remain for what the hash can't see (data files, external APIs, a suspect cache). The [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) bakes this loop in for AI-driven projects: after each edit it verifies the intended tasks actually reran and answers warnings with the right exit.

```
# preferred way: reset single task, this will automatically run all upstream dependencies
flow.reset(TaskGetData, confirm=False) # remove confirm=False to avoid accidentally deleting data

# force execution including upstream tasks
flow.run([TaskTrain()],forced_all=True, confirm=False)

# force run everything
flow.run(forced_all_upstream=True, confirm=False)
```

### Which reset method: `reset` / `reset_upstream` / `reset_downstream`

All three are available on both `Workflow` and `WorkflowMulti`. Each invalidates task outputs so the next `run()` recomputes them; pick by *how much* of the DAG you want to reset:

- `flow.reset(task)` — **one task.** Invalidate just this task's output. The next run recomputes it and — because `complete()` is recursive — anything downstream of it; upstream tasks stay complete and are reused. The everyday choice.
- `flow.reset_upstream(anchor)` — **the whole upstream cone.** Invalidate `anchor` and every task it transitively depends on. Add `only=Family` (or a list of families) to reset *only* those families within the cone: the traversal still walks the full upstream to discover the instances, but only the matching ones are invalidated.

```
flow.reset_upstream(Sector)                          # reset everything upstream (leaf included)
flow.reset_upstream(Sector, only=CountryFeatures)    # reset just this family across the cone
flow.reset_upstream(Sector, only=[CountryFeatures, DataLoadState])   # multiple families
```

`only=` matches by **family/type**, which is what lets it reach tasks deep in the DAG whose params are *internal* (e.g. a per-`country`/`state` task you can't easily name from the flow's params) — and the families need not be adjacent. This is the tool for "reset the derived layers everywhere but keep the expensive source task." See [advtasksdyn](https://docs.oryxflow.dev/docs/advtasksdyn/index.md) for the hierarchical example it comes from. On a `WorkflowMulti` you can omit `anchor` — it defaults to the flow's default task, so `flow.reset_upstream(only=CountryFeatures)` resets that family across every flow.

- `flow.reset_downstream(task, task_downstream=None)` — **a task/family and everything downstream of it.** Only the *family* of `task` is used (pass the class), so — like `only=` — it reaches deep tasks whose params are internal to the DAG without naming instances. `task_downstream` is the terminal task the walk stops at and **defaults to the flow's default task**. Every task on the paths between them is invalidated **explicitly** (each output deleted), so the downstream recomputes even when the recursive `complete()` cascade is unavailable. Tasks upstream of the named family (the expensive source) are left intact.

```
flow.reset_downstream(CountryFeatures)                 # CountryFeatures + all downstream, up to the flow root
flow.reset_downstream(CountryFeatures, Sector)         # explicit terminal task
flow.reset_downstream([CountryFeatures, CountryRisk])  # several families + their downstream, one call
```

`reset_upstream(root, only=F)` vs `reset_downstream(F)` — both target a family without naming instances, but differ in *what* they reset and whether they lean on the cascade:

- `reset_upstream(root, only=CountryFeatures)` invalidates **only** the `CountryFeatures` instances; `Sector` (downstream) recomputes on the next run *via recursive* `complete()`. Use it when the cascade is reliable (the default, `check_dependencies=True`).
- `reset_downstream(CountryFeatures)` invalidates `CountryFeatures` **and everything downstream** explicitly, so it does not depend on the cascade. Use it when you want to be certain, or where the cascade can't be trusted (`check_dependencies=False`).

Mental model: `reset` = one node; `reset_upstream` = the cone *above* a node (optionally filtered to families; downstream recompute relies on the cascade); `reset_downstream` = a node/family and everything *below* it down to a terminal task (invalidated explicitly).

### When to reset and rerun tasks?

Three things make a cached result out of date, and each has its *own* right verb — reset is only one of them:

- **parameters changed** → nothing to do; a new parameter is a new identity and reruns automatically, keeping the outputs for each parameter set side by side.
- **code changed** (this task's `run()` or a helper it imports) → nothing to do; the task plus everything downstream recompute automatically. Don't hand-chain resets for code changes. (A task pinned with an explicit `code_version` is the exception: bump its token in the same edit.)
- **data or an external input changed** (a raw file, an API response — things code-change detection can't see) → **reset** the task that ingests it.

The full "which verb, when" decision table is in [Managing Complex Workflows](https://docs.oryxflow.dev/docs/managing-workflows/#managing-complex-workflows). The sections below cover each case.

### Handling Parameter Change

As long as the parameter is defined in the task, oryxflow will automatically rerun tasks with different parameters.

```
flow = oryxflow.WorkflowMulti(Task1, {'flow1':{'preprocess':False},'flow2':{'preprocess':True}})
flow.run() # executes 2 flows, one for each task
```

For oryxflow to intelligently figure out which tasks to rerun, the parameter has to be defined in the task. The downstream task (TaskTrain) has to pass on the parameter to the upstream task (TaskPreprocess).

```
class TaskGetData(oryxflow.tasks.TaskPqPandas):
# no parameter dependence

class TaskPreprocess(oryxflow.tasks.TaskCachePandas):  # save data in memory
    do_preprocess = oryxflow.BoolParameter(default=True) # parameter for preprocessing yes/no

@oryxflow.requires(TaskPreprocess)
class TaskTrain(oryxflow.tasks.TaskPickle):
    # pass parameter upstream
    # no need for to define it again: do_preprocess = oryxflow.BoolParameter(default=True)
```

See [handling parameter inheritance](https://docs.oryxflow.dev/docs/advparam/#avoid-repeating-parameters-in-every-class).

### Default Parameter Values in Config

As an alternative to inheriting parameters, you can define defaults in a config files. When you change the config it will automatically rerun tasks.

```
class TaskPreprocess(oryxflow.tasks.TaskCachePandas):  
    do_preprocess = oryxflow.BoolParameter(default=cfg.do_preprocess) # store default in config
```

### Handling Data Change

A raw data file or an external API response is invisible to oryxflow — no parameter changes and no code change is detected, so nothing reruns on its own. When you know an input changed, `reset()` the task that *ingests* it (the loader/source task) so the recompute starts where the new data enters and cascades downstream. Resetting a task further downstream would just reload the same cached old input.

### Handling Code Change

Code changes need no action: oryxflow compares what each task's code *does*, not how it's written — so editing a task's logic (or a helper it imports) reruns the task *and everything downstream* on the next run, while comment, docstring, and formatting-only edits are ignored, with no resets to chain. Verify it took — `result.reasons` shows `code change (auto: <file>)` — and if an expected rerun didn't happen, the change is somewhere it can't see (data file, installed package, dynamic call): reset it. Two deliberate exceptions hold their cache and **warn** instead of rerunning: tasks pinned with an explicit `code_version` (recompute only on a bump) and expensive tasks whose last run exceeded `settings.code_version_auto_expensive_s` (default 600s — answer with reset / `accept_code` / pin). See [Code changes: handled automatically](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation) for the full model, the pin workflow, the three exits (recompute / `accept_code` / reset), and `keep_versions` for keeping old versions side by side.

### Forcing a Single Task to Run

You can always run single tasks by calling the run() function. This is useful during debugging. However, this will only run this one task and not take care of any downstream dependencies.

```
# forcing execution
flow.get_task().run()
# or
TaskTrain().run()
```

## Hiding Execution Output

By default, the workflow execution summary is shown, because it shows important information which tasks were run and if any failed. At times, eg during deployment, it can be desirable to not show the execution output.

```
oryxflow.settings.execution_summary = False # global
# or
flow.run(execution_summary=False) # at each run
```

## Logging

oryxflow can log engine activity (task start/complete, timing, failures) and gives each task a contextual `self.logger` for logging from inside your own `run()`. Logging is disabled by default. Quick start:

```
import oryxflow
oryxflow.enable_logging()                 # INFO+ to stderr
oryxflow.enable_logging(level="DEBUG")    # also I/O, cached-skips, dependency detail
oryxflow.disable_logging()                # silence again
```

See [logging](https://docs.oryxflow.dev/docs/logging/index.md) for the full guide, including `self.logger`, log levels, and routing oryxflow records into your application's own loguru sinks.

## Cloud Storage

Point your pipeline at cloud storage and the whole team reads and writes the same outputs — no one re-runs a task someone else already ran, and results are backed up off your laptop. By default task output is written under the local data directory (`oryxflow.set_dir()`). You can instead store output in cloud storage (S3, GCS, etc.) - oryxflow uses [fsspec](https://github.com/fsspec) / [universal-pathlib](https://pypi.org/project/universal-pathlib/) under the hood, so task code does not change.

Install the relevant extra first, e.g. `pip install oryxflow[gcs]` or `pip install oryxflow[s3]` (`cloud-base` for other fsspec protocols), then enable it once before running:

```
import oryxflow

# Google Cloud Storage shortcut
oryxflow.enable_gcs(bucket='my-bucket', prefix='myproject')

# any fsspec protocol (s3, gcs, dropbox, ...)
oryxflow.enable_cloud_storage(protocol='s3', bucket='my-bucket', prefix='myproject')

flow = oryxflow.Workflow(TaskTrain)
flow.run()   # task output now reads/writes under s3://my-bucket/myproject/
```

`prefix` is optional and behaves like a top-level folder within the bucket.

# Task I/O Targets

The format your task output is saved in matters more than it first appears: it decides how fast your pipeline reads and writes between steps, whether a result survives a restart or lives only for the session, and whether a teammate can open the file directly. oryxflow lets you pick that format by **choosing a parent class** — you never write save/load code, and you can switch a task from parquet to CSV (or to an in-memory cache while you iterate) by changing one base class.

## How is task data saved and loaded?

Task data is saved in a file or memory (cache). You control the format by choosing the right parent class for a task. In the example below, data is saved as parquet and loaded as a pandas dataframe because the parent class is `TaskPqPandas`.

```
class YourTask(oryxflow.tasks.TaskPqPandas):
```

**Your output does not have to be a dataframe.** A task can save a plain python dict as JSON, any python object as pickle, or a report as markdown, just as easily. Pick the class that matches the object you already have — you should never have to reshape a dict into a dataframe to get it saved.

### Task Output Location

By default file-based task output is saved in `data/`. You can customize where task output is saved.

```
oryxflow.set_dir('../data')
```

## Which task class should I use?

Start from the object you want to save and read across:

| What you're saving                    | Use                     | Saved as                          | You get back                   |
| ------------------------------------- | ----------------------- | --------------------------------- | ------------------------------ |
| A dataframe                           | `TaskPqPandas`          | `.parquet`                        | dataframe                      |
| A dataframe someone needs to open     | `TaskCSVPandas`         | `.csv`                            | dataframe                      |
| A large dataframe as CSV              | `TaskCSVGZPandas`       | `.csv.gz`                         | dataframe                      |
| Several dataframes for one Excel file | `TaskExcelPandas`       | one `.xlsx`, one sheet per output | dataframe                      |
| Several dataframes as separate files  | `TaskExcelPandasSingle` | one `.xlsx` per output            | dataframe                      |
| A dict, list, or config               | `TaskJson`              | `.json`                           | the same dict/list             |
| A trained model or any python object  | `TaskPickle`            | `.pkl`                            | the same object                |
| A written report                      | `TaskMarkdown`          | `.md` **and** `.html`             | markdown string                |
| Anything, but only for this session   | `TaskCache`             | memory                            | the same object                |
| A dataframe, only for this session    | `TaskCachePandas`       | memory                            | dataframe                      |
| A dataframe to a database             | `TaskSQLPandas`         | SQL table                         | dataframe (premium, see below) |

A rough guide: reach for **parquet** (`TaskPqPandas`) for most dataframes — it's fast and compact and keeps dtypes; **CSV/Excel** when a human needs to open the file; **JSON** for anything dict-shaped you'd like to be able to read and diff; **pickle** for trained models or python objects JSON can't express; and the in-memory **cache** targets (`TaskCache*`) for intermediate results you don't need on disk between runs (fastest, but gone when the process exits).

dask, SQL and pyspark are premium features, see below.

## Saving dicts and JSON

`TaskJson` saves anything JSON-serializable — a dict, a list, nested combinations — and gives you back exactly that object. It's the natural choice for configs, parameter sets, summary metrics, API responses and label maps, and the file stays readable and diffable in git.

```
import oryxflow

class GetConfig(oryxflow.tasks.TaskJson):

    def run(self):
        self.save({'features': ['x1', 'x2'], 'nrows': 100})

@oryxflow.requires(GetConfig)
class SummarizeConfig(oryxflow.tasks.TaskJson):

    def run(self):
        cfg = self.inputLoad()   # a plain python dict, no dataframe involved
        self.save({'nfeatures': len(cfg['features'])})

flow = oryxflow.Workflow(SummarizeConfig)
flow.run()
flow.outputLoad()   # {'nfeatures': 2}
```

Multiple outputs work the same way as for dataframes — declare `persists` and save a dict keyed by those names (see [saving more than one output](https://docs.oryxflow.dev/docs/tasks/#save-output-data)):

```
class ScoreModel(oryxflow.tasks.TaskJson):
    persists = ['metrics', 'warnings']

    def run(self):
        self.save({'metrics': {'auc': 0.81}, 'warnings': ['few samples in fold 3']})

flow_score = oryxflow.Workflow(ScoreModel)
flow_score.run()
metrics = flow_score.outputLoad(keys='metrics')
```

## Saving models and other python objects

If JSON can't express it — a fitted model, a scaler, a set, a custom class — use `TaskPickle`. It takes any picklable object and hands the same object back.

```
class TrainModel(oryxflow.tasks.TaskPickle):

    def run(self):
        from sklearn.linear_model import LogisticRegression
        self.save(LogisticRegression())

flow_model = oryxflow.Workflow(TrainModel)
flow_model.run()
model = flow_model.outputLoad()
```

**NB**: don't save a dict of dataframes as pickle — save them as multiple outputs instead, see "save more than one output" in [Tasks](https://docs.oryxflow.dev/docs/tasks/#save-output-data).

## Saving reports

`TaskMarkdown` takes a markdown string and writes both a `.md` file and a styled `.html` file next to it, so a summary task can produce something you can send to someone.

```
class WriteReport(oryxflow.tasks.TaskMarkdown):

    def run(self):
        self.save('# Results\n\nThe model scored 0.81 AUC.\n')

flow_report = oryxflow.Workflow(WriteReport)
flow_report.run()
flow_report.outputPath()   # data/WriteReport/...-data.md (plus the .html alongside)
```

## Premium Targets (Dask, SQL, Pyspark)

### Database Targets

oryxflow premium has database targets.

### Dask Targets

oryxflow premium has dask targets.

### Pyspark Targets

oryxflow premium has pyspark targets.

## Community Targets

### Keras Model Targets

For saving Keras model targets

```
from oryxflow.tasks.h5 import TaskH5Keras
```

## Writing Your Own Targets

This is often relatively simple since you mostly need to implement load() and save() functions. For more advanced cases you also have to implement exist() and invalidate() functions. Check the source code for details or raise an issue.

# Logging

oryxflow uses [loguru](https://loguru.readthedocs.io) for logging. So it never interferes with your application's own logging, **oryxflow logging is disabled by default** — out of the box oryxflow emits nothing except the execution summary print. You opt in when you want to see what the engine is doing or to log from inside your own tasks.

## Turning logging on and off

```
import oryxflow

oryxflow.enable_logging()                 # INFO and above, written to stderr
oryxflow.enable_logging(level="DEBUG")    # more detail (see levels below)
oryxflow.disable_logging()                # silence oryxflow again

flow = oryxflow.Workflow(MyTask)
flow.run()
```

With the default `sink=sys.stderr` you get one clean oryxflow log stream: `enable_logging()` removes loguru's pristine default stderr handler and installs a oryxflow-filtered one at the chosen level (so records are not printed twice). Calling it again just replaces that handler rather than stacking another. `disable_logging()` silences oryxflow again at the source.

Color is auto-detected: by default (`colorize=None`) records are colored only when the sink is an interactive terminal, so redirected or captured output (files, pipes, pytest capture) stays free of ANSI escape codes. Force it either way with `enable_logging(colorize=True)` / `enable_logging(colorize=False)`.

`enable_logging()` returns the loguru handler id of the sink it added. Keep it if you want to remove that specific sink later:

```
from loguru import logger
hid = oryxflow.enable_logging()
...
logger.remove(hid)
```

## What gets logged

The default `enable_logging()` level is `INFO`. Each level is cumulative — `DEBUG` shows everything `INFO` shows, plus more.

- **INFO** — the "what's happening" stream: task start, task complete (with duration), task and dependency failures, run summary, and task invalidation.
- **DEBUG** — adds the verbose detail: skipped (already-complete) tasks, save / load / input I/O with their keys, and generator yields.
- **WARNING** — an `external=True` task whose output is missing.
- **ERROR** — a task's `run()` raised; the full traceback is logged via loguru.

Example INFO output for a two-task flow:

```
... | INFO  | oryxflow.core - task start: Task1 (Task1__99914b932b)
... | INFO  | oryxflow.core - task complete: Task1__99914b932b in 0.041s
... | INFO  | oryxflow.core - task start: Task3 (Task3__a1b2c3d4e5)
... | INFO  | oryxflow.core - task complete: Task3__a1b2c3d4e5 in 0.046s
... | INFO  | oryxflow.core - run summary: scheduled=2 ran=2 complete=0 failed=0
```

At `DEBUG` you additionally see the I/O and cached skips:

```
... | DEBUG | oryxflow.tasks - saved Task1__99914b932b keys=['data']
... | DEBUG | oryxflow.tasks - loaded input for Task3__a1b2c3d4e5 keys=['input1']
... | DEBUG | oryxflow.core - task skipped (already complete): Task1__99914b932b
```

## Logging inside your own tasks

Every task has a contextual `self.logger`: a loguru logger pre-tagged with this task's `task_id` and `task_family`. Use it in your `run()` (or any task method) so your messages carry the task identity automatically:

```
class TaskTrain(oryxflow.tasks.TaskPickle):
    def run(self):
        df = self.inputLoad()
        self.logger.info("training on {} rows", len(df))    # tagged task_id / task_family
        model = train(df)
        self.logger.debug("converged in {} iterations", model.n_iter_)
        self.save(model)
```

`self.logger` lives in oryxflow's logging namespace (its records show under the name `oryxflow.task`), so — like the engine logs — it is silent until you call `oryxflow.enable_logging()` and is silenced again by `disable_logging()`, no matter which module your task class is defined in. The `task_id` / `task_family` tags are attached to each record's `extra` dict; include them in a custom format to display them:

```
from loguru import logger
oryxflow.enable_logging(level="DEBUG")
logger.add("flow.log", filter="oryxflow",
           format="{time} | {level} | {extra[task_family]} | {message}")
```

If you would rather log independently of oryxflow's on/off switch, just use your own `from loguru import logger` directly in your task code instead of `self.logger`.

Note

Lines you emit with `self.logger` are also captured to oryxflow's event stream as `task_log` events during a run — independently of whether `enable_logging()` is on. So logging a decision-relevant scalar (`self.logger.info("corr_avg={}", corr)`) does double duty: it shows in the live log *and* becomes durable, queryable memory (`oryxflow.events.runs(...)`) for a later session that no longer has the original stderr. See [Managing Complex Workflows](https://docs.oryxflow.dev/docs/managing-workflows/#managing-complex-workflows) for the event stream. The [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) makes this a habit: the agent logs its decision-relevant scalars inside every `run()` so the next session inherits them.

## Routing oryxflow logs into your application's logging

By default `enable_logging()` takes over loguru's default stderr handler (see above). If your application configures its own loguru handlers and you want oryxflow's records to flow into *your* sinks rather than have oryxflow touch any handler, pass `sink=None` — it only re-enables the namespace:

```
from loguru import logger
logger.add("app.log")          # your application's own sink

import oryxflow
oryxflow.enable_logging(sink=None)   # re-enable the namespace, add/remove NO handler
# oryxflow records now go wherever your app's loguru sinks point
```

Note: with `sink=None`, if you have your own catch-all sink it will receive oryxflow records at whatever level *that* sink is set to (`enable_logging`'s `level=` only governs the sink it adds, which it does not add in this mode).

## Default log level setting

When you call `enable_logging()` without a `level=`, it uses `oryxflow.settings.log_level` (default `'INFO'`). Set it once to change the default for every later `enable_logging()` call, or pass `level=` to override per call:

```
oryxflow.settings.log_level = 'DEBUG'   # change the global default
oryxflow.enable_logging()               # now defaults to DEBUG
oryxflow.enable_logging(level='INFO')   # per-call override still wins
```

# Sharing Workflows and Outputs

## Introduction

Handing off analysis is usually painful: you zip up a folder of data files and a separate script, and the person on the other end has to figure out which file came from which step and how to regenerate them. Because a oryxflow task bundles the code, its parameters, and its output together, you can share the **whole reproducible pipeline** — the recipient runs the tasks that still need running and loads any result by name, no manual file-shuffling.

Common cases where you want to do this:

- data engineers share cleaned, ready-to-use data with data scientists — without re-sending it every time it changes
- vendors sharing data with clients — as a pipeline the client can re-run, not a one-off dump
- teachers sharing data with students — everyone starts from the same reproducible outputs

oryxflow gives you three approaches, from simplest to most advanced — this page covers them in order:

1. **Share the data folder** — version `data/` with Git LFS (optionally split by `env=` so you hand off only what you mean to). The recipient clones and gets the exact outputs.
1. **Share code-free stubs** — `FlowExport` hands over the outputs as loadable tasks *without* the `run()` code that produced them, for when the logic is private.
1. **Bridge separate flows** — `attach_flow` lets one flow read another flow's outputs at run time, across projects or environments.

## Sharing the data itself: Git LFS

Because oryxflow writes every task output under `data/`, the simplest way to share results is to version that folder alongside your code with [Git LFS](https://git-lfs.com). Then you version and share your data as easily as your code: a teammate clones the repo and gets the exact datasets each run produced, so nobody re-runs the expensive tasks just to obtain outputs someone already computed. This is the recommended approach for most teams.

The [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) sets this up for you in one step with `/oryxflow:init-gitlfs` (it puts `data/` under Git LFS and wires the `.gitattributes`). To do it by hand, install Git LFS and track the data directory:

```
git lfs install
git lfs track "data/**"
git add .gitattributes data/
git commit -m "Track data outputs with Git LFS"
```

Tracking `data/**` also picks up the small `.oryxflow-code-status.json` record oryxflow keeps *inside* the data directory (the code fingerprints behind [Automatic code invalidation](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation)), so it travels with the outputs it describes — move or restore a data directory whole and the freshness information comes along. The separate `.oryxflow/` event log (run history) is high-frequency exhaust, not a shared output: keep it gitignored and export history deliberately if a teammate needs it.

The next section refines this — splitting `data/` by `env=` so you share only part of it. The later Export/Import and Attach sections cover the cases Git LFS doesn't: handing off the *task code* to another project, or reading one flow's outputs from inside another flow.

## Separating environments with `env=`

Often you don't want to share *everything* under `data/` — production outputs, a colleague's scratch experiments, and your own dev runs may all live there, and only some of it is worth handing off. The `env=` argument to `Workflow` keeps them in separate subfolders so you can be selective about what gets shared (Git-LFS-track or commit just the one you mean to).

Passing `env='prod'` writes all task output under `data/env=prod/` instead of `data/`:

```
flow = oryxflow.Workflow(TaskTrain, env='prod')
flow.run()          # output now under data/env=prod/
# a separate dev environment, isolated from prod:
flow_dev = oryxflow.Workflow(TaskTrain, env='dev')
flow_dev.run()      # output under data/env=dev/
```

Because each environment is its own directory, you can share just the one you mean to — commit `data/env=prod/` and leave `data/env=dev/` out — without the environments overwriting each other or leaking. When you later read these outputs (a plain `import` or `FlowImport`, both below), pass the same `env=` to point at the environment you want:

```
flow_prod = oryxflow.Workflow(TaskTrain, env='prod')
df = flow_prod.outputLoad()     # reads data/env=prod/
```

Just make sure the environment you point at is the one whose outputs actually exist — reading `env='prod'` when the shared data was saved without an `env` (plain `data/`) will find the tasks incomplete and rerun them.

## Sharing outputs without the code (`FlowExport`)

The point of exporting is to **share output data without sharing the code that produced it.** Often the `run()` logic is the sensitive part — a proprietary model, a paid data source, an internal cleaning routine — but the *output* is what a colleague, client, or student actually needs. `FlowExport` lets you hand over the results as a first-class oryxflow flow while keeping that logic private.

It works because the exported file contains only **stub** task definitions. For each task it emits the parent class (so the output format is known), the `persists` names, the `path`, the `task_group`, and the parameters — everything needed to *locate and load* the output — plus `external=True`, which tells oryxflow to treat the output as already-produced and never run the task. The original `run()` body is **not** included. The recipient wires the stubs into a `Workflow` (see *Loading shared outputs in your project* below) and calls `outputLoad()` to work with your results through oryxflow — parameter management and all — without ever seeing how they were made.

You can Export your tasks into a new File or print the tasks in the console. All parameters, paths, task_group will be exported.

```
class Task1(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(...)   # your private logic — NOT exported

@oryxflow.requires(Task1)
class Task2(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(...)   # your private logic — NOT exported

flow = oryxflow.Workflow(Task2)

# This will only export Task 2 to console
e = oryxflow.FlowExport(tasks=Task2())
e.generate()

# This will export All the flow (Task1, Task2) to a file
e = oryxflow.FlowExport(flows=flow, save=True, path_export='tasks_export.py')
e.generate()
```

The generated `tasks_export.py` holds stubs like this — note there is no `run()`:

```
import oryxflow
import datetime

class Task1(oryxflow.tasks.TaskPqPandas):
    external=True
    persists=['data']

class Task2(oryxflow.tasks.TaskPqPandas):
    external=True
    persists=['data']
```

Ship this file together with the `data/` directory (the Git-LFS approach above is an easy way), and the recipient can load every output through oryxflow without the source code.

## Loading shared outputs in your project

The simplest way to use `tasks_export.py` is to drop it into your project next to the shared `data/` directory and `import` it like any other module. The stubs are ordinary task classes, so you use them in a standard `Workflow` — point oryxflow at the data directory, then load:

```
import oryxflow
import tasks_export                     # the stub file you were given

oryxflow.set_dir('data/')               # the shared data directory

flow = oryxflow.Workflow(tasks_export.Task2)
flow.complete()                         # True — external stub sees the existing output file
df = flow.outputLoad()                  # load the results, no producer code needed
```

Because the stubs are `external=True`, oryxflow treats their output as already-produced: nothing runs, `complete()` is `True` as long as the output files are in place, and `outputLoad()` just reads them. If the tasks have parameters, pass them as usual to select which output you want (`oryxflow.Workflow(tasks_export.Task2, {'country': 'US'})`) — the same parameter → path resolution applies, so you never chase file paths by hand.

## Loading shared outputs from another project (`FlowImport`)

The plain `import` above assumes `tasks_export.py` and its `data/` sit inside your project. When the file and data live in **another directory or project**, `FlowImport` loads the module and resolves its data path for you, so you don't have to copy anything in:

```
scraper = oryxflow.FlowImport(path='../another-project/', module='tasks_export.py', path_data='data/')
flow_import = oryxflow.Workflow(scraper.tasks.Task2, path=scraper.dirpath)
df = flow_import.outputLoad()
```

`FlowImport` returns an object exposing the imported task classes under `.tasks` and the resolved data directory under `.dirpath`; you pass that `dirpath` as the workflow's `path` so the flow reads from the other project's `data/`. It's the same idea as the plain import — just without moving files between projects.

## Reading another flow's outputs (`attach_flow`)

In more complex projects, users need to import data from many sources. Flows can be attached together in order to access the data generated in one flow inside the other.

Attach a flow to a workflow with `attach_flow(flow, name)`; every task in that workflow then sees it under `self.flows[name]`, so a task's `run()` can load another flow's output without wiring an explicit `requires()` dependency.

### `requires` vs `attach_flow`: the defining difference

The two mechanisms look similar — both let one task read another's output — but they solve different problems, and the difference comes down to two things: **whose data root** the read uses, and **whether oryxflow tracks it**.

A normal `@oryxflow.requires` edge assumes both tasks live in **one flow with one data root**. When you build a `Workflow`, it pushes its `path` (and `env`) down onto *every* upstream task instance — that is how the whole DAG agrees on where `data/` is. oryxflow then *tracks* that edge: if the upstream output is missing, it builds it.

`attach_flow` is the opposite on both counts:

- the attached flow is a **self-contained handle that keeps its own** `path`/`env`, so `self.flows['x'].outputLoad()` reads from *that* flow's data location — which can be a different project's `data/` or a different `env=`;
- it is **not tracked as a dependency** — oryxflow will not build the source flow for you.

So the mental split is: `requires` = "this is part of my pipeline, same data root, build it if needed"; `attach_flow` = "reach into a *separate* flow that has its own lifecycle and data root, and read what it already produced."

### Why not just load it yourself?

The obvious alternative is to load the other flow's output in your driver script and pass the DataFrame into your task. **Don't** — it throws away the one thing oryxflow is for. In oryxflow an output's location is a deterministic function of *(task, parameters)*: you never type a path, you ask for a result and the engine computes where it lives. Hand-loading reintroduces exactly the path bookkeeping the library removes, and it gets worse the moment the source flow is parameterized:

- a source with **no parameters** — you already have to know or hardcode one path;
- a **parameterized** source — now there is a *path per parameter combination*, and to pass the right DataFrame you would have to reconstruct the source's `task_id` → path mapping yourself for every parameter set, and keep it in sync as parameters change. That is reimplementing oryxflow's parameter management by hand, in glue code.

`attach_flow` avoids all of this. Because you attach the flow **object**, not its data, the "ask by parameters, not paths" property survives across the flow boundary — the attached flow resolves its own output paths from its own parameters:

```
self.flows['scraped'].outputLoad(Task2)                 # single flow: its parameters
self.flows['scraped'].outputLoad(Task2, flow='2024')    # WorkflowMulti: pick a parameter set by name
```

You never touch a path, parameterized or not, cross-project or not.

```
class Task1(oryxflow.tasks.TaskCachePandas):
    def run(self):
        self.save(pd.DataFrame({'a': [1, 2]}))

class Task3(oryxflow.tasks.TaskCachePandas):
    def run(self):
        # read the attached flow's default-task output
        temp_flow_df = self.flows['flow'].outputLoad()
        self.save(temp_flow_df)

# Define both flows and run the source flow first so its output exists
flow = oryxflow.Workflow(Task1)
flow2 = oryxflow.Workflow(Task3)
flow.run()

# Attach the first flow to the second under the name 'flow', THEN run
# (attachment is propagated to the tasks when flow2.run() executes)
flow2.attach_flow(flow, 'flow')
flow2.run()

df = flow2.outputLoad()   # Task3's output, sourced from the attached flow
```

Step by step, here is what actually happens:

1. `flow.run()` executes `Task1` and saves its output (here into the in-memory cache, because `Task1` is a `TaskCachePandas`). At this point `flow` is a live `Workflow` object that knows how to load that output — `flow.outputLoad()` would return the DataFrame.
1. `flow2.attach_flow(flow, 'flow')` records the whole `flow` *object* — not its data — under the key `'flow'` on `flow2`. Nothing runs yet; you are just registering that `flow2` may need to reach into `flow` later.
1. `flow2.run()` propagates that registration onto the task instances just before executing them: every task in `flow2` gets `self.flows = {'flow': flow}`. This is why the attach must come *before* the run — a task created and run without it would have `self.flows` empty.
1. Inside `Task3.run()`, `self.flows['flow']` is the attached `flow` object, so `self.flows['flow'].outputLoad()` calls `outputLoad()` on it and returns `Task1`'s DataFrame. `Task3` then saves it as its own output.

The key idea is that the link is between *flows*, resolved lazily at load time — `Task3` never declares `Task1` in its `requires()`. That keeps the two flows independent (separate projects, separate `data/` directories, separate run/reset scopes) while still letting one read the other's results. Crucially, unlike a `requires` edge, the attached flow **keeps its own** `path`/`env`, so `self.flows['flow'].outputLoad()` reads from *that* flow's data root — which may be a different project's `data/` or a different `env=`. That is what makes it the right tool for the cross-project case above: import another project's tasks, wrap them in a `Workflow` pointed at that project's `dirpath`, attach it, and read its outputs by parameter without ever resolving a path.

```
# another project's flow, pointed at ITS data root (see FlowImport above)
scraper = oryxflow.FlowImport(path='../another-project/', module='tasks_export.py', path_data='data/')
flow_prod = oryxflow.Workflow(scraper.tasks.Task2, path=scraper.dirpath, env='prod')

# your flow whose task consumes the scraped data
flow_mine = oryxflow.Workflow(MyTask)
flow_mine.attach_flow(flow_prod, 'scraped')
flow_mine.run()          # inside MyTask.run(): self.flows['scraped'].outputLoad(Task2)
```

Why not just `@oryxflow.requires(scraper.tasks.Task2)` on `MyTask`? Because `flow_mine` would push *your* `path` onto the imported task, so it would look for the output under *your* `data/` instead of `../another-project/data/` — find it missing, and try to rebuild it (needing that project's raw inputs and code). `attach_flow` sidesteps that precisely because the attached flow retains its own path.

Two trade-offs to know:

- **Not tracked as a dependency.** oryxflow will not build the source flow for you: if the output doesn't exist yet, `self.flows['flow'].outputLoad()` raises rather than running it, so you must run the source flow first (as `flow.run()` does above).
- **Attached at a fixed parameter slice.** `attach_flow` hands you the source flow configured at attach time — clean when the consumer wants a *fixed* slice ("the prod-2024 scrape"). If instead you want *each instance* of a parameterized consumer to automatically pull the **matching** parameter set from the source (consumer `country=US` → source `country=US`), that per-instance coupling is what a real `@oryxflow.requires` edge does best — parameters propagate upstream automatically and it is tracked and auto-built.

Decision rule:

- **Same project, and the upstream should follow the consumer's parameters** → `@oryxflow.requires` (parameters propagate, tracked, auto-built). Don't reach for `attach_flow` to wire tasks that belong to *one* pipeline.
- **Separate data root (another project or another** `env` **), or an independently-managed flow** → `attach_flow`, which keeps parameter → path resolution across the boundary — the thing manual DataFrame passing destroys.

# Managing Complex Workflows

As a project grows, the expensive part is rarely the final result — it's the *granular* work underneath it. Fetching data from a slow or paid API. Training a model once per hyperparameter setting. Re-running that work every time you tweak something downstream is what makes iteration painful.

oryxflow's answer is to make each granular unit its own **task**, cached independently. The engine tracks completeness per task, so when you change one thing it re-runs exactly what changed — and leaves the expensive, unchanged work alone. This page walks through the pattern data scientists reach for constantly:

1. **Iterate granular** — one task per item (per hyperparameter, per state, per input file), each cached on its own.
1. **Aggregate** — combine those outputs one level up into a single result.
1. **Selectively reset** — when something the engine *can't* see changes (a data file, an external API), invalidate just that family of tasks and let everything downstream recompute. The expensive leaves stay cached. (Parameter and code changes need no reset — they rerun on their own; see below.)

## Worked example: hyperparameter tuning

The classic case: load a dataset once (expensive), train a model once per hyperparameter value (the granular loop), then aggregate the scores to pick a winner.

```
import oryxflow
import pandas as pd

ALPHAS = [0.01, 0.1, 1.0, 10.0]          # the grid you iterate over — your own domain data

class LoadData(oryxflow.tasks.TaskCachePandas):
    """Expensive: a slow/paid data pull you never want to repeat needlessly."""
    def run(self):
        self.save(load_training_data())   # imagine a long API call

@oryxflow.requires(LoadData)              # wires requires(), copies params
class TrainModel(oryxflow.tasks.TaskCachePandas):
    """One granular task per hyperparameter value. Cached independently."""
    alpha = oryxflow.FloatParameter()
    def run(self):
        df = self.inputLoad()             # the shared dataset, loaded once
        score = fit_and_score(df, alpha=self.alpha)
        self.save(pd.DataFrame({'alpha': [self.alpha], 'score': [score]}))

class Tune(oryxflow.tasks.TaskCachePandas):
    """Aggregate: stack every trained model's score into one frame."""
    def requires(self):
        return {a: TrainModel(alpha=a) for a in ALPHAS}
    def run(self):
        df = self.inputLoadConcat()       # one row per alpha; 'alpha' column tags each
        self.save(df.sort_values('score', ascending=False))
```

Run it and read off the best hyperparameter:

```
flow = oryxflow.Workflow(Tune)
flow.run()
results = flow.outputLoad()               # every alpha's score, one frame
best_alpha = results.iloc[0]['alpha']
```

The `requires()` dict is what fans the DAG out into one `TrainModel` per `alpha`, and `self.inputLoadConcat()` stacks their outputs, tagging each row with that task's parameters so your `alpha` column survives the concat. (The mechanics of dict-`requires()` and `inputLoadConcat` are covered as reference in [Advanced: Dynamic Tasks](https://docs.oryxflow.dev/docs/advtasksdyn/index.md).)

## Extend the grid — nothing wasted

Decide you want to try more values? Add them to `ALPHAS` and run again:

```
ALPHAS = [0.01, 0.1, 1.0, 10.0, 100.0]    # added 100.0
flow.run()
```

Only `TrainModel(alpha=100.0)` runs — the four you already trained are complete and are skipped, and the expensive `LoadData` is not touched at all. `Tune` recomputes because it now has a new input. You didn't reset anything; adding a parameter value is enough.

## What reruns, and when you do nothing

A task is "complete" when its output exists **for its current parameters and its current code**. Two of the three things that can make a result stale are handled for you, no action required:

- **You changed a parameter.** The task id is derived from the parameters, so a new value is a new id — no output yet, the task runs, and the old output for the old parameters is left untouched beside it. Add a value to the grid (a new `alpha`) and *only* that new task runs.
- **You changed the code.** oryxflow fingerprints each task's logic (next section). Edit a task's `run()` — or a helper it calls — and the task plus everything downstream recomputes on the next run.

The third — a change the engine *can't* see, like a new data file or an API response — is the only one you drive by hand, with `reset()` (see [Selectively reset when code versioning doesn't apply](https://docs.oryxflow.dev/docs/managing-workflows/#selectively-reset-when-code-versioning-doesnt-apply) below). Because completeness is checked recursively down the graph, an aggregator recomputes automatically whenever any of its inputs changes or a new one appears, with no bookkeeping on your part.

## Automatic code invalidation

Suppose you improve `TrainModel` — add a second scoring metric to its `run()`. The parameters didn't change, so a parameters-only cache would happily serve the stale output. oryxflow instead **fingerprints the code**: on the next run every `TrainModel` recomputes, and so does everything downstream of it (`Tune`), overwriting in place at the same paths. `LoadData` is untouched. You did nothing — no reset, no version to bump.

The fingerprint covers each task's own class **and every project-local symbol it references**, followed transitively across your modules — so editing a *helper function* the task calls is caught too, where the change actually lives, not just in `run()`. Two properties make this safe to leave on:

- **Cosmetic edits never recompute.** oryxflow compares what your code *does*, not how it's written, so comments, docstrings, and formatting changes never trigger a rerun.
- **Granularity is per symbol, not per file.** One monolithic `tasks.py` is fine: editing one task's `run()` reruns that task alone, and editing a shared helper recomputes exactly the tasks that reference it (directly or through other helpers). The rerun reason even names the changed symbol — `code change (auto: tasks.py::TrainModel)`.

Referencing another *task* in `requires()` is dependency wiring, not a code reference — it never pulls that task's body into your fingerprint; the dependency cascade carries that staleness instead. What symbol analysis can't split apart (module-level side effects, star imports, dynamically created classes) falls back to whole-file granularity — extra reruns at worst, never a missed one.

**Expensive tasks warn instead of silently recomputing.** A rerun overwrites the old output, so burning a long computation should be a decision, not a side effect of a refactor. A task whose last run took longer than `settings.code_version_auto_expensive_s` (default 600 seconds) is held complete when its code changes, and the run **warns** instead — answer it with one of the [three exits](https://docs.oryxflow.dev/docs/managing-workflows/#the-three-exits-for-any-code-change) (reset, `accept_code`, or pin it with `code_version`). Set the threshold to `None` to make every code change recompute.

To turn automatic tracking off entirely — parameters-only identity plus explicit pins — set `oryxflow.settings.code_version_auto = False`.

## When you changed something but nothing reran

The fingerprint follows Python `import` statements under your project root. It **cannot** see: data-file contents, external APIs, installed packages, dynamic imports or monkeypatching, and tasks defined in a notebook or REPL (which aren't hashable at all). A change in any of those is invisible to the hash, so the task keeps its cache.

The symptom is a **skip you didn't expect**: you changed something, ran, and the summary says `0 ran`. Make this one check a habit — after any change you expect to recompute, confirm the affected tasks appear in `result.ran` (or `oryxflow.events.runs()`) with a matching reason such as `code change (auto: pipeline/train.py::TrainModel)`:

- `ran=0` **after a change** means the change is in a blind spot. `reset()` the task that ingests it, or give that task an explicit `code_version` if it happens repeatedly.
- `ran=0` **on an untouched pipeline** is the healthy "cache is trusted" signal.

A blind spot never produces a *false* rerun or a false "verified unchanged" — it just degrades to parameters-only identity, the same trust level every cache file has always had.

## Pinning a task: the explicit `code_version`

Sometimes automatic is the wrong sensitivity — a training task so expensive that a refactor-triggered recompute must be a deliberate decision, or logic the hash can't see (behavior driven by a config file, dynamic dispatch). Declare an explicit `code_version` and that task's own logic is **pinned**: it recomputes only when you bump the token, and a code edit *without* a bump produces a staleness warning instead of a rerun.

```
@oryxflow.requires(LoadData)
class TrainModel(oryxflow.tasks.TaskCachePandas):
    code_version = 2          # pinned: bump deliberately to recompute
    alpha = oryxflow.FloatParameter()
    def run(self):
        ...
```

The token can be an int or a string (`'v2-log-features'`). Pins are **free to toggle**: the `code_version` line is itself stripped by AST normalization (like a comment), so adding, removing, or bumping it is never itself a source change. Under the hood oryxflow stores both the token and the code hash at each run and compares whichever matches the current mode — which is what gives the toggle these useful properties:

- Pinning a task whose code is unchanged costs nothing — no recompute.
- Pinning *in the same edit as a logic change* still recomputes: the hash catches what the pin would otherwise have blessed.
- Unpinning just resumes automatic tracking; if you edited while pinned and never bumped, that one masked edit is caught and recomputes once.
- Toggling never ripples downstream — dependents key on whether an upstream actually *rematerialized*, not on which mode tracked it.

Pinned and automatic tasks mix freely in one pipeline, and a pinned task still reruns when its *upstream* recomputes — the pin covers its own logic, not its inputs. Why pin at all, if automatic needs nothing? The token makes the cache decision — "this logic changed, downstream must recompute" — an explicit, diffable part of the same edit, visible in code review and `git log`. That's why AI-agent projects often prefer pins on their key tasks.

Note

**One first-time trap.** Bringing an output under tracking for the *first* time — a pre-upgrade artifact, or the first pin in a `code_version_auto = False` project — in the *same edit* that changes its logic can bless the stale output, because there is no prior record to compare against. oryxflow warns when it can detect this (`output predates current code`); answer it with `reset()` to recompute, or `flow.accept_code()` if the outputs really are current. Once a task has one record on disk, the trap is closed for good — the stored hashes expose any edit no matter how you toggle the pin.

## The three exits for any code change

Every code change — whether it auto-recomputed or a pin warned — has the same three exits, and they are **not** equal in risk:

- **Recompute** — for automatic tasks, just run (it already happens); for pinned tasks, bump `code_version`. Safe by default.
- **Reset** — `flow.reset(...)` recomputes regardless, with no version bookkeeping. Also safe.
- **Accept** — the change is output-equivalent (a rename, an extracted function, an added log line): `flow.accept_code()` (or `oryxflow.accept_code(task)`) re-stamps the current code as accepted *without* rerunning. For an automatic task this is how you *skip* a recompute it would otherwise do.

**Accept is the one exit that can silently bless a stale output** — use it only when you're certain; when unsure, recompute (cheap insurance; a wrongly-blessed cache is not). It stamps the task **and its whole upstream tree**, so call it on the most-downstream task you judge equivalent. It prints a one-line summary of what it re-stamped; "nothing accepted" means it didn't reach the target. A bare `flow.accept_code()` covers your whole pipeline — every task it can compute, whichever final it hangs from — so one call blesses a multi-final pipeline, and it works from a fresh script (a one-shot bless after an upgrade needs no prior run). You can also name the tasks yourself, one or several: `flow.accept_code([FinalA, FinalB])`. Prefer the instance/`flow` form over the bare/class form: it is the only one that can stamp a baseline for record-less outputs, and anything the other forms miss simply recomputes (the safe direction). On a `WorkflowMulti` use `flow.accept_code()` (all flows, or one with `flow=...`) — the module-level form doesn't know your flows' parameters. Tasks a flow reaches only *dynamically* (yielded inside a `run()`) aren't in the static tree; accept those with an explicit instance if they warn.

Note

The staleness warning for a pinned task prints without `enable_logging()`:

```
StalenessWarning: task TrainModel: pipeline/train.py::TrainModel changed since cached run;
code_version still 2 -- reusing cached output. Bump code_version to recompute, or
oryxflow.accept_code(TrainModel) only if certain the output is equivalent.
```

It dedupes per process — one line per distinct message, however many flows or parameter values hit it — and re-arms when the condition changes or the tasks rerun. `result.warnings` lists each distinct warning once, so its length answers "how many pending warnings do I have?" no matter how many flows or parameter values are involved. `oryxflow.events.warnings()` still records every occurrence, so even if your environment suppresses the print the record is never lost, and a strict `-W error` setup won't turn the advisory into a failed build.

## Which verb, when

| I changed…                                                                                 | Do this                                                         | Why                                                                                                                           |
| ------------------------------------------------------------------------------------------ | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| a value/knob that is a `Parameter`                                                         | nothing                                                         | new identity auto-reruns; old output kept side by side                                                                        |
| logic (a task's `run()` or a helper it imports), output will differ                        | nothing — just run                                              | the code fingerprint moved; the task and everything downstream recompute                                                      |
| logic of a **pinned** task (one that declares `code_version`)                              | bump `code_version`                                             | the pin is the authority; without a bump you get a warning, not a rerun                                                       |
| logic of an **expensive** task (last run > `code_version_auto_expensive_s`)                | answer the warning: `reset()` / `accept_code` / pin             | the guard holds it cached so a refactor can't silently burn a long run                                                        |
| code, but output is provably identical (rename, extract, log line)                         | `flow.accept_code()` — only if certain; when unsure, recompute  | re-stamps without recompute; the one non-recomputing exit                                                                     |
| nothing oryxflow can detect, but I need a fresh compute (data file changed, suspect cache) | `reset()`                                                       | the change is outside the code oryxflow tracks; force it at the task that ingests the change                                  |
| something, but the run skipped it (`0 ran` after an edit)                                  | `reset()` — or pin that task with `code_version`                | the change is somewhere oryxflow can't detect; treat the skip as a signal, not a convenience                                  |
| I want the outputs gone                                                                    | `reset()`                                                       | delete                                                                                                                        |
| pin added or removed (code untouched)                                                      | nothing                                                         | the record carries both dimensions; a pure mode flip never recomputes or ripples                                              |
| first time bringing an **untracked** output under tracking right after editing it          | `reset()` once                                                  | no record exists yet, so grandfathering would bless the stale output (once a record exists this trap is caught automatically) |
| nothing — but pre-existing outputs warn `output predates current code`                     | `flow.accept_code()` if the outputs are current, else `reset()` | the guard can't tell a fresh checkout from a stale cache; accepting stamps record-less outputs a baseline in one call         |

## Keeping old versions side by side

By default a bump overwrites in place. To keep previous versions at readable paths — the compare-two-versions workflow — set `keep_versions = True` on the task (ideally with a string version):

```
class TrainModel(oryxflow.tasks.TaskCachePandas):
    code_version = 'v2-log-features'
    keep_versions = True     # outputs live at data/TrainModel/v2-log-features/...
```

Bumping then writes to a new `v.../` directory and the old one stays on disk. `keep_versions` keys off the explicit token only — automatically-tracked tasks (no `code_version`) always overwrite in place. Note that *turning on* `keep_versions` for an existing task relocates its output path (it gains the version segment), so the task recomputes once and the old non-versioned artifact is left behind — it's not a transparent toggle.

## Selectively reset when code versioning doesn't apply

`reset` remains the right tool when there's no code token to move: a data file changed, you suspect a corrupt cache, or you simply want outputs deleted. The point is to reset **only** the family that changed, so the shared `LoadData` stays cached:

```
flow = oryxflow.Workflow(Tune)
flow.reset_upstream(Tune, only=TrainModel)   # every TrainModel in the DAG, nothing else
flow.run()
```

`reset_upstream(Tune, only=TrainModel)` walks the whole graph upstream of `Tune` and invalidates just the `TrainModel` instances — every `alpha` at once, found via the DAG so you never hand-list them. `LoadData` is a *different* family and is left complete, so the slow data pull does **not** repeat. On the next `run()` all four models retrain and `Tune` recomputes on top (recursive completeness), while `LoadData` is served from cache.

Contrast the three reset scopes you'll actually use:

- `flow.reset(TrainModel(alpha=0.1))` — one specific instance. Use when you're debugging a single case.
- `flow.reset_upstream(Tune, only=TrainModel)` — one *family*, everywhere it appears upstream. Use when something the code hash can't see changed for that family and you want every instance recomputed, cheap dependencies preserved.
- `flow.reset_upstream(Tune)` — the whole upstream, including `LoadData`. Use only when the raw inputs themselves are stale.

This is the core discipline: **reset at the level you changed, not above it.** It's what keeps a tweak to a fast step from triggering hours of expensive re-fetching or re-computation.

## The event stream: what ran, when, and why

Every run appends structured events to a plain-text JSONL stream — always on, written asynchronously (microseconds), so the record exists even when your script discards the run result. The file layout is a contract you can script against:

- current month: `.oryxflow/events.jsonl` (a stable head — `tail -30` always shows the latest)
- offloaded months: `.oryxflow/events-YYYYMM.jsonl` (immutable once offloaded)
- all history: glob `.oryxflow/events*.jsonl`

Events include `run_started` / `task_ran` / `task_failed` / `run_finished` / `code_warning` / `code_accepted` / `task_log`. Each `task_ran` carries the full recipe — params, `code_version`, code fingerprint, source hashes, git SHA, duration — and the **reason** it ran (`output missing` / `code change (auto: pipeline/train.py)` / `code change (1 -> 2)` / `upstream rerun`), so "why did this recompute?" and "was this produced by current code?" are queries, not guesses. Anything a task logs via `self.logger` is captured as a `task_log` event — log your decision-relevant scalars (`self.logger.info("corr_avg={}", corr)`) and they become next session's memory.

```
oryxflow.events.print_status()   # session-start orientation, printed: pending code
                                 # warnings, last run per family, recent failures
oryxflow.events.status()         # the same as data (a dict) -- it returns, doesn't print
oryxflow.events.runs(task_family='TrainModel', last=2)   # diff params/code_version/hashes
oryxflow.events.runs(flow='Retail')                      # per-flow when using WorkflowMulti
```

From the shell, no Python needed:

```
tail -30 .oryxflow/events.jsonl
grep TrainModel .oryxflow/events*.jsonl
jq 'select(.type=="task_ran") | {task_id, reason, duration_s}' .oryxflow/events.jsonl
```

`run()` returns the same story in memory: `result.reasons` maps each task that ran to why, and `result.warnings` lists unacknowledged code changes.

Add `.oryxflow/` to your `.gitignore` — run records are high-frequency exhaust, not source. Disable entirely with `oryxflow.settings.events = False`.

## Where the freshness records live

The code fingerprints live in one small JSON file per data directory (`<dirpath>/.oryxflow-code-status.json`). It describes those exact artifacts, so **move or restore a data directory whole** — file and artifacts together, always via the same channel. An artifact without a record is simply treated as current (the same trust level every file has today); partial restores are on you. Correctness depends only on this file, never on the event log.

## What caching does *not* protect against

Everything above solves one family of problems — the *mechanical* one: staleness, provenance, and memory. That family genuinely yields to tooling, and a cached result you've kept fresh is worth trusting *as a computation*. But be clear about the boundary: oryxflow guarantees a result was produced by the code and inputs it records — **not** that the result is correct. A perfectly versioned, fully reproducible pipeline can still hand you a wrong number, and nothing here will flag it:

- a join that should be many-to-one runs many-to-many and inflates every downstream aggregate;
- a percentage computed against the wrong denominator, or a `groupby` that silently drops null-keyed rows;
- a dtype coercion that eats a ZIP code's leading zero, or a timezone shift that moves rows a day;
- a backtest that peeks at the future, or a correlation read off overlapping windows as if the points were independent;
- a number quoted from memory or eyeballed off a chart instead of from the saved artifact.

None of these raise; each completes cleanly and prints something plausible. Versioning the code just makes the wrong join *reproducible*. These are caught by **habit, not machinery** — validate the merge and assert the row relationship, look at the frame's shape and null counts before stating a finding, quote every number from an artifact you can re-open. (Note this is a different thing from the hash blind spots above: a changed data file is invisible but has a verb — `reset` the loader; a wrong join has no verb, only vigilance.) That is why the [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) ships those conventions alongside the library, delivered at the point you're writing the task rather than in a review afterward. And the judgment calls — is this method right for the question, is this effect within noise, does the data actually behave the way I assumed — are yours; no cache decides them.

## CLAUDE.md snippet for AI-agent projects

For projects driven by AI coding agents, the recommended setup is the [oryxflow Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) — it ships these rules (and more: project scaffolding, task templates, conventions) as a skill that loads automatically, stays current with the library, and needs no per-project copy. If you can't use the plugin (a different agent, a locked-down environment), paste this snapshot of the rules into the project's `CLAUDE.md`:

```
## oryxflow cache & provenance rules

1. Session start / after `/clear`: call `oryxflow.events.print_status()` — pending code
   warnings, last run per task family, recent failures — before assuming anything about
   cache state (`events.status()` returns the same as a dict for filtering; it prints
   nothing). No-Python fallback: `tail -30 .oryxflow/events.jsonl`.
2. Editing task or helper logic needs NO cache action — code invalidation is automatic
   (AST-hash of the task's own class + the repo-local symbols it references, transitively;
   comments/formatting never count, and editing an unrelated task in the same file never
   reruns its siblings — one monolithic tasks.py is fine) and propagates downstream. Do not
   hand-chain `reset()` calls for code changes.
   Expensive tasks (last run > `settings.code_version_auto_expensive_s`, default 600s)
   warn instead of silently recomputing — answer with reset / accept_code / pin.
   Exception: a task that declares `code_version` is PINNED — automatic tracking of its own
   logic is off, so bump its token **in the same edit**. Pins toggle freely: adding or
   removing one on unchanged code never recomputes (and never ripples downstream); an edit
   masked while pinned is caught the moment the pin comes off.
3. **Verify the rerun happened.** After any code edit, the next run must show the edited
   band in `result.ran` (or `oryxflow.events.runs()`) with a matching reason —
   `code change (auto: <file>::<symbol>)` or `code change (1 -> 2)`. `ran=0` after an edit means the
   hash didn't see the change (data file, installed package, dynamic call, notebook-defined
   task): `reset()` the affected task, or pin it with `code_version` if it recurs. `ran=0`
   on an untouched pipeline is the healthy signal.
4. Expensive recompute you judge output-equivalent (pure refactor): `flow.accept_code()` /
   `oryxflow.accept_code(anchor_task)` re-stamps the task and its whole upstream tree
   without rerunning — only if certain; when unsure, let it rerun. `preview()` first to see
   the pending band. A bare `flow.accept_code()` covers the whole pipeline — every task
   the flow can compute, multi-final pipelines included, from a fresh process; a list
   also works (`flow.accept_code([FinalA, FinalB])`). It prints what it re-stamped — "nothing
   accepted" means it didn't reach the target (use the instance/flow form, not the
   class/bare form). An `output predates current code` warning (outputs with no record
   yet, e.g. after an upgrade) is answered the same way: `flow.accept_code()` if the
   outputs are current, `reset()` if not. On WorkflowMulti use `flow.accept_code()`
   (all flows) — the module-level bulk form doesn't know the flows' parameters. Answer
   every staleness warning with one of its exits — bump, accept, or reset. Never leave one
   firing across runs.
5. After a run, read the returned result: `result.reasons` says why each task ran;
   `result.warnings` lists unacknowledged code changes (each distinct warning once, so its
   length is the pending count). Never hand-roll aggregation —
   `MultiRunResult` exposes `.ran`/`.complete`/`.failed`/`.reasons`/`.warnings` across
   flows, and the per-build verdict is logged durably as `run_finished` events.
6. "The numbers changed and I don't know why": compare the last two runs —
   `oryxflow.events.runs(task_family='TaskX', last=2)` — and diff params, code_version,
   source_hashes.
7. Log decision-relevant scalars inside `run()` via `self.logger.info(...)` — they're
   captured as `task_log` events and become next session's memory.
8. Experiments you want side by side: string `code_version` plus `keep_versions = True`
   (explicit token only — auto-tracked tasks overwrite in place).
9. Raw stream: current = `.oryxflow/events.jsonl`; offloaded months = `events-YYYYMM.jsonl`;
   all history = glob `events*.jsonl`. Prefer `events.runs()`/`status()` in Python.
10. Data-file or external-API changes are invisible to the code hash — `reset()` the task
    that ingests them (a downstream reset re-loads the cached old input).
```

## Scaling up: hierarchies and independent experiments

The same three moves scale in two directions.

**Deeper hierarchies.** Aggregators compose: a per-state task feeds a per-country task feeds a per-sector task, each level a `requires()` fan-out combined with `inputLoadConcat()`. The whole tree is still one DAG, so preview, the run summary, and selective reset all reach every level.

**Independent experiments.** When you want to manage several runs separately — each with its own output and its own reset scope — drive the top with [WorkflowMulti](https://docs.oryxflow.dev/docs/workflow/index.md) over a params grid instead of one more aggregator, then combine across flows with `outputLoadConcat`:

```
flow = oryxflow.WorkflowMulti(Sector, params={'sector': ['Retail', 'Office']})
flow.run()
dfall = flow.outputLoadConcat(Sector)                # all sectors, one tagged frame
flow.reset_upstream(Sector, only=CountryFeatures)    # reset one family across every flow
```

A complete, runnable version of this multi-level dev loop — iterate on one `(sector, country)` first, then roll the change out to every flow *without re-fetching the expensive per-state source* — is in `docs/example-flow-multi.py`. The full reference for dict-`requires()`, `inputLoadConcat`, `outputLoadConcat`, and the `only=` reset filter is [Advanced: Dynamic Tasks](https://docs.oryxflow.dev/docs/advtasksdyn/index.md).

Tip

This is exactly what the [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) is built to manage. Describe the hierarchy in plain language and it writes the fan-out `requires()` and the `inputLoadConcat()` aggregators; when you iterate, it scopes the reset for you — resetting just the family you changed (`reset_upstream(..., only=...)`) so the expensive leaf tasks are preserved.

# Experiment tracking with oryxflow

**An experiment tracker and oryxflow do different halves of the same project. Using both is the normal setup, not a compromise.**

This is the comparison people most often read as either/or, so to be explicit about the split:

- A **tracker** (MLflow, Weights & Biases, Neptune, Comet) is a *record of results*. It collects metrics, parameters, and artifacts from runs you already did, and gives you a UI to sort and chart them.
- **oryxflow** is the *machinery that produces those runs*. It decides which steps have to execute, reuses the ones that don't, and guarantees the features your model just scored on were built by current code.

A tracker can't tell you a logged run was trained on a stale intermediate. oryxflow can't draw you a leaderboard. Neither is trying to.

## Which question does it answer?

| The question you're asking                                                | The tool that answers it |
| ------------------------------------------------------------------------- | ------------------------ |
| Which run scored 0.91, and with which parameters?                         | **Tracker**              |
| Show me every run's metrics side by side, sorted                          | **Tracker**              |
| Which chart do I paste into the review deck?                              | **Tracker**              |
| Which steps must rerun to reproduce that 0.91 — and are its inputs stale? | **oryxflow**             |
| Was this model trained on features built by the current code?             | **oryxflow**             |
| Can I skip the ten-minute data pull and just retrain?                     | **oryxflow**             |
| Where is the output for `window=60` — did I already compute it?           | **oryxflow**             |

Read the table one way and it's a feature comparison. Read it the other way and it's a division of labor: the left column is your *record*, the right column is your *computation*.

## What each tool cannot do for you

Being honest in both directions is the whole point of this page.

**A tracker records what you ran — it doesn't decide what to run.** It will faithfully log that a run scored 0.91 without knowing whether the features feeding it were regenerated after your last edit. It won't skip the expensive step that didn't change, and it won't rerun the steps a code change actually affected. Those are properties of the computation, not of the log.

**oryxflow has no dashboard.** No hosted UI, no run-comparison view, no team-wide leaderboard, no model registry, no shareable report link. If those are what you're missing, you want a tracker — see the [MLflow alternatives roundup](https://docs.oryxflow.dev/blog/comparisons/mlflow-alternatives/index.md) for which one.

**And one boundary that applies to oryxflow specifically:** it makes your analysis *reproducible*, not *correct*. It guarantees a result came from the code and inputs it records, and that stale steps get rerun. It does not check that your join grain was right or your denominator sensible. That last mile is still your review.

## The integration pattern: log from inside the task

The whole integration is one idea — **the tracker call lives inside a task's `run()`**, so every logged run is also a cached, reproducible one:

```
import mlflow
import oryxflow

@oryxflow.requires(BuildFeatures)
class Train(oryxflow.tasks.TaskPickle):
    model = oryxflow.ChoiceParameter(default='rf', choices=['rf', 'gbm'])

    def run(self):
        features = self.inputLoad()                    # upstream output, already cached
        clf, auc = fit_and_score(features, kind=self.model)

        mlflow.log_params(self.to_str_params())        # this task's parameters, as a dict
        mlflow.log_metric('auc', auc)
        mlflow.set_tag('oryxflow_task_id', self.task_id)

        self.save(clf)                                 # oryxflow: caches + invalidates
        self.saveMeta({'auc': auc})                    # the score, kept next to the model
```

Three small things are doing real work here:

- **`self.to_str_params()`** hands you every parameter of the task as a `{name: value}` dict, so the tracker's parameter list can't drift from what the task was actually run with — you don't maintain a second copy of the parameter names.
- **`self.task_id`** identifies the exact cached output this logged run produced. Tag the run with it and a row in the tracker points back at a file you can reopen, months later, without guessing.
- **`self.saveMeta(...)`** keeps the score beside the model in oryxflow's own cache, so a script can read scores back with `flow.outputLoadMeta()` without going through the tracker's API.

Swap `mlflow` for `wandb`, `neptune`, or `comet_ml` — the shape doesn't change. oryxflow has no tracker integration to configure, because none is needed: it's your code, in your `run()`.

## Sweeping parameters: one comparable row per configuration

This is where the pairing pays off most visibly. Sweep a model parameter and oryxflow reruns only the affected tasks — the shared feature build happens **once** — while the tracker fills up with one comparable row per configuration.

Drive the sweep with `WorkflowMulti`, one flow per configuration:

```
flow = oryxflow.WorkflowMulti(Train, params={
    'rf':  {'model': 'rf'},
    'gbm': {'model': 'gbm'},
})

result = flow.run()               # BuildFeatures runs once; Train runs once per model
print(result.ran)                 # exactly what recomputed
print(result.reasons)             # and why each of them did

scores = flow.outputLoadMeta()    # {'rf': {'auc': ...}, 'gbm': {'auc': ...}}
best = flow.outputLoad(flow='gbm')                 # one flow's model, loaded by name
```

Add a third configuration tomorrow and only that one runs; the two you already trained are complete and skipped, and the expensive feature build is not touched at all. Your tracker gains exactly one new row. Nothing was recomputed to produce it, and nothing was recomputed *wrongly* either — if you had edited the feature code in between, every configuration would have retrained on the new features automatically.

When every flow's output is a schema-compatible DataFrame, `flow.outputLoadConcat(Train)` row-stacks them into one frame tagged by each flow's parameters — a local leaderboard for the cases where you don't want to open the UI at all. The full treatment of sweeps, fan-out aggregation, and scoping a rerun to one family of tasks is in [Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md).

## When one tool alone is enough

Most projects want both. Two honest exceptions:

- **A tracker alone is fine** if you run one script end to end, the run is cheap or happens on someone else's infrastructure, and what you want is a leaderboard. There's nothing to cache and no dependency graph to get wrong, so oryxflow would be ceremony.
- **oryxflow alone is fine** if you're iterating locally, you read results out of a DataFrame or a notebook rather than a UI, and the pain you keep hitting is stale intermediates and expensive reruns. That's a computation problem, and a dashboard doesn't fix it. `pip install oryxflow`, no server or account, done.

The failure mode to avoid is picking one tool to solve the *other* one's problem: installing a tracker because your pipeline isn't reproducible, or expecting a caching layer to give you charts.

## Where to read the head-to-heads

This page states the division of labor. The arguments behind it live elsewhere, so you don't get them twice:

- **[Do you need MLflow, or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md)** — the two problems that both get called "experiment management," and how to tell which one is biting you.
- **[MLflow alternatives](https://docs.oryxflow.dev/blog/comparisons/mlflow-alternatives/index.md)** — an honest roundup of the trackers (W&B, Neptune, Comet, ClearML, Aim, DVCLive, SageMaker Experiments) and which job each one is best at.
- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** — the full comparison table across trackers, orchestrators, and DVC.

## Takeaway

- **Different halves, same project.** A tracker owns the record of results; oryxflow owns the computation that produced them. Expect to run both.
- **The integration is one line of placement**: put the tracker call inside a task's `run()`, and every logged run is also cached and reproducible.
- **Sweeps are where it shows.** Vary a parameter and only the affected tasks rerun, while the tracker gains one comparable row per configuration.
- **Neither tool covers the other.** oryxflow won't chart your runs; a tracker won't tell you a run was trained on something stale. And oryxflow makes analysis reproducible, not correct.

```
pip install oryxflow
```

- **[Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)** — a running, self-caching pipeline in minutes.
- **[Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md)** — parameter sweeps, aggregation, and scoping a rerun to just what changed.
- **[Running workflows](https://docs.oryxflow.dev/docs/run/index.md)** — `Workflow`, `WorkflowMulti`, and what reruns when.
- **[Transition from scripts](https://docs.oryxflow.dev/docs/transition/index.md)** — turn an existing training script into tasks.
- **[Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md)** — let an AI agent scaffold the pipeline around your tracker calls.

## Frequently asked questions

**Can I use oryxflow and MLflow together?** Yes, and that's the recommended setup — not a compromise. Put your tracker's logging calls inside an oryxflow task's `run()`, so oryxflow decides what has to recompute and caches the rest, while MLflow (or Weights & Biases, Neptune, Comet) keeps the searchable record of what each run scored. Every row in your tracker then corresponds to a cached, reproducible computation instead of a run you can't rebuild.

**Is oryxflow an MLflow alternative?** Not a replacement — a complement. MLflow records and charts the runs you already did; oryxflow is the machinery that produces them, deciding which steps must execute, reusing the ones that don't, and guaranteeing your model scored on features built by current code. Use both. The one case where oryxflow alone is enough is when what you actually wanted from a tracker was caching and reproducible reruns rather than a dashboard.

**Do I still need an experiment tracker if I use oryxflow?** If you want a hosted dashboard, side-by-side run comparison, or a leaderboard your team can browse, yes — oryxflow does not provide any of those and is not trying to. oryxflow gives you reproducible, minimally-recomputed pipelines underneath whichever tracker you pick. If you iterate locally, read your results from a DataFrame, and never wanted the UI, oryxflow on its own covers what you need.

**How do I log metrics from an oryxflow task?** Call your tracker inside the task's `run()`, next to `self.save()`. Nothing about oryxflow has to change: compute the metric, log it with `mlflow.log_metric(...)` or `wandb.log(...)`, and save the artifact with `self.save(...)`. `self.to_str_params()` hands you the task's parameters as a dict to log in one call, and `self.task_id` identifies the exact cached output the logged run came from.

**What's the difference between pipeline caching and experiment tracking?** Experiment tracking answers which run got which metric with which parameters, and gives you a UI to sort and chart it. Pipeline caching — what oryxflow does — answers which steps must actually rerun and which are already computed, so an unchanged feature build isn't recomputed and a model is never scored on stale inputs. A tracker records what happened; oryxflow makes the computation behind it reproducible and cheap to repeat.
# Advanced

# Advanced: Dynamic Tasks

Sometimes you might not know exactly what other tasks to depend on until runtime. There are several cases of dynamic dependencies.

## Fixed Dynamic

If you have a fixed set parameters, you can make requires() "dynamic".

```
# cfg_params.py -- the enumeration is your own domain data, kept in a config module
PARAMS = ['a', 'b', 'c']

class TaskInput(oryxflow.tasks.TaskPqPandas):
    param = oryxflow.Parameter()
    ...

class TaskYieldFixed(oryxflow.tasks.TaskPqPandas):

    def requires(self):
        return {s: TaskInput(param=s) for s in cfg_params.PARAMS}

    def run(self):
        df = self.inputLoad()
        df = pd.concat(df)
        self.save(df)
```

You could also use this to load an unknown number of files as a starting point for the workflow.

```
def requires(self):
    return {s: TaskInput(param=s) for s in glob.glob('*.csv')}
```

## Hierarchical iterate-and-aggregate

Note

This section is the mechanics reference. For *why* and *when* you'd reach for this pattern — caching expensive granular work and resetting it selectively as you iterate — start with [Managing Complex Workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md).

A common pattern is to iterate over some dimension (e.g. per-state tasks), then aggregate the results one level up (e.g. a per-country task that combines all of its states). Do this with a **native DAG aggregator**: the aggregating task's `requires()` returns a dict of the per-item task instances, and `run()` stacks them with `self.inputLoadConcat()`. Each dependency's significant params are added as columns automatically, so your groupby keys (state, country) survive the concat.

```
STATES = {'US': ['CT', 'NY'], 'UK': ['London', 'Belfast']}

class DataLoadState(oryxflow.tasks.TaskPqPandas):
    country = oryxflow.Parameter()
    state = oryxflow.Parameter()

    def run(self):
        self.save(fetch_raw(self.country, self.state))

@oryxflow.requires(DataLoadState)              # copies country+state params, wires requires()
class ProcessState(oryxflow.tasks.TaskPqPandas):

    def run(self):
        df = self.inputLoad()                  # raw data for this state
        df['value_norm'] = df['value'] / df['value'].sum()   # per-state feature engineering
        self.save(df)

class Country(oryxflow.tasks.TaskPqPandas):
    country = oryxflow.Parameter()

    def requires(self):
        return {s: ProcessState(country=self.country, state=s) for s in STATES[self.country]}

    def run(self):
        self.save(self.inputLoadConcat())      # stacks states, keeps state/country cols
```

Because the whole hierarchy is now **one DAG in one** `run()` call (rather than a nested flow-within-a-flow built inside `run()`), you get three wins for free:

- `oryxflow.Workflow(Country, {'country': 'US'}).preview()` shows every per-state task in the tree.
- The run summary lists the per-state tasks (they land in the same `RunResult`).
- Central reset cascades along `requires()` edges — `reset_upstream`/`reset_downstream` reach every `DataLoadState`/`ProcessState` instance, no hand-tracking inside the task.

To reset just one family everywhere it appears in the DAG (every state/country), pass `only=`:

```
flow = oryxflow.WorkflowMulti(Country, params={'country': list(STATES)})
flow.run()
flow.reset_upstream(Country, only=DataLoadState)   # only DataLoadState instances everywhere
flow.run()                                         # ProcessState/Country auto-recompute
# flow.reset_upstream(Country)                     # or reset the whole upstream (no `only=`)
```

The `only=` filter enumerates every `DataLoadState` (`US/CT`, `US/NY`, `UK/London`, `UK/Belfast`) via the DAG — no hand-listing. Since `check_dependencies` makes `complete()` recursive, invalidating just `DataLoadState` forces `ProcessState`/`Country` to recompute on the next run.

### The enumeration is your own data, not a oryxflow object

`STATES` above is **plain domain data** describing the hierarchy's shape — keep it in a `cfg.py`. Your `requires()` methods just index into it to decide how many children to depend on; that fan-out is the *only* thing that builds the DAG. Name it for what it holds (`STATES`, `STATES_BY_COUNTRY`) — avoid `grid`, which invites confusion with the unrelated `WorkflowMulti` params (covered below and in [Constructing the params grid](https://docs.oryxflow.dev/docs/workflow/#constructing-the-params-grid)).

### Nesting further (multi-level)

The pattern composes to any depth: each aggregating level is another task whose `requires()` returns a dict of the level below and whose `run()` calls `self.inputLoadConcat()`. For a sector → country → state hierarchy, add a `Sector` task that aggregates countries on top of the `Country` task that aggregates states:

```
# cfg.py — plain domain config (nested enumeration), NOT a oryxflow object
UNIVERSE = {
    'Retail': {'US': ['CT', 'NY'], 'UK': ['London']},
    'Office': {'US': ['CA']},
}

class DataLoadState(oryxflow.tasks.TaskPqPandas):
    sector = oryxflow.Parameter()
    country = oryxflow.Parameter()
    state = oryxflow.Parameter()

    def run(self):
        self.save(fetch_raw(self.sector, self.country, self.state))

@oryxflow.requires(DataLoadState)
class ProcessState(oryxflow.tasks.TaskPqPandas):

    def run(self):
        df = self.inputLoad()                  # raw data for this state
        df['value_norm'] = df['value'] / df['value'].sum()   # per-state feature engineering
        self.save(df)

class Country(oryxflow.tasks.TaskPqPandas):          # aggregate states within a country
    sector = oryxflow.Parameter()
    country = oryxflow.Parameter()

    def requires(self):
        return {s: ProcessState(sector=self.sector, country=self.country, state=s)
                for s in cfg.UNIVERSE[self.sector][self.country]}

    def run(self):
        self.save(self.inputLoadConcat())

class Sector(oryxflow.tasks.TaskPqPandas):           # aggregate countries within a sector
    sector = oryxflow.Parameter()

    def requires(self):
        return {c: Country(sector=self.sector, country=c) for c in cfg.UNIVERSE[self.sector]}

    def run(self):
        self.save(self.inputLoadConcat())
```

Each level's `inputLoadConcat()` tags frames with that level's dependency params, so the `Sector` step re-writes the `sector`/`country` columns the `Country` frames already carry — an idempotent overwrite with the same values, so there is no double-counting. If a lower level's tag column ever needs different handling, use `tagkeys=` (tag only these params), `tag=False` (tag nothing), or `concat_fn=` (full control) on `inputLoadConcat`.

### Fan-out vs. independent runs (do you need WorkflowMulti?)

There is really only **one** mechanism here — fan-out via `requires()` over your enumeration. The outer `sector` dimension is just more fan-out, so you have a choice for how to drive the top:

**One DAG (no WorkflowMulti).** Add one more aggregator on top and fan out over sectors too. The whole three-level tree is a single `build()` — one run, one combined output, one reset scope:

```
class AllSectors(oryxflow.tasks.TaskPqPandas):

    def requires(self):
        return {sec: Sector(sector=sec) for sec in cfg.UNIVERSE}

    def run(self):
        self.save(self.inputLoadConcat())

flow = oryxflow.Workflow(AllSectors)
flow.run()
dfall = flow.outputLoad()                                # sector/country/state columns present
flow.reset_upstream()                                    # resets every leaf across the tree
```

**Independent runs (WorkflowMulti).** Keep each sector as a *separate flow* — its own run summary, its own `outputLoad`, its own reset scope — when you want to manage sectors independently. Here the top-level `params` is a list of runs (see [Constructing the params grid](https://docs.oryxflow.dev/docs/workflow/#constructing-the-params-grid)), **not** part of DAG construction:

```
flow = oryxflow.WorkflowMulti(Sector, params={'sector': list(cfg.UNIVERSE)})
flow.run()
dfall = flow.outputLoadConcat(Sector)                   # combine the per-sector flows
flow.reset_upstream(Sector, only=DataLoadState)         # reset one family, all sectors
```

Same result frame either way. Reach for fan-out (`AllSectors`) when you want one combined run; reach for `WorkflowMulti` when sectors are separately-managed experiments.

A complete, runnable version of this sector → country → state example — including the dev loop where you add a feature to the country-level task, iterate on one `(sector, country)` first, then roll it out to every flow *without re-fetching the expensive per-state source* — is in `docs/example-flow-multi.py`.

Tip

These advanced dynamic-loop flows are exactly what the [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) is built to manage. Describe the hierarchy in plain language and it writes the fan-out `requires()` and the `inputLoadConcat()` aggregators; when you iterate, it scopes the reset for you — resetting just the family you changed (`reset_upstream(..., only=...)`) so the expensive leaf tasks are preserved. The hand-tracking this section warns about is what the plugin removes.

## Collector Task

To run several tasks together without combining their outputs, just pass them as a list — you don't need a task for that at all.

```
flow = oryxflow.Workflow()
flow.run([TrainModel1, TrainModel2])
```

When something downstream needs to depend on the whole group, give the group a name with TaskAggregator. List its members with @oryxflow.requires and leave the body empty — the group saves nothing itself, and it's done when all of its members are done.

```
@oryxflow.requires(TrainModel1,TrainModel2) # inherit all params from input tasks
class TrainAllModels(oryxflow.tasks.TaskAggregator):
    pass

flow = oryxflow.Workflow(TrainAllModels)
flow.preview()          # shows the group and every member below it
flow.run()
models = flow.outputLoad()   # one entry per member
```

A group task works like any other task: `preview()` shows what's still pending inside it, per-flow settings reach its members, and `flow.reset_upstream()` resets them.

To run the *same* task for many parameter combinations, use WorkflowMulti — you get one independently managed run per combination.

```
params = dict()
params_all = oryxflow.utils.params_generator_single({'param':['a','b']},params)

flow = oryxflow.WorkflowMulti(tasks_search.SearchModelTrain, params=params_all)
flow.run()
```

## Fully Dynamic

This doesn't work yet, and it's actually quite rare that you need it. Parameters normally fall in a fixed range which can be solved with the approaches above. Another typical reason you would want to do this is to load an unknown number of input files which you can do manually, see "Load External Files" in [tasks](https://docs.oryxflow.dev/docs/tasks/index.md).

```
class TaskA(oryxflow.tasks.TaskCache):
    param = oryxflow.IntParameter()
    def run(self):
        self.save(self.param)

class TaskB(oryxflow.tasks.TaskCache):
    param = oryxflow.IntParameter()

    def requires(self):
        return TaskA()

    def run(self):
        value = 1
        df_train = self.input(param=value).load()
```

# Advanced: Parameters

Intelligent parameter management is one of the most powerful features of oryxflow. Parameters are how you try different settings — a preprocessing flag, a model choice, a date range — without copying files or renaming outputs by hand. Give a task parameters and oryxflow keeps a **separate cached output per parameter set**, so you can compare runs side by side and switch between them instantly; change a parameter and it reruns exactly the tasks that depend on it and leaves the rest untouched. This is what makes experimentation cheap. New users often have questions on parameter management, this is an important section to read.

## Specifying parameters

Tasks can take any number of parameters.

```
import datetime

class TaskTrain(oryxflow.tasks.TaskPqPandas):
    do_preprocess = oryxflow.BoolParameter(default=True)
    model = oryxflow.Parameter(default='xgboost')
```

## Running tasks with parameters

Just pass the parameters values, everything else is the same.

```
oryxflow.Workflow(TaskTrain).run() # use default do_preprocess=True, model='xgboost'
oryxflow.Workflow(TaskTrain, dict(do_preprocess=False, model='nnet')).run() # specify non-default parameters
# or
params = dict(do_preprocess=False, model='nnet')
oryxflow.Workflow(TaskTrain, params).run() # specify non-default parameters
```

Note that you can pass parameters for upstream tasks directly to the terminal task, they will be automatically passed to upstream tasks. See below for details.

## Loading Output Data with Parameters

If you are [using parameters](https://docs.oryxflow.dev/docs/advparam/index.md) this is how you load outputs. Make sure you run the task with that parameter first.

```
df = oryxflow.Workflow(TaskTrain).outputLoad() # load data with default parameters
params = dict(do_preprocess=False, model='nnet')
df = oryxflow.Workflow(TaskTrain, params).outputLoad() # specify non-default parameters
```

## Parameter types

Parameters can be typed.

```
import datetime

class TaskTrain(oryxflow.tasks.TaskPqPandas):
    do_preprocess = oryxflow.BoolParameter(default=True)
    dt_start = oryxflow.DateParameter(default=datetime.date(2010,1,1))
    dt_end = oryxflow.DateParameter(default=datetime.date(2020,1,1))

    def run(self):
        if self.do_preprocess:
            if self.dt_start>datetime.date(2010,1,1):
                pass
```

For the full list of parameter types and their options, see the [API reference](https://docs.oryxflow.dev/docs/reference/index.md).

## Avoid repeating parameters in every class

You often need to pass parameters between classes. With oryxflow, you do not need to repeat parameters in every class, they are automatically managed, that is they are automatically passed to upstream tasks from downstream tasks.

```
class TaskTrain(oryxflow.tasks.TaskPqPandas):
    do_preprocess = oryxflow.BoolParameter(default=True)
    dt_start = oryxflow.DateParameter(default=datetime.date(2010,1,1))
    dt_end = oryxflow.DateParameter(default=datetime.date(2020,1,1))
    # ...

@oryxflow.requires(TaskTrain) # automatically inherits parameters
class TaskEvaluate(oryxflow.tasks.TaskPickle):

    # requires() is automatic
    # do_preprocess => inherited from TaskTrain
    # dt_start => inherited from TaskTrain
    # dt_end => inherited from TaskTrain

    def run(self):
        print(self.do_preprocess) # inherited
        print(self.dt_start) # inherited

oryxflow.Workflow(TaskEvaluate, {'do_preprocess': False}).preview()  # specify non-default parameters
'''
+--[TaskEvaluate-{'do_preprocess': 'False', 'dt_start': '2010-01-01', 'dt_end': '2020-01-01'} (PENDING)]
+--[TaskTrain-{'do_preprocess': 'False', 'dt_start': '2010-01-01', 'dt_end': '2020-01-01'} (PENDING)] => automatically passed upstream
'''
```

Note that you can pass parameters for upstream tasks directly to the terminal task, they will be automatically passed to upstream tasks. do_preprocess=False will be passed down from TaskEvaluate to TaskTrain.

If you require multiple tasks, you can inherit parameters from those tasks. TaskEvaluate depends on both TaskTrain and TaskPredict.

```
class TaskTrain(oryxflow.tasks.TaskPqPandas):
    do_preprocess = oryxflow.BoolParameter(default=True)

class TaskPredict(oryxflow.tasks.TaskPqPandas):
    dt_start = oryxflow.DateParameter(default=datetime.date(2010,1,1))
    dt_end = oryxflow.DateParameter(default=datetime.date(2020,1,1))

@oryxflow.requires(TaskTrain,TaskPredict) # inherit all params from input tasks
class TaskEvaluate(oryxflow.tasks.TaskPickle):
    # do_preprocess => inherited from TaskTrain
    # dt_start => inherited from TaskPredict
    # dt_end => inherited from TaskPredict

    def run(self):
        print(self.do_preprocess) # inherited from TaskTrain
        print(self.dt_start) # inherited from TaskPredict

oryxflow.Workflow(TaskEvaluate, {'do_preprocess': False}).preview()  # specify non-default parameters
'''
+--[TaskEvaluate-{'do_preprocess': 'False', 'dt_start': '2010-01-01', 'dt_end': '2020-01-01'} (PENDING)]
   |--[TaskTrain-{'do_preprocess': 'False'} (PENDING)] => automatically passed upstream
   +--[TaskPredict-{'dt_start': '2010-01-01', 'dt_end': '2020-01-01'} (PENDING)] => automatically passed upstream
'''
```

@oryxflow.requires also works with aggregator tasks.

```
@oryxflow.requires(TaskTrain,TaskPredict) # inherit all params from input tasks
class TaskEvaluate(oryxflow.tasks.TaskAggregator):
    pass
```

For another ML example, see [Example (ML)](https://docs.oryxflow.dev/docs/example-ml/index.md).

For more details, see the [API reference](https://docs.oryxflow.dev/docs/reference/index.md).

A project scaffolded with [`/oryxflow:init-project`](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) wires parameter inheritance in for you.

## Avoid repeating parameters when referring to tasks

To run tasks and load their output for different parameters, you have to pass them to the task. Instead of hardcoding them each time, it is best to keep them in a dictionary and pass that to the task.

```
# avoid this
flow = oryxflow.Workflow(TaskTrain, dict(do_preprocess=False, model='nnet'))
flow.run()
flow.outputLoad()

# better
params = dict(do_preprocess=False, model='nnet')
flow = oryxflow.Workflow(TaskTrain, params)
flow.run()
flow.outputLoad()
```

# Functional Tasks

## What are functional tasks?

Functional tasks are meant to provide a nice decorator based way of defining tasks.

## How to create a functional task?

For defining our tasks we will need to first define a Workflow() object.

```
from oryxflow.functional import Workflow
flow = Workflow()
```

Each function is decorated with a flow.task decorator - that takes a oryxflow.tasks.TaskName as parameter

```
@flow.task(oryxflow.tasks.TaskPqPandas)
def your_functional_task(task):
    print("Running a complicated task!!")
```

You might have noticed we provide a task parameter to the function above.

This is deliberate.

If you have worked with oryxflow.task before you would remember having a self parameter passed to run() method.

Here task is exactly that. It contains all methods available in oryxflow.task.Task

## Running a functional task

All functional tasks are run as oryxflow.task under the hood.

So we require to run them as you would run any oryxflow.task

Workflow() object comes with a run method which does exactly that.

```
flow.run(your_functional_task)
```

Below is a minimal example of functional task that encompasses everything mentioned above.

```
import oryxflow
from oryxflow.functional import Workflow
import pandas as pd

flow = Workflow()

@flow.task(oryxflow.tasks.TaskCache)
def sample_functional_task(task):
    df = pd.DataFrame({'a':range(3)})
    print("Functional task running!")
    task.save(df)

flow.run(sample_functional_task)
```

## Additional decorators

These decorators are to be decorated after @flow.task

- @flow.persists
- Takes in a list of variables that need to be persisted for the flow task.
- `python @flow.persists(['a1', 'a2'])`
- @flow.params
- Takes in keyword-arguments of parameters and their types to be used in the function body.
- `python @flow.params(example_argument=oryxflow.IntParameter(default=42))`
- @flow.requires
- Defines dependencies between flow tasks.
- `python @flow.requires({"foo": func1, "bar": func2}) @flow.requires(func1)`

Example -

```
...
@flow.task(oryxflow.tasks.TaskCache)
@flow.requires({"a":get_data1, "b":get_data2})
@flow.persists(['aa'])
def example_function(task):
    df = task.inputLoad()
    a = df["a"]
    b = df["b"]
    print(a,b)
    output = pd.DataFrame({'a':range(4)})
    task.save({'aa':output})
...
```

## Passing parameters to the run() method

We saw in one of the above section how to run functional tasks.

oryxflow also allows you to pass in parameters to these functions dynamically using @flow.params()

Below is an example of passing a 'multiplier' paramter to a functional task.

```
@flow.params(multiplier=oryxflow.IntParameter(default=0))
def print_parameter(task):
    print(task.multiplier)

flow.run(print_parameter, params={'multiplier':42})
```

So basically, you define the parameter name and its type with @flow.params and then use the run() method's params to pass in the actual value

## Additional methods

Some of the functions that are in oryxflow are available in the Workflow() object too!

Here's a list of them -

- preview(function)
- outputLoad(function)
- run(functions_as_list)
- reset(function)
- outputLoadAll()

There are also some functions unique to the functional workflow.

- add_global_params(example_argument=oryxflow.IntParameter(default=42))
- resetAll()
- delete(function)
- deleteAll()
# Build with Claude Code

# Claude Code for data science: faster, cheaper, more trustworthy AI data analysis

**oryxflow makes AI data analysis faster, cheaper, and more trustworthy.** It's a Claude Code plugin and skill for data science, backed by a Python library, that teaches your coding agent to build the work as a cached pipeline. Two things follow immediately:

- **It stops paying twice.** The agent reuses results it already computed instead of recomputing them — so you're not waiting on the 10-minute data pull again, or spending tokens on an agent watching it run.
- **It stops being confidently wrong.** The agent can't quietly train a model on last week's features, because a change to the data, a parameter, or the code reruns exactly what that change affects. You get the number your current code actually implies.

If you only remember one thing: AI writes the analysis fast, but the hard part — *did the right data produce this result, and can I check that?* — is exactly what a plugin can enforce and a raw agent can't.

## What are Claude Code plugins and skills for data science?

Claude Code plugins extend the agent with new abilities; **skills** are the part that teaches it *how to work* — conventions and procedures that load into context automatically when they're relevant. (For the precise distinction, see [the glossary](https://docs.oryxflow.dev/docs/glossary/#what-is-the-difference-between-a-claude-code-plugin-a-skill-and-a-slash-command).) For data science, the useful plugins fall into a few jobs: connecting to your data, running notebooks, scaffolding a project, and — the one most tools skip — making sure the analysis is still **accurate** ten iterations in, when the agent has edited half the pipeline and nothing has raised an error.

The oryxflow plugin owns that last job. It installs an `oryxflow` skill that activates whenever you work in a data-science pipeline, plus slash commands to scaffold and migrate projects. The skill makes the agent a *disciplined* user of a cache and a lineage log: it checks what's already computed before recomputing, verifies its own edits actually took effect, and records what ran and why. That's the difference between an agent that writes plausible pandas and one whose numbers you can stand behind — and reproduce next week.

To be precise about what it is: oryxflow ships a **Claude Code plugin (a skill plus slash commands)** — not an MCP server. It drives the open-source, MIT-licensed oryxflow library, which does the actual caching and lineage on your machine.

## The problem: AI writes data analysis fast — but is the number right?

A coding agent's weakness in data work isn't syntax; it's **invisible state**. Over a long session it loses track of what's already computed and whether it's still valid, then quietly builds on stale intermediates or re-runs a 40-minute job it didn't need to. Nothing errors. The number is just wrong, or the run just cost you ten minutes and a pile of tokens.

These are trust failures, and they get *worse* as the agent writes more of the code:

- **Stale intermediates** — a feature changes, a cached file doesn't, and the model trains on yesterday's data.
- **Lost lineage** — no one can say which code and inputs produced `model_final_v3.pkl`.
- **Wasted recomputation** — a one-line downstream edit re-runs the expensive data pull.

None of these are math errors. They're mechanics-of-the-pipeline errors — and they're the ones an agent introduces most.

## How oryxflow keeps Claude Code's data analysis accurate

The library carries the discipline the agent can't hold in its head, and the skill makes the agent *use* it correctly. Concretely, the plugin has the agent:

- **start every session by reading cache state** — pending staleness warnings, last runs, recent failures — so it never assumes a stale result is fresh;
- **verify after each edit that the intended steps actually reran**, so a change that should have invalidated downstream work can't pass silently;
- **answer every staleness or expensive-recompute warning with the right move** — recompute, accept an output-equivalent refactor, or pin — instead of guessing;
- **record decision-relevant results as lineage**, so they become the agent's memory across sessions.

Underneath, the library gives each step an identity derived from its parameters and its code, caches its output, and reruns exactly what a parameter, data, or **code** change affects. Two consequences, which are the whole reason to install this: **the agent stops re-paying for work it already did** — your time and your token budget — and **it stops being able to hand you a number built from stale inputs**. Reproducibility and lineage are how that's achieved, and they're what let you *check* the agent instead of taking its word. See [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md) for the full picture.

## Install

```
/plugin marketplace add https://github.com/oryxintel/oryxflow-claude-plugin.git
/plugin install oryxflow@oryxflow
```

Once installed, the `oryxflow` skill **auto-activates** whenever you work in an oryxflow project — you don't invoke it, it's just on. If you install it into an empty directory and nothing happens, run [`/oryxflow:init-project`](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) first (or just ask the agent to set up a project) so there's a pipeline for the skill to work on. Full walkthrough: [Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md).

## Start with a quick EDA — it scales from there

You don't need a pipeline to justify installing this, and the plugin won't make you build one before you've looked at the data. It covers both ends of a project's life:

- **Exploring.** Ask the agent to poke at a new dataset and it writes a read-only probe under `eda/<subject>/`, stating the question in a docstring and printing the answer legibly — instead of scattering one-off snippets it can't re-run next session. Findings that matter get written into the project's data doc as it goes, so you don't re-ask the same question in a week.
- **Growing.** When a probe turns out to be load-bearing — you keep re-running it, or something depends on its output — `/oryxflow:migrate` lifts it and the rest of the script into cached, parameterized tasks. It reads the existing code as the spec, shows you the step-to-task map, and writes files only when you approve; it never deletes your source.

So the honest answer to "isn't this overkill for a quick analysis?" is: **start simple, and let it scale.** Simple scripts on day one, an arbitrarily complex pipeline later, with no rewrite in between — which matters because data-science projects only ever grow more complicated. It pays off hardest where an agent is otherwise most error-prone: expensive intermediate steps, model training you iterate on, parameter sweeps, and research code someone else has to reproduce.

**Reach for something else when** you need production scheduling, retries, and SLAs — that's [Airflow, Prefect, or Dagster](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md)'s job, not oryxflow's — or when what you want is a searchable dashboard of every run's metrics, which is an [experiment tracker's](https://docs.oryxflow.dev/docs/experiment-tracking/index.md) job (MLflow, Weights & Biases) and composes cleanly beside oryxflow rather than replacing it.

## How it compares to other Claude Code data-science tools

The plugin landscape is early, so choose by the **job you need done**, not by a "best plugin" label. Most tools cover data access or notebook execution; oryxflow covers the reproducibility layer that keeps AI-generated analysis trustworthy as it grows.

| Job to be done                                         | Reach for                 |
| ------------------------------------------------------ | ------------------------- |
| Keep AI analysis reproducible, cached, lineage-tracked | **oryxflow plugin**       |
| Query a warehouse / connect a data source              | a data-connector plugin   |
| Run and edit notebooks                                 | a notebook plugin         |
| Track and chart experiment metrics in a UI             | MLflow / Weights & Biases |

These aren't mutually exclusive — oryxflow sits *underneath* the analysis and composes with a tracker or a data connector. For the full breakdown, see [The best Claude Code plugins and tools for data science](https://docs.oryxflow.dev/blog/ai-agents/best-claude-code-plugins-for-data-science/index.md).

## What it looks like

You describe the analysis; the agent writes tasks like these, and the engine handles caching and reruns:

```
import oryxflow
import pandas as pd

class GetData(oryxflow.tasks.TaskPqPandas):        # output saved as parquet — no file paths
    def run(self):
        self.save(pd.DataFrame({'x': range(10)}))

@oryxflow.requires(GetData)                         # declare the dependency
class ProcessData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                       # GetData's output, already loaded
        df['x2'] = df['x'] ** 2
        self.save(df)

flow = oryxflow.Workflow(ProcessData)
flow.run()                                          # runs GetData, then ProcessData
```

Run `flow.run()` again and nothing recomputes — both outputs already exist. Edit `ProcessData`'s code and only it (and anything downstream) reruns, automatically. That's what the agent is taught to rely on and verify.

## Frequently asked questions

**Does oryxflow work with any AI coding agent, or only Claude Code?** The library is a plain Python package — it works no matter who writes the code, including you by hand. The *plugin* packages the disciplines specifically for Claude Code; other agents can follow the same [CLAUDE.md conventions](https://docs.oryxflow.dev/docs/claude-plugin/index.md) manually.

**How do I stop Claude Code from rerunning expensive steps or building on stale data?** Install the oryxflow Claude Code plugin. It teaches the agent to cache every step, verify after each edit that the right tasks actually reran, and answer staleness warnings instead of ignoring them — so it reuses expensive results and never trains on stale intermediates. The library externalizes the 'did I already run this, is it still valid?' state the agent can't reliably hold across a long session.

**Is there a Claude Code skill or plugin for data science?** Yes — oryxflow ships an official Claude Code **plugin (skill + slash commands)** for data science that makes an AI agent's analysis reproducible and cached by default. The skill auto-activates in an oryxflow project and applies data-science conventions as the agent writes; the slash commands scaffold a project, migrate an existing notebook, and check standards. It runs on a local Python library — no server or account, and not an MCP server.

**Do I have to restructure my project to use it?** No — adopt it one task at a time. Point the agent at an existing script with `/oryxflow:migrate`, or start fresh with `/oryxflow:init-project`.

**Is the oryxflow plugin overkill for a quick exploratory analysis?** No, because it covers exploration too rather than demanding a pipeline upfront. Ask the agent to look at a new dataset and it writes a re-runnable read-only probe under `eda/`, documenting the question and recording anything it learns, instead of scattering one-off snippets. When a probe turns out to be load-bearing, `/oryxflow:migrate` lifts it into cached tasks. So you start with simple scripts and scale to any complexity later, with no rewrite in between.

## Takeaway

- Claude Code writes data analysis fast. oryxflow makes it **faster, cheaper, and more trustworthy**: the agent reuses expensive results instead of burning your minutes and tokens redoing them, and can't hand you a number built from stale inputs.
- **A code, data, or parameter change reruns exactly what it affects** — which is what makes the agent's work checkable rather than something you have to take on faith.
- **Start small.** Exploration stays exploration; `/oryxflow:migrate` grows it into a pipeline when the analysis earns one.
- It's a **plugin (skill + slash commands)**, not an MCP server, driving a local, MIT-licensed library — no server, no account, no telemetry.

Ready to build?

```
/plugin marketplace add https://github.com/oryxintel/oryxflow-claude-plugin.git
/plugin install oryxflow@oryxflow
```

- **[Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md)** — the full plugin section: commands, trust model, project structure.
- **[Built for AI coding agents](https://docs.oryxflow.dev/docs/ai-ready/index.md)** — the docs themselves are machine-readable (`llms.txt`, unit-tested core examples, docstring-generated reference), so any agent can learn oryxflow in one request.
- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** — the positioning and how the library works.
- **[Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)** — from nothing to a self-caching pipeline in minutes.
- **[Migrate a messy notebook project](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md)** — already out of control? Start here.

# oryxflow is built for AI coding agents — documentation included

Most libraries are documented for humans and then discovered by agents by accident. oryxflow is written for both. Two halves:

- **The library** gives an agent the discipline it can't hold in its head — every step cached with an identity derived from its parameters *and* its code, so it reuses expensive results and can't build on stale ones. That's the story in [Claude Code for data science](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md).
- **The documentation** gives an agent something it can actually read, in bulk, and trust. That's this page, collected in one place so you don't have to take "AI-friendly" on faith.

## Read the entire documentation in one request

The docs publish the [llms.txt convention](https://llmstxt.org/), so an agent doesn't have to crawl page by page and guess what it missed:

| URL                                                         | What it is                                                                          | Use it for                                              |
| ----------------------------------------------------------- | ----------------------------------------------------------------------------------- | ------------------------------------------------------- |
| [`/llms.txt`](https://docs.oryxflow.dev/llms.txt)           | A sectioned index — every page, grouped, with a one-line description of the library | Cheap orientation: the agent picks the pages it needs   |
| [`/llms-full.txt`](https://docs.oryxflow.dev/llms-full.txt) | The complete documentation concatenated into a single file                          | One request, whole corpus — no crawling, nothing missed |

The fastest way to make any agent competent at oryxflow is to hand it the second URL at the start of a session:

```
Read https://docs.oryxflow.dev/llms-full.txt, then convert my script in analysis.py
into oryxflow tasks.
```

Both files are regenerated on every deploy, so they never describe an older version of the library than the one you installed.

## The examples an agent reads first are executed by the test suite

An agent that reads a broken example writes broken code with total confidence. So the pages an agent learns the library from aren't just illustrative — the [home page](https://docs.oryxflow.dev/index.md), the [quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md) and the [I/O formats guide](https://docs.oryxflow.dev/docs/targets/index.md) are compiled into real pytest files and run top-to-bottom on every build. If one stops working, the build fails and it gets fixed; it doesn't sit there misleading people.

That's the core learning path, so you can tell an agent to copy a pattern from it and expect it to run. To be straight about the boundary: examples on the deeper guide pages and the blog aren't all executed, so treat those as illustrative — and the machine-readable [API reference](https://docs.oryxflow.dev/docs/reference/index.md) below is generated from the source either way.

## The API reference is generated from the docstrings — all of them

[API Reference](https://docs.oryxflow.dev/docs/reference/index.md) is built from the source, not maintained alongside it, so it can't drift. **100% of the public API carries a docstring**, with arguments, return values and a usage example where the call isn't obvious.

This matters for agents specifically: docstrings are available at runtime, so an agent working in a REPL or a notebook can check a signature without leaving the process, and without inventing an argument that doesn't exist.

```
help(oryxflow.requires)      # what it does, what it takes, and an example
help(oryxflow.runLoad)
```

## Answer engines get a machine-readable description of the project

Every page carries structured data (JSON-LD): the library as a single identified entity cross-linked to its GitHub and PyPI identities, each page as an article, and the pages with a Q&A section as a `FAQPage`. Every answer in that structured data is generated from the visible prose, so an assistant quoting oryxflow quotes what the page actually says.

Canonical URLs are stamped on every page and the sitemap's freshness dates come from git commit history, so an agent or crawler comparing versions sees one authoritative copy with an honest last-modified date.

## The conventions ship as a skill, not just prose

Reading the docs is the general-purpose route. For Claude Code there's a shortcut: the [oryxflow plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) installs a skill that auto-activates in a pipeline project, so the agent applies the conventions as it writes instead of needing to be reminded. The slash commands scaffold a project, migrate an existing script, and check an existing project against the standards.

Projects scaffolded by the plugin also get a `CLAUDE.md` describing the layout and rules — which any agent that reads repo instructions can follow, not only Claude Code.

## Frequently asked questions

**Does oryxflow have an llms.txt?** Yes. [`/llms.txt`](https://docs.oryxflow.dev/llms.txt) is a sectioned index of every page, and [`/llms-full.txt`](https://docs.oryxflow.dev/llms-full.txt) is the entire documentation concatenated into one file, so an agent can read the whole thing in a single request instead of crawling page by page.

**How do I get an AI coding agent to write correct oryxflow code?** Point it at `https://docs.oryxflow.dev/llms-full.txt` once at the start of the session, or install the [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md), which loads the conventions automatically. Both work because the examples an agent reads first — the home page, the quickstart and the I/O formats guide — are executed by the test suite on every build, so what it copies is code that actually runs.

**Can I trust the code examples in the oryxflow docs?** The core learning path is unit-tested: the home page, the quickstart and the I/O formats guide are compiled into pytest files with phmdoctest and run top-to-bottom on every build, so an example that stopped working fails CI instead of quietly misleading you or your agent. Examples on the deeper guide pages and blog posts are not all executed.

**Does oryxflow work with agents other than Claude Code?** Yes. The machine-readable docs, the tested examples and the docstrings are plain files on a public site, so any agent or assistant can use them. The Claude Code plugin is a convenience that packages the same conventions as a skill; the library itself is an ordinary Python package.

## Next

- **[Claude Code for data science](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md)** — the other half: what the library stops an agent from getting wrong.
- **[Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md)** — the plugin, its commands and its trust model.
- **[API Reference](https://docs.oryxflow.dev/docs/reference/index.md)** — the generated symbol reference.
- **[Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)** — from nothing to a self-caching pipeline in minutes.

# Build trustworthy AI data analysis with Claude Code

**The oryxflow Claude Code plugin makes an AI coding agent's data analysis reproducible.** It teaches Claude Code to use oryxflow's cache and lineage log correctly: the agent reuses expensive work, verifies its own reruns, and never builds on stale data. It ships as a skill plus slash commands — not an MCP server.

AI coding agents now write real data-science pipelines — feature engineering, model training, experiment sweeps. They write plausible code fast. The hard part isn't the code; it's making that code **trustworthy**: not silently rerunning expensive steps, not building on stale intermediates, and always knowing which data and code produced a result.

The **oryxflow Claude Code plugin** is how you get that. It pairs the oryxflow library — which carries the caching, lineage, and reproducibility an agent can't hold in its head — with a skill that makes the agent a *disciplined user* of that cache. The result is **faster, cheaper, and more trustworthy AI data analysis**: the agent reuses cached work instead of paying to recompute it, verifies its own edits actually took effect, and leaves a queryable record of what ran and why.

The short version

Install the plugin, then just describe what you want. It scaffolds a project, wires the DAG, and follows the [house conventions](https://github.com/oryxintel/oryxflow-claude-plugin/blob/main/skills/oryxflow/conventions.md) so the pipeline is reproducible by default — no manual task wiring, no stale-cache surprises.

## Install

```
/plugin marketplace add https://github.com/oryxintel/oryxflow-claude-plugin.git
/plugin install oryxflow@oryxflow
```

Once installed, the `oryxflow` skill **auto-activates** whenever you work in an oryxflow project (editing `tasks.py` / `flow.py` / `run.py` / `cfg.py` / `flow_params.py`) — you don't invoke it, it's just on.

Note

The skill activates inside an oryxflow **project**. If you install it into an empty directory and nothing happens, run [`/oryxflow:init-project`](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) first (or ask the agent to set up an oryxflow project) so there's a pipeline for the skill to work on.

## Why this matters

A coding agent's weakness in data work isn't syntax — it's **invisible state**. Across a long session it loses track of what's already computed and whether it's still valid, then trains on stale features or re-runs a 40-minute job it didn't need to. Those are trust failures, and they are exactly what oryxflow externalizes into a cache and a lineage log.

The plugin makes the agent *use* that machinery correctly. It:

- **starts every session by reading cache state** — pending warnings, last runs, recent failures — so it never assumes a stale cache is fresh;
- **verifies after each edit that the intended tasks actually reran**, so a change that should have invalidated downstream work can't pass silently;
- **answers every staleness or expensive-recompute warning with the right move** — recompute, accept an output-equivalent refactor, or pin — instead of guessing (see [Automatic code invalidation](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation));
- **records decision-relevant results as lineage**, so they persist across sessions and become the agent's memory next time.

These are the same disciplines documented in the [CLAUDE.md snippet for AI-agent projects](https://docs.oryxflow.dev/docs/managing-workflows/#claudemd-snippet-for-ai-agent-projects) — shipped as a skill so they load automatically and stay current with the library, instead of a copy you paste and forget.

## In this section

- **[Commands](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md)**

  The five slash commands — scaffold a project, migrate an existing analysis, check standards, and put data under Git LFS.

- **[Trustworthy AI data analysis](https://docs.oryxflow.dev/docs/claude-plugin/trust/index.md)**

  Why you shouldn't take an agent's numbers on faith — and how the plugin makes its work *cheap to verify* instead: session status, rerun verification, and durable lineage.

- **[Data-science project structure](https://docs.oryxflow.dev/docs/claude-plugin/project-structure/index.md)**

  The load-bearing scaffold that keeps AI-generated data-science code from rotting — separation of concerns that's the shape of the code, not just its filing — and grows with the project.

- **[Stop AI data analysis turning into a mess](https://docs.oryxflow.dev/docs/claude-plugin/coding-standards/index.md)**

  Canonical names, code grouped by subject, docstrings as documentation — loaded into the agent's context so they shape the analysis code as it's written, not audited after.

- **[Why library + plugin is a matched pair](https://docs.oryxflow.dev/docs/claude-plugin/why/index.md)**

  The division of labor: what the library carries, what the agent carries, and why the pairing gets *more* valuable as a project grows.

## Learn more

- Plugin repository and issues: <https://github.com/oryxintel/oryxflow-claude-plugin>
- House conventions the plugin follows: <https://github.com/oryxintel/oryxflow-claude-plugin/blob/main/skills/oryxflow/conventions.md>
- Plugin changelog: <https://github.com/oryxintel/oryxflow-claude-plugin/blob/main/docs/CHANGELOG.md>
- New to oryxflow? Start with **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** and the **[Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)**.

This library is the engine the plugin drives; the full API is documented throughout the rest of these docs.

# Plugin commands

The plugin adds five slash commands. Most of the time you won't need them explicitly — the `oryxflow` skill activates on its own inside a project — but they're the fast path for the common setup and maintenance jobs.

- **`/oryxflow:init-project`** — set up a ready-to-run project structure in an empty directory, so you start writing tasks straight away instead of building the folders, files, and conventions by hand.
- **`/oryxflow:migrate`** — restructure an existing ad-hoc analysis (monolithic notebooks, linear scripts, hardcoded paths) into a cached, parameterized oryxflow pipeline, **one task at a time** — so you get reproducibility and caching without a risky big-bang rewrite.
- **`/oryxflow:init-gitlfs`** — put `data/` under Git LFS, so you version and share your data as easily as your code — teammates clone the repo and get the exact datasets each run produced.
- **`/oryxflow:update-project`** — bring an older project up to the current project structure, so you pick up the latest conventions and layout without a manual migration.
- **`/oryxflow:check-standards`** — check names, style, and docstrings against the house standards, so the codebase stays consistent and easy for teammates (and the AI) to navigate and extend.

## The migration path most people want

If you already have a notebook or script that works, `/oryxflow:migrate` is the on-ramp. It converts the analysis into tasks incrementally — each step becomes a cached, parameterized task with its dependencies wired — so at every point you have a working pipeline, not a half-rewritten one. The end state is reproducible and lineage-tracked, and the expensive steps stop rerunning on every edit.

See the companion guide [From notebook to a reproducible, cached pipeline](https://docs.oryxflow.dev/blog/reproducibility/notebook-to-reproducible-pipeline/index.md) for what that transformation looks like step by step.

## After scaffolding

Once the project exists, the skill takes over automatically — it keeps the wiring consistent, verifies your edits actually reran the tasks you expected, and answers staleness warnings the right way. That ongoing discipline is the subject of [Trustworthy AI data analysis](https://docs.oryxflow.dev/docs/claude-plugin/trust/index.md) and [Why library + plugin is a matched pair](https://docs.oryxflow.dev/docs/claude-plugin/why/index.md).

The `init-project` scaffold and the graduated path a growing project follows are covered in [data-science project structure](https://docs.oryxflow.dev/docs/claude-plugin/project-structure/index.md); the naming, code-organization, and docstring conventions `check-standards` enforces are in [data-science coding standards](https://docs.oryxflow.dev/docs/claude-plugin/coding-standards/index.md).

# Trustworthy AI data analysis

**You should not take an AI's data analysis on faith — but you also shouldn't have to re-derive it by hand to check it.** The oryxflow plugin's job is to make the agent's work *cheap to verify*: reproducible, inspectable, and honest about what actually ran.

An AI coding agent writes correct pandas as well as most people. That isn't where the risk lives. The risk is that data mistakes rarely announce themselves. In ordinary programming a wrong result tends to crash, fail a test, or refuse to render. In data work it usually doesn't: a join quietly changes the row count, an arithmetic step misaligns two indexes, a percentage is computed against the wrong denominator — and the pipeline still runs to completion and prints a plausible number. The number is wrong and nothing says so.

That's why the useful question isn't *"is the agent trustworthy?"* but *"can I check its work without rebuilding it?"* The plugin is built to make the answer yes.

## What tends to go wrong

A handful of failure modes come up again and again, and none are exotic:

- **Silent data errors.** A merge that should be one-to-many is actually many-to-many and inflates every downstream total. These complete without error.
- **Reusing stale output after an edit.** The agent changes a task's code, reruns, and — if the engine treated the task as already done — trains on the *previous* output. The run looks successful; the number is from before the edit.
- **Results you can't reproduce.** An analysis assembled from inline snippets over one session produced its number once. Asked for it again next month, there's no reliable way to regenerate it.
- **State that was never written down.** What was in memory, which cell ran, in what order — when the result depends on things nothing captured, it can't be audited.

The common thread: the output looks identical whether the computation underneath was right or wrong. Trust has to come from somewhere other than the output.

## What the plugin does about it

The library carries the state an agent can't hold in its head — a cache, a dependency graph, and a lineage log. The plugin makes the agent *use* that machinery like a careful analyst would:

- **It starts every session by reading cache state**, not assuming it. Before touching anything, the agent checks pending staleness warnings, the last run of each task, and recent failures — so it never mistakes a stale cache for a fresh one.
- **It verifies that an edit actually took effect.** After changing a task, "the run didn't error" is not evidence the new code ran. The agent confirms the edited task shows up as *having recomputed* — a one-line check that turns an easy-to-miss failure into a caught one.
- **It answers every staleness warning with the right exit** — recompute, accept an output-equivalent refactor, or pin a task deliberately — instead of letting warnings pile up and get ignored.
- **It leaves decision-relevant numbers as durable lineage.** Row counts, drop rates, headline metrics get logged and persisted, so next session has a memory of what happened rather than starting blind.
- **It applies the silent-error habits as it writes** — validate the merge and assert the row relationship, look at the frame's shape and null counts before stating a finding, quote every number from a saved artifact rather than eyeballing it off a chart. These are ordinary practices; the value is that they're loaded into the agent's working context at the moment the task is written, not run as a review checklist afterward.

Together these make the work *checkable by construction*: any result can be re-opened, any rerun confirmed, and the common silent errors are caught by habit instead of luck.

## The honest boundary: reproducible ≠ correct

Being trustworthy means being clear about the edge. oryxflow guarantees that a result was produced by the exact code and inputs it recorded. It does **not** guarantee the result is *right*. A pipeline with a bug in its feature logic is reproduced just as faithfully as a correct one — you'll get the same wrong number every time, cleanly tied to the flawed code that made it.

That's not a weakness; it's the honest scope. Whether the model suits the question, whether the method is sound, whether an analysis that runs cleanly is even answering the right question — those stay judgment, and judgment doesn't delegate to a library or an agent. What changes is that *checking* is now cheap: you can re-open any result, confirm what actually ran, and re-derive the conclusion, instead of reconstructing the whole analysis before you can begin to question it.

## Takeaway

In this domain the failure isn't a crash — it's a confident wrong number, produced as fluently as a right one, from a mistake that raised no error. The realistic response isn't to hope the agent is careful; it's to keep the work on a footing where results are reproducible, reruns are checkable, and silent errors are caught by habit. The library provides that footing; the plugin supplies the habits. Together they don't make the agent trustworthy — they make its work **cheap to verify**. That's the more useful thing, and the whole point of *faster, cheaper, and more trustworthy* AI data analysis.

**Read next**

- [Why library + plugin is a matched pair](https://docs.oryxflow.dev/docs/claude-plugin/why/index.md) — the division of labor in full.
- [The load-bearing project scaffold](https://docs.oryxflow.dev/docs/claude-plugin/project-structure/index.md) — the shape that keeps AI code from rotting.
- [Stop AI data analysis turning into a mess](https://docs.oryxflow.dev/docs/claude-plugin/coding-standards/index.md) — the conventions loaded where the work happens.
- [What caching does *not* protect against](https://docs.oryxflow.dev/docs/managing-workflows/#what-caching-does-not-protect-against) — the full boundary.
- [Best practices for AI-assisted data analysis](https://docs.oryxflow.dev/blog/ai-agents/best-practices-ai-assisted-data-analysis/index.md) — the same discipline as a standalone guide.

# A data-science project structure that stays clean as it grows

**A folder template tells you where a data-science file goes. It doesn't stop the analysis inside from rotting.** The oryxflow Claude Code plugin scaffolds a data-science project structure that's *load-bearing* — one that makes the messy shape harder to write than the clean one — so an AI agent (which defaults to the flat script and the reused variable) produces a project that stays well-formed as it grows.

Data-science code rots in a predictable way: one notebook becomes two, then a folder of them plus a few scripts, all reading the same directory, each with its own copy of the cleaning logic. Names drift, functions never appear, and six weeks later nobody can say which cell produced the headline number. Hand the work to a coding agent and the failure arrives *faster*, because the agent writes the path of least resistance by default.

## Start from a runnable scaffold

`/oryxflow:init-project` copies a minimal, runnable project into an empty directory — you start writing real tasks immediately instead of hand-building folders and wiring:

```
project/
├── tasks.py          # task definitions — what each step produces
├── cfg.py            # global config: environment, dates, credentials
├── flow_params.py    # workflow parameters (kept separate from global config)
├── flow.py           # the workflow instance — defined once, imported everywhere
├── run.py            # execute the workflow: python run.py
├── visualize.py      # load outputs for analysis and reporting
└── docs/oryxflow-data.md   # a place to record what you learn about the data
```

`python run.py` works from the first minute; you replace the placeholder tasks with your real pipeline. It never overwrites files you already have.

## Why the structure is load-bearing, not decorative

A picture-perfect directory tree can still contain a single notebook that re-cleans the data on every run and reuses `df` for four different frames. The layout was satisfied; the code still rotted. oryxflow's structure is different because it's the *shape of the code*, not just its filing:

- **The pipeline is a graph, not a top-to-bottom script.** Each step declares its dependencies (`@oryxflow.requires(...)`). "Runs top to bottom instead of as a DAG" — the single most-cited data-science code sin — isn't available to write. And because only what changed recomputes, the reproducible version is *also* the fast one, so nobody is tempted back to the flat script to save time.
- **Naming a task forces decomposition.** A task is named for the output it produces — a noun like `FeatureMatrix` or `TrainedModel`, not a verb like `GetData`. To add a step you must say what it produces and what it consumes, which drags "an absence of functions" toward its opposite: the work arrives already cut into named, single-purpose pieces.
- **Separation of concerns is real, not suggested.** Config in `cfg.py`, parameters in `flow_params.py`, definitions in `tasks.py`, the workflow instance in `flow.py`, execution in `run.py`, analysis in `visualize.py`. These are the seams the imports run along (`from flow import flow`, everywhere) — you change behavior by editing the layer that owns it, which is what keeps the thousand-line everything-script from forming.
- **Outputs are durable artifacts, not scrollback.** Every task saves a typed result you reload by asking for the task that made it. The headline number lives in a file you can re-open next month, not a printed line that scrolled away.

None of this is novel — it's the ordinary discipline the best-practice checklists recommend, with one difference that matters: it's enforced by the shape of the thing rather than left to whether you (or the agent) remember to be disciplined this afternoon.

## Bring an existing mess into it

If you already have a notebook or script that works, you don't rewrite it in one risky pass. `/oryxflow:migrate` restructures an ad-hoc analysis into cached, parameterized tasks **one step at a time**, so at every point you have a working pipeline — never a half-rewritten one. The end state is reproducible and lineage-tracked, and the expensive steps stop rerunning on every edit. See [From notebook to a reproducible, cached pipeline](https://docs.oryxflow.dev/blog/reproducibility/notebook-to-reproducible-pipeline/index.md) for what that looks like step by step.

## It grows with the project

Most projects stay flat — one `tasks.py`, `run.py`, `flow.py`, `flow_params.py` — and that's the right shape for the majority that stay research-only. When a project *does* grow, the plugin nudges you along a graduated path instead of over-building up front:

1. **Naming families** — broad-to-narrow prefixes (`Features*`, `Model*`) cluster related tasks so a branch reads together.
1. **Comment section-headers** divide phases within one file — carrying it well past 500 lines before any split.
1. **Split into modules** (`tasks_features.py`, `tasks_model.py`) only when the file is genuinely long or a separable subsystem appears. This is cache-safe: a task's identity is its class name, not its module path, so *moving* a task doesn't invalidate its cache.
1. **A production tier** — frozen parameters and selective resets — appears when work goes to prod, kept separate from the fast experiment loop.

The agent offers the next structural step on a concrete trigger (a genuinely long file, going to prod, a separable subsystem) — restructuring as the project earns it, which is exactly the discipline data scientists tend to skip.

## Takeaway

The template gives you the filing cabinet. Making the structure *load-bearing* — a shape the code has to take — gets you a project that stays well-formed even when the thing writing it is an agent drawn to the mess. It won't make the analysis correct; a clean, reproducible, well-organized pipeline can still answer the wrong question. But it removes the whole class of "which cell made this, and can I run it again?" — which is a large, real fraction of what trustworthy AI-assisted work requires.

**Read next**

- [Stop AI data analysis turning into a mess](https://docs.oryxflow.dev/docs/claude-plugin/coding-standards/index.md) — the conventions that ride alongside the layout.
- [Trustworthy AI data analysis](https://docs.oryxflow.dev/docs/claude-plugin/trust/index.md) — verifying what the agent actually did.
- [Plugin commands](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) — `init-project`, `migrate`, and the rest.
- [Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md) — the cache and lineage the structure sits on.

# Stop AI data analysis from turning into a mess

**A data-analysis project rarely starts messy — it turns messy.** One clean notebook becomes a tangle of drifting column names, copy-pasted cleaning, and functions that never got written; hand the work to an AI agent and the drift arrives *faster*. A coding standard only prevents that if it's present the moment the code is written. The oryxflow Claude Code plugin ships its data-science conventions *where the agent works* — loaded into context as it edits, phrased as rules with the reasoning attached — so they're applied as the line is typed, not reviewed afterward in a wiki nobody opens.

These aren't motivational-poster generalities. They're specific, opinionated defaults that keep a project readable and safe as it grows, and they matter more with an AI in the loop: an agent will happily produce four variants of the same column name across three files unless something steers it otherwise.

## Naming: one canonical name, carried end to end

Every rename is a mapping a reader has to hold and can mis-apply — and a round-trip (raw → display → raw) is a classic wrong-column bug. The convention minimizes those layers:

- **One canonical `snake_case` name per column**, renamed once at ingestion and never re-aliased downstream. `yield_dividend`, not `div_yld` in one file and `Dividend Yield` in another.
- **Order tokens broad → narrow**, so a family shares a leading prefix and clusters: `yield_dividend`, `yield_earnings`, `yield_fcf` — not `dividend_yield`. The same rule names tasks (`FundamentalsLeadLag`, not `LeadLagAnalysis`) and variables (`df_returns_gross`).
- **The operation goes last, as a suffix** — `_yoy`, `_ma4`, `_lag1`, and stats like `_avg` / `_min` / `_max`. Never a leading `avg_` / `pct_` / `n_` prefix, which hides the metric and scatters a family under the unit instead of the subject.
- **Pretty labels live only at the plotting layer** — Title-Case axis and legend names in `viz/`, never a second data-level rename.

The payoff: a whole count/ratio family shares one stem, so the arithmetic reads straight off the names — `coverage_pct = covered / total` — and a searched-for metric is where you'd expect it.

## Code organization: group by subject

The flat starter stops scaling once a project has many tasks plus reusable helpers and plots. The convention groups supporting code by the *subject* it concerns — a task, a dataset, or a cross-cutting concept — mirroring how the pipeline itself is keyed on tasks:

- **`eda/<subject>/`** — read-only exploration and verification probes (this is where "just checking X" goes, instead of an inline one-off). A probe writes no pipeline artifact.
- **`utils/<subject>.py`** — one subject's helpers; a helper shared by two or more subjects moves to a concept module (`utils/geo.py`). Only truly generic helpers live in `__init__.py`, so it never becomes a junk drawer.
- **`viz/<subject>.py`** — that subject's figures. The starter `visualize.py` graduates here once you have per-subject or shared plotting.

Files are named for the specific thing they do, dropping the redundant subject token (`eda/sales/verify_coercion.py`, not `verify_sales.py`) — so the second probe in a folder always has a clear home.

## Docstrings are the documentation

In an oryxflow project the code *is* the pipeline doc — there's no separate description that can drift out of sync. So a task's docstring isn't a throwaway one-liner; it states what the task produces and its input-to-output contract:

```
@oryxflow.requires(DataSales)
class MonthlyRevenue(oryxflow.tasks.TaskPqPandas):
    """Revenue aggregated to one row per (region, month).

    In:  raw order lines (from DataSales).
    Out: one row per region-month; revenue_gross, revenue_net, order_count.
         Null revenue_net where a refund post-dates the close.
    """
```

That single habit means "what does this pipeline do?" is answered by reading the code, not by re-scanning the whole project or trusting the agent's memory of a past session.

## Code style that avoids quiet breakage

A few defaults prevent the failure modes that don't announce themselves:

- **Log with the task logger, not `print`.** The engine already logs task scheduling, timing, and completion for free; inside a task, log the domain signal you'd watch live — shapes, drop rates, headline metrics — through the task logger so it survives into the lineage record.
- **No `try/except` that swallows errors.** Let code fail natively so a real problem surfaces instead of hiding behind a plausible fallback.
- **Use off-the-shelf libraries.** Reach for statsmodels / scipy / sklearn rather than hand-rolling the math; a broken import means a broken environment to fix, not a cue to reimplement around it.
- **ASCII-only in code and output**, so nothing breaks on a Windows console.

## Check an existing codebase against them

`/oryxflow:check-standards` reviews names, style, and docstrings against the house standards and reports what's drifted — so a project the agent (or you) built quickly can be brought back into line without a manual audit. It's the maintenance counterpart to the conventions the skill applies automatically while writing new code.

## Why loaded-in-context beats a style guide

A style guide reviewed after the fact catches a fraction of what it should and annoys everyone. A convention loaded into the agent's working context — phrased as an imperative plus the failure it prevents — gets applied as the line is written. That's the difference between a standard that's aspirational and one that's *operative*, and it's why these conventions live next to the skill rather than in a document someone would have to remember to open.

The point isn't that these particular rules are the one true style. It's that getting pro-level structure *by construction* — as the code is typed — rather than by after-the-fact vigilance is what keeps an AI-built project readable, navigable, and safe to extend as it grows.

**Read next**

- [Data-science project structure](https://docs.oryxflow.dev/docs/claude-plugin/project-structure/index.md) — the layout these conventions ride alongside.
- [Trustworthy AI data analysis](https://docs.oryxflow.dev/docs/claude-plugin/trust/index.md) — verifying the work the conventions help produce.
- [Why library + plugin is a matched pair](https://docs.oryxflow.dev/docs/claude-plugin/why/index.md) — why the conventions ship with the skill.
- [Plugin commands](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) — including `check-standards`.

# Why the oryxflow library and Claude Code plugin are a matched pair for AI data analysis

The oryxflow library and the Claude Code plugin solve two halves of trustworthy AI data analysis, and they're most valuable together. This page is the honest account of who carries what — and why the pairing gets *stronger* as a data-science project grows, when most tooling buckles.

## The agent's real weakness: invisible state

A coding agent writing data-science code isn't tripped up by syntax. It's tripped up by state it can't see across turns:

- **Stale intermediates** — it writes a cached file early, changes the code later, forgets to regenerate, and trains on stale data. No error is raised; the numbers are just wrong.
- **Expensive recompute in the inner loop** — its whole style is run → observe → edit → run, and in a plain script every loop recomputes the slow steps.
- **Path and load bookkeeping** — it hardcodes output paths, loses track of what's saved where, and occasionally loads the wrong file into the wrong step.

These are *memory* problems, not intelligence problems — and they're structural, because the agent's recollection of "did I already run this, is it still valid?" degrades over a long session.

## The division of labor

The library and the plugin split the problem cleanly:

**The library carries the state the agent is structurally bad at holding.**

- The dependency graph is *data* (your `requires()` edges), not something the agent must remember.
- Re-running is cheap and correct by default — completed tasks load from cache; a code change reruns exactly the affected tasks and everything downstream, automatically.
- There are no filenames to get wrong — results are addressed by task identity.
- Every run appends a queryable lineage record of what ran, with which code and params, and why.

**The plugin carries the disciplines the library can't enforce from inside.**

- Start each session by reading cache state (`events.print_status()`) before assuming anything.
- After every edit, **verify the rerun actually happened** — a `ran=0` after a change means the edit landed somewhere oryxflow can't detect (a data file, a dynamic call), and the fix is `reset()`.
- Answer every staleness or expensive-recompute warning with the right exit.
- Select the right named input when a task has multiple parents and outputs — a spot agents otherwise fumble.

The library removes the state-tracking; the skill supplies the verification. Neither half is complete alone: a library can't make the agent *check* its work, and a prompt can't give the agent a durable cache and lineage log.

## What neither half does — and why that's stated plainly

Being trustworthy means being clear about the boundary. oryxflow guarantees a result was produced by the code and inputs it records — **not** that the result is *correct*. A perfectly reproducible pipeline can still hand you a wrong number: a many-to-many join that should be many-to-one, a leaked test set, a timezone shift that moves rows a day. None of those raise; each completes cleanly and prints something plausible.

Those are caught by **habit, not machinery** — validate the merge, check the frame's shape and null counts, quote every number from a saved artifact. The plugin ships exactly those habits *alongside* the code, at the moment you're writing the task rather than in a review afterward. The judgment calls — is this method right for the question, is this effect within noise — remain yours. See [What caching does not protect against](https://docs.oryxflow.dev/docs/managing-workflows/#what-caching-does-not-protect-against) for the full boundary.

## Why the pairing scales *up*

A first look at a dataset needs no task graph, and the plugin doesn't force one on you: exploration gets a home of its own — read-only probes under `eda/<subject>/`, each documenting the question it answers — and `/oryxflow:migrate` promotes a probe into a cached task the day it turns out to be load-bearing. So you can start with simple scripts and grow, rather than deciding your architecture before you know what the analysis is.

The value then inverts, super-linearly, as a project gains **depth** (a silent stale intermediate near the top corrupts everything below), **expensive nodes** (the cache is the difference between a tractable inner loop and one where every experiment costs minutes or dollars), and **experiment matrices** (hand-managing output paths across a Cartesian product is hopeless). Those are exactly the traits that make an AI agent error-prone without a DAG — and exactly where the library-plus-plugin pairing helps most.

That's the whole point: this pairing gets *more* valuable as complexity rises, not less — the opposite of tooling that collapses under scale. And because the same skill covers both ends, there's no cliff between the exploratory script and the pipeline it becomes.

## Get started

```
/plugin marketplace add https://github.com/oryxintel/oryxflow-claude-plugin.git
/plugin install oryxflow@oryxflow
```

- **[Commands](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md)** — scaffold, migrate, and maintain a project.
- **[Trustworthy AI data analysis](https://docs.oryxflow.dev/docs/claude-plugin/trust/index.md)** — how the agent's work becomes cheap to verify.
- **[Keep a data-science project clean](https://docs.oryxflow.dev/docs/claude-plugin/project-structure/index.md)** — the load-bearing scaffold.
- **[Stop AI data analysis turning into a mess](https://docs.oryxflow.dev/docs/claude-plugin/coding-standards/index.md)** — the conventions that ship with the skill.
- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** — the positioning in full.
- **[Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md)** — the cache, lineage, and invalidation the plugin drives.
# Reference

# oryxflow glossary

Short, plain-language definitions of the terms used across these docs and in oryxflow itself. Each entry links to where the concept is covered in full.

## What is a reproducible pipeline?

A reproducible pipeline is an analysis where every output can be traced to the exact code, parameters, and inputs that produced it — so you can recreate any result on demand. oryxflow makes your data-science work reproducible by default: each step is a cached task tied to its inputs and code. See [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md).

## What is data lineage (provenance)?

Data lineage — also called provenance — is the record of what ran, when, with which parameters and code, and why it recomputed. It turns "is this result stale?" and "was it built with the current code?" into queries instead of guesses. oryxflow writes this record automatically as you run. See [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md).

## What is code-change invalidation?

Code-change invalidation means a pipeline reruns a step when its **code** changes — not only when its parameters or data change. oryxflow compares what your code does, not how it's written, so edits to comments or formatting are ignored, while a real logic change reruns that step and everything downstream. See [Managing workflows](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation).

## What is a task DAG?

A task DAG (directed acyclic graph) is your analysis expressed as steps — tasks — connected by their dependencies, with no cycles. oryxflow runs the tasks in dependency order, skips any whose output already exists, and reruns only what a change affects. You declare each task; the engine works out the order. See [Writing tasks](https://docs.oryxflow.dev/docs/tasks/index.md).

## What is the difference between a Claude Code plugin, a skill, and a slash command?

They're nested, not competing. A **plugin** is the installable package you add to Claude Code. A **skill** is one thing a plugin can contain — a bundle of instructions and conventions the agent loads *on its own* when the context matches, without you invoking anything. A **slash command** is an action you invoke explicitly, like `/oryxflow:init-project`. A plugin can also ship hooks and other pieces.

oryxflow ships a plugin containing the `oryxflow` **skill** (the data-science conventions and cache disciplines, which auto-activate when you edit pipeline files) plus a handful of **slash commands** you call deliberately — `/oryxflow:init-project` to scaffold, `/oryxflow:migrate` to convert an existing script or notebook, and a few more. It is **not** an MCP server. In practice you install the plugin once and the skill just works in the background while you describe the analysis you want. See [Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md) and [the command reference](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md).

## What is a cached intermediate?

A cached intermediate is the saved output of a pipeline step, reused instead of recomputed on the next run. oryxflow caches every task's output by its identity, so re-running a pipeline only pays for what actually changed — no hand-rolled pickle files to manage or accidentally leave stale. See [Task I/O formats](https://docs.oryxflow.dev/docs/targets/index.md).

## What is a parameter sweep?

A parameter sweep runs the same analysis across many configurations — model × features × window — to compare results. oryxflow computes each shared upstream step once and reruns only what each configuration changes, so a sweep costs far less than re-running everything per combination. See [Parameter sweeps without rerunning](https://docs.oryxflow.dev/blog/caching/parameter-sweeps-without-rerunning/index.md).

## What is a task id and task family?

Every oryxflow task has a **task family** (its name) and a **task id** that also encodes its parameters, so two runs with different parameters are distinct cached outputs you can tell apart. This is how oryxflow keeps results from different configurations from overwriting each other. See [Parameters](https://docs.oryxflow.dev/docs/advparam/index.md).

# Security & supply chain

This page is for the security and platform teams who vet what gets past a corporate package firewall (Sonatype Nexus, JFrog Curation, Socket, Snyk, and similar). It gathers oryxflow's provenance, its clean-vulnerability status, and the public entries you can check yourself, so approving oryxflow doesn't need a back-and-forth.

## At a glance

- **Signed provenance on every file.** From 26.7.21 onward, releases are built and uploaded by this project's GitHub Actions workflow — no maintainer's laptop, no long-lived upload token — and each file on PyPI carries a PyPI-recorded attestation (PEP 740 / Sigstore) tying it to the exact repository and workflow that produced it. See [Provenance](#provenance).
- **No known vulnerabilities.** oryxflow has no CVEs or advisories in any public database.
- **Named publisher, permissive licence.** Published by Oryx Intelligence LLC under the MIT licence, from a public source repository.
- **Small, tier-one dependency set.** Five direct dependencies, all mainstream and widely audited: `pandas`, `pyarrow`, `openpyxl`, `markdown`, `loguru`.
- **Clean own-code scores.** Socket rates oryxflow's own package 100 on Vulnerability, Quality, Maintenance and Licence. The only open alert is inherited from a dependency, not oryxflow's code — see [Known dependency alerts](#known-dependency-alerts).

## Verify it yourself

Every claim above is checkable from a public source — no need to take our word for it:

| Source                | What it tells you                                   | Link                                                                                   |
| --------------------- | --------------------------------------------------- | -------------------------------------------------------------------------------------- |
| **PyPI**              | Publisher, versions, licence, release history       | [pypi.org/project/oryxflow](https://pypi.org/project/oryxflow/)                        |
| **OSV.dev**           | Aggregated vulnerability database (GHSA, PyPA, NVD) | [osv.dev](https://osv.dev/list?ecosystem=PyPI&q=oryxflow)                              |
| **Snyk**              | Vulnerabilities, licence, health score              | [security.snyk.io/package/pip/oryxflow](https://security.snyk.io/package/pip/oryxflow) |
| **deps.dev** (Google) | Dependency graph, advisories, scorecard             | [deps.dev/pypi/oryxflow](https://deps.dev/pypi/oryxflow)                               |
| **Socket**            | Behavioural supply-chain analysis                   | [socket.dev/pypi/package/oryxflow](https://socket.dev/pypi/package/oryxflow)           |
| **Source**            | Full source, issues, history                        | [github.com/oryxintel/oryxflow](https://github.com/oryxintel/oryxflow)                 |

## Provenance

Every file published from 26.7.21 onward can be traced back to the commit and CI run that built it, and PyPI verified that link at upload time.

On any file's detail page on PyPI you will see the publisher recorded as the GitHub repository `oryxintel/oryxflow` and the workflow `release.yml`. Nothing is uploaded by hand, so there is no API token that could be stolen and used to publish a file we didn't build.

To check it without a browser, ask PyPI for a file's attestation directly:

```
curl https://pypi.org/integrity/oryxflow/26.7.21/oryxflow-26.7.21-py3-none-any.whl/provenance
```

The response names the publisher (`repository`, `workflow`, `environment`) and carries the Sigstore signing material behind it. Releases before 26.7.21 were uploaded manually and have no attestation — that is expected, not a tampering signal.

## Dependencies

oryxflow keeps its dependency footprint deliberately small. The core install pulls in only:

- **pandas**, **pyarrow** — dataframes and Parquet/Arrow I/O
- **openpyxl** — Excel I/O
- **markdown** — Markdown report output
- **loguru** — logging (off by default; see [Logging](https://docs.oryxflow.dev/docs/logging/index.md))

Everything else — cloud storage (`gcs`/`s3`), Dask, flow export — is an opt-in [install extra](https://docs.oryxflow.dev/docs/installation/index.md), so you only bring in what you use.

## Known dependency alerts

Behavioural scanners occasionally raise a flag on a *dependency*. For completeness, here is the one you may see, and why it is not a concern:

loguru — `Obfuscated code` / `AI-detected potential security risk`

Both alerts point at the same file in **loguru** (not oryxflow): `loguru/_recattrs.py`, the `RecordException` class. loguru makes log records picklable so they survive being passed between processes (loguru's `enqueue=True` option), and the scanner flags that nested-pickle pattern as a *potential* deserialization risk.

This is a false positive for oryxflow's usage: the risk only exists if an attacker can feed **untrusted pickle bytes** into the deserializer, and oryxflow never deserializes untrusted input through loguru. The alert's own notes confirm the module "contains no explicit exfiltration or network behavior." loguru is one of the most-downloaded Python packages and is widely audited.

If your policy engine requires a disposition, this alert can be accepted/ignored with the justification: *"False positive — trusted upstream dependency (loguru); no untrusted deserialization path in use."*

## Requesting an allowlist entry

New packages with low download history are sometimes auto-quarantined by reputation/age policies even when their security record is clean. If oryxflow is held by such a policy in your environment, point your reviewer at this page and the public sources above, or reach out at [dev@oryxintel.com](mailto:dev@oryxintel.com) and we'll help provide whatever your process needs.

# API Reference

Auto-generated from the source docstrings. For task-writing guidance start with the [Guide](https://docs.oryxflow.dev/docs/tasks/index.md); this page is the exhaustive symbol reference.

## Top-level API

The public API re-exported from `oryxflow` — `run`, `preview`, `Workflow`, `WorkflowMulti`, the `requires` / `inherits` decorators, the `Parameter` types, and the `invalidate_*` / `enable_*` helpers.

## Engine — `oryxflow.core`

### oryxflow.core

Mini-engine: a small, self-contained execution engine for oryxflow. Provides the `Task` base class, target bases, `flatten`/`getpaths`, deterministic task ids, the `inherits`/`requires` decorators, `find_deps` and a sequential `build`.

The execution engine is **sequential**: the DAG is run in dependency order in-process. The `workers` argument is accepted for API compatibility but ignored.

#### Register

Bases: `type`

Minimal metaclass providing a class-level `task_family` property and instance memoization.

A property defined directly on class:`Task` is only invoked for instances; reading `SomeTaskClass.task_family` needs this metaclass property (used eg by `WorkflowMulti`).

`__call__` memoizes instances so two `Cls(**same_params)` calls return the identical object (and `__init__` runs only on the first), preserving the instance identity that `Workflow`'s path/flow propagation depends on.

Source code in `oryxflow/core.py`

```
class Register(type):
    """
    Minimal metaclass providing a class-level ``task_family`` property and instance memoization.

    A property defined directly on :py:class:`Task` is only invoked for instances; reading
    ``SomeTaskClass.task_family`` needs this metaclass property (used eg by ``WorkflowMulti``).

    ``__call__`` memoizes instances so two ``Cls(**same_params)`` calls return the identical
    object (and ``__init__`` runs only on the first), preserving the instance identity that
    ``Workflow``'s path/flow propagation depends on.
    """

    @property
    def task_family(cls):
        return cls.__name__

    def __call__(cls, *args, **kwargs):
        try:
            params = cls.get_params()
            param_values = cls.get_param_values(params, args, kwargs)
            param_objs = dict(params)
            key = (cls, tuple((n, param_objs[n].serialize(v)) for n, v in param_values))
            hash(key)
        except Exception:
            # On any trouble building a stable key, fall back to a fresh (uncached) instance.
            return super().__call__(*args, **kwargs)

        inst = _instance_cache.get(key)
        if inst is None:
            inst = super().__call__(*args, **kwargs)
            _instance_cache[key] = inst
        return inst
```

#### Task

Base class for all oryxflow tasks.

Subclasses declare class:`~oryxflow.parameter.Parameter` members and override meth:`run`, meth:`requires` and meth:`output`.

Source code in `oryxflow/core.py`

```
class Task(metaclass=Register):
    """
    Base class for all oryxflow tasks.

    Subclasses declare :py:class:`~oryxflow.parameter.Parameter` members and override
    :py:meth:`run`, :py:meth:`requires` and :py:meth:`output`.
    """

    # Explicit code identity token (str or int), OPT-IN: when set it is the authority
    # for this task's own logic -- bump it to rerun this task and everything downstream
    # (code edits without a bump only warn). None (the default) with
    # settings.code_version_auto (the default) derives the token automatically from the
    # AST hash of the task's module + repo-local imports, so logic edits rerun with no
    # attribute to maintain; with code_version_auto=False, None leaves caching purely
    # parameter-based.
    code_version = None

    @classmethod
    def get_params(cls):
        """Return all ``(name, Parameter)`` pairs for this task, in declaration order."""
        params = []
        for param_name in dir(cls):
            param_obj = getattr(cls, param_name)
            if not isinstance(param_obj, Parameter):
                continue
            params.append((param_name, param_obj))

        params.sort(key=lambda t: t[1]._counter)
        return params

    @classmethod
    def get_param_names(cls, include_significant=False):
        """Return parameter names. ``include_significant=True`` returns all params."""
        return [name for name, p in cls.get_params() if include_significant or p.significant]

    @classmethod
    def get_param_values(cls, params, args, kwargs):
        """
        Resolve parameter values from positional args, kwargs and defaults.

        :param params: list of ``(param_name, Parameter)``
        :param args: positional arguments
        :param kwargs: keyword arguments
        :returns: list of ``(name, value)`` tuples, one per parameter
        """
        result = {}
        params_dict = dict(params)
        task_family = cls.get_task_family()
        exc_desc = '%s[args=%s, kwargs=%s]' % (task_family, args, kwargs)

        # Positional arguments.
        positional_params = [(n, p) for n, p in params if p.positional]
        for i, arg in enumerate(args):
            if i >= len(positional_params):
                raise UnknownParameterException(
                    '%s: takes at most %d parameters (%d given)' % (exc_desc, len(positional_params), len(args)))
            param_name, param_obj = positional_params[i]
            result[param_name] = param_obj.normalize(arg)

        # Keyword arguments.
        for param_name, arg in kwargs.items():
            if param_name in result:
                raise DuplicateParameterException(
                    '%s: parameter %s was already set as a positional parameter' % (exc_desc, param_name))
            if param_name not in params_dict:
                raise UnknownParameterException('%s: unknown parameter %s' % (exc_desc, param_name))
            result[param_name] = params_dict[param_name].normalize(arg)

        # Defaults for anything still unset.
        for param_name, param_obj in params:
            if param_name not in result:
                if not param_obj.has_task_value():
                    raise MissingParameterException(
                        "%s: requires the '%s' parameter to be set" % (exc_desc, param_name))
                result[param_name] = param_obj.task_value()

        return [(param_name, result[param_name]) for param_name, param_obj in params]

    def __init__(self, *args, **kwargs):
        params = self.get_params()
        param_values = self.get_param_values(params, args, kwargs)

        for key, value in param_values:
            setattr(self, key, value)

        self.param_kwargs = dict(param_values)

        self.task_id = task_id_str(self.get_task_family(), self.to_str_params(only_significant=True))
        self.__hash = hash(self.task_id)

    @property
    def task_family(self):
        """The task family (class name) of this task instance."""
        return self.__class__.__name__

    @property
    def logger(self):
        """Contextual logger for task authors; auto-tagged with task identity.

        Lives in the ``oryxflow`` namespace, so it is silent until
        ``oryxflow.enable_logging()`` is called.
        """
        if getattr(self, "_logger", None) is None:
            self._logger = TaskLogger(task_id=self.task_id, task_family=self.task_family)
        return self._logger

    @classmethod
    def get_task_family(cls):
        """The task family (class name) for this class."""
        return cls.__name__

    def to_str_params(self, only_significant=False, only_public=False):
        """
        Convert parameters to a ``{name: serialized_value}`` dict.

        :param bool only_significant: only include parameters marked significant.
        :param bool only_public: accepted for API compatibility; visibility is not modeled.
        """
        params_str = {}
        params = dict(self.get_params())
        for param_name, param_value in self.param_kwargs.items():
            if (not only_significant) or params[param_name].significant:
                params_str[param_name] = params[param_name].serialize(param_value)
        return params_str

    def clone(self, cls=None, **kwargs):
        """
        Create a new task instance from this one, overriding some args.

        Parameters common to this task and ``cls`` are carried over; ``kwargs`` take precedence.
        """
        if cls is None:
            cls = self.__class__

        new_k = {}
        for param_name, param_class in cls.get_params():
            if param_name in kwargs:
                new_k[param_name] = kwargs[param_name]
            elif hasattr(self, param_name):
                new_k[param_name] = getattr(self, param_name)

        return cls(**new_k)

    def __hash__(self):
        return self.__hash

    def __repr__(self):
        params = dict(self.get_params())
        repr_parts = []
        for param_name, param_value in self.param_kwargs.items():
            if params[param_name].significant:
                repr_parts.append('%s=%s' % (param_name, params[param_name].serialize(param_value)))
        return '{}({})'.format(self.get_task_family(), ', '.join(repr_parts))

    def __eq__(self, other):
        return self.__class__ == other.__class__ and self.task_id == other.task_id

    def complete(self):
        """``True`` if all of this task's outputs exist.

        Note: ``TaskData`` OVERRIDES this to ALSO require the stored code
        fingerprint to match the current one (``TaskData._code_ok``), so a
        ``code_version`` bump makes a task incomplete and forces a rerun -- the
        fingerprint is authoritative here, not merely advisory. The AST
        source-hash is a SEPARATE, warn-only advisory (fires when code changed
        but ``code_version`` did not); it does not gate completeness.
        """
        import warnings
        outputs = flatten(self.output())
        if len(outputs) == 0:
            warnings.warn(
                "Task %r without outputs has no custom complete() method" % self,
                stacklevel=2,
            )
            return False
        return all(output.exists() for output in outputs)

    def output(self):
        """The output Target(s) this task produces. Default: none."""
        return []

    def requires(self):
        """The Task(s) this task depends on. Default: none."""
        return []

    def input(self):
        """The outputs of the tasks returned by :py:meth:`requires`."""
        return getpaths(self.requires())

    def deps(self):
        """Flattened list of required tasks."""
        return flatten(self.requires())

    @property
    def _code_fingerprint(self):
        """Recursive code identity, compared against the state store by
        TaskData.complete() -- a mismatch forces a rerun (authoritative). The own
        token is the explicit ``code_version`` when declared; otherwise, with
        ``settings.code_version_auto`` (the default), the AST hash of the task's own
        class plus the repo-local symbols it transitively references, so logic edits
        rerun automatically -- and edits to unrelated siblings in the same file don't.
        None when neither applies here or upstream (feature inert).
        For tasks WITH an explicit code_version the AST source-hash stays a
        warn-only advisory and never gates completeness."""
        # Not memoized: instances are process-long-lived via the Register cache, so a
        # cached fingerprint would go stale on runtime bumps; recompute is a cheap md5
        # over a small DAG (task_hashes carries its own mtime-revalidated cache).
        dep_fps = [d._code_fingerprint for d in self.deps()]
        own = self.code_version
        if own is None:
            from oryxflow import settings as _settings_
            if _settings_.code_version_auto:
                from oryxflow import codehash as _codehash_
                h = _codehash_.task_code_hash(self)
                own = 'auto:{}'.format(h) if h is not None else None
        if own is None and all(f is None for f in dep_fps):
            return None
        parts = [self.task_family, str(own)] + sorted(f or '' for f in dep_fps)
        return hashlib.md5('|'.join(parts).encode('utf-8')).hexdigest()[:16]

    def run(self):
        """The task computation, to be overridden in a subclass."""
        pass
```

##### task_family

```
task_family
```

The task family (class name) of this task instance.

##### logger

```
logger
```

Contextual logger for task authors; auto-tagged with task identity.

Lives in the `oryxflow` namespace, so it is silent until `oryxflow.enable_logging()` is called.

##### get_params

```
get_params()
```

Return all `(name, Parameter)` pairs for this task, in declaration order.

Source code in `oryxflow/core.py`

```
@classmethod
def get_params(cls):
    """Return all ``(name, Parameter)`` pairs for this task, in declaration order."""
    params = []
    for param_name in dir(cls):
        param_obj = getattr(cls, param_name)
        if not isinstance(param_obj, Parameter):
            continue
        params.append((param_name, param_obj))

    params.sort(key=lambda t: t[1]._counter)
    return params
```

##### get_param_names

```
get_param_names(include_significant=False)
```

Return parameter names. `include_significant=True` returns all params.

Source code in `oryxflow/core.py`

```
@classmethod
def get_param_names(cls, include_significant=False):
    """Return parameter names. ``include_significant=True`` returns all params."""
    return [name for name, p in cls.get_params() if include_significant or p.significant]
```

##### get_param_values

```
get_param_values(params, args, kwargs)
```

Resolve parameter values from positional args, kwargs and defaults.

:param params: list of `(param_name, Parameter)` :param args: positional arguments :param kwargs: keyword arguments :returns: list of `(name, value)` tuples, one per parameter

Source code in `oryxflow/core.py`

```
@classmethod
def get_param_values(cls, params, args, kwargs):
    """
    Resolve parameter values from positional args, kwargs and defaults.

    :param params: list of ``(param_name, Parameter)``
    :param args: positional arguments
    :param kwargs: keyword arguments
    :returns: list of ``(name, value)`` tuples, one per parameter
    """
    result = {}
    params_dict = dict(params)
    task_family = cls.get_task_family()
    exc_desc = '%s[args=%s, kwargs=%s]' % (task_family, args, kwargs)

    # Positional arguments.
    positional_params = [(n, p) for n, p in params if p.positional]
    for i, arg in enumerate(args):
        if i >= len(positional_params):
            raise UnknownParameterException(
                '%s: takes at most %d parameters (%d given)' % (exc_desc, len(positional_params), len(args)))
        param_name, param_obj = positional_params[i]
        result[param_name] = param_obj.normalize(arg)

    # Keyword arguments.
    for param_name, arg in kwargs.items():
        if param_name in result:
            raise DuplicateParameterException(
                '%s: parameter %s was already set as a positional parameter' % (exc_desc, param_name))
        if param_name not in params_dict:
            raise UnknownParameterException('%s: unknown parameter %s' % (exc_desc, param_name))
        result[param_name] = params_dict[param_name].normalize(arg)

    # Defaults for anything still unset.
    for param_name, param_obj in params:
        if param_name not in result:
            if not param_obj.has_task_value():
                raise MissingParameterException(
                    "%s: requires the '%s' parameter to be set" % (exc_desc, param_name))
            result[param_name] = param_obj.task_value()

    return [(param_name, result[param_name]) for param_name, param_obj in params]
```

##### get_task_family

```
get_task_family()
```

The task family (class name) for this class.

Source code in `oryxflow/core.py`

```
@classmethod
def get_task_family(cls):
    """The task family (class name) for this class."""
    return cls.__name__
```

##### to_str_params

```
to_str_params(only_significant=False, only_public=False)
```

Convert parameters to a `{name: serialized_value}` dict.

:param bool only_significant: only include parameters marked significant. :param bool only_public: accepted for API compatibility; visibility is not modeled.

Source code in `oryxflow/core.py`

```
def to_str_params(self, only_significant=False, only_public=False):
    """
    Convert parameters to a ``{name: serialized_value}`` dict.

    :param bool only_significant: only include parameters marked significant.
    :param bool only_public: accepted for API compatibility; visibility is not modeled.
    """
    params_str = {}
    params = dict(self.get_params())
    for param_name, param_value in self.param_kwargs.items():
        if (not only_significant) or params[param_name].significant:
            params_str[param_name] = params[param_name].serialize(param_value)
    return params_str
```

##### clone

```
clone(cls=None, **kwargs)
```

Create a new task instance from this one, overriding some args.

Parameters common to this task and `cls` are carried over; `kwargs` take precedence.

Source code in `oryxflow/core.py`

```
def clone(self, cls=None, **kwargs):
    """
    Create a new task instance from this one, overriding some args.

    Parameters common to this task and ``cls`` are carried over; ``kwargs`` take precedence.
    """
    if cls is None:
        cls = self.__class__

    new_k = {}
    for param_name, param_class in cls.get_params():
        if param_name in kwargs:
            new_k[param_name] = kwargs[param_name]
        elif hasattr(self, param_name):
            new_k[param_name] = getattr(self, param_name)

    return cls(**new_k)
```

##### complete

```
complete()
```

`True` if all of this task's outputs exist.

Note: `TaskData` OVERRIDES this to ALSO require the stored code fingerprint to match the current one (`TaskData._code_ok`), so a `code_version` bump makes a task incomplete and forces a rerun -- the fingerprint is authoritative here, not merely advisory. The AST source-hash is a SEPARATE, warn-only advisory (fires when code changed but `code_version` did not); it does not gate completeness.

Source code in `oryxflow/core.py`

```
def complete(self):
    """``True`` if all of this task's outputs exist.

    Note: ``TaskData`` OVERRIDES this to ALSO require the stored code
    fingerprint to match the current one (``TaskData._code_ok``), so a
    ``code_version`` bump makes a task incomplete and forces a rerun -- the
    fingerprint is authoritative here, not merely advisory. The AST
    source-hash is a SEPARATE, warn-only advisory (fires when code changed
    but ``code_version`` did not); it does not gate completeness.
    """
    import warnings
    outputs = flatten(self.output())
    if len(outputs) == 0:
        warnings.warn(
            "Task %r without outputs has no custom complete() method" % self,
            stacklevel=2,
        )
        return False
    return all(output.exists() for output in outputs)
```

##### output

```
output()
```

The output Target(s) this task produces. Default: none.

Source code in `oryxflow/core.py`

```
def output(self):
    """The output Target(s) this task produces. Default: none."""
    return []
```

##### requires

```
requires()
```

The Task(s) this task depends on. Default: none.

Source code in `oryxflow/core.py`

```
def requires(self):
    """The Task(s) this task depends on. Default: none."""
    return []
```

##### input

```
input()
```

The outputs of the tasks returned by meth:`requires`.

Source code in `oryxflow/core.py`

```
def input(self):
    """The outputs of the tasks returned by :py:meth:`requires`."""
    return getpaths(self.requires())
```

##### deps

```
deps()
```

Flattened list of required tasks.

Source code in `oryxflow/core.py`

```
def deps(self):
    """Flattened list of required tasks."""
    return flatten(self.requires())
```

##### run

```
run()
```

The task computation, to be overridden in a subclass.

Source code in `oryxflow/core.py`

```
def run(self):
    """The task computation, to be overridden in a subclass."""
    pass
```

#### Target

Minimal abstract target base.

Source code in `oryxflow/core.py`

```
class Target:
    """Minimal abstract target base."""

    def exists(self):
        raise NotImplementedError
```

#### LocalTarget

Bases: `Target`

Tiny local target base.

`self.path` is stored as-is (NOT coerced to `str`); subclasses (`CacheTarget`/`_LocalPathTarget`) override the rest and normalize the path.

Source code in `oryxflow/core.py`

```
class LocalTarget(Target):
    """
    Tiny local target base.

    ``self.path`` is stored as-is (NOT coerced to ``str``); subclasses
    (``CacheTarget``/``_LocalPathTarget``) override the rest and normalize the path.
    """

    def __init__(self, path=None):
        self.path = path

    def exists(self):
        return False
```

#### inherits

Copy parameters (and nothing else) from one or more task classes onto the decorated task, and add `clone_parent`/`clone_parents` helpers. Avoids pythonic inheritance.

Supports positional tasks (`clone_parents` returns a list) or named tasks via keyword arguments (`clone_parents` returns a dict).

Source code in `oryxflow/core.py`

```
class inherits:
    """
    Copy parameters (and nothing else) from one or more task classes onto the decorated task,
    and add ``clone_parent``/``clone_parents`` helpers. Avoids pythonic inheritance.

    Supports positional tasks (``clone_parents`` returns a list) or named tasks via keyword
    arguments (``clone_parents`` returns a dict).
    """

    def __init__(self, *tasks_to_inherit, **kw_tasks_to_inherit):
        super(inherits, self).__init__()
        if not tasks_to_inherit and not kw_tasks_to_inherit:
            raise TypeError("tasks_to_inherit or kw_tasks_to_inherit must contain at least one task")
        if tasks_to_inherit and kw_tasks_to_inherit:
            raise TypeError("Only one of tasks_to_inherit or kw_tasks_to_inherit may be present")
        self.tasks_to_inherit = tasks_to_inherit
        self.kw_tasks_to_inherit = kw_tasks_to_inherit

    def __call__(self, task_that_inherits):
        task_iterator = self.tasks_to_inherit or self.kw_tasks_to_inherit.values()
        for task_to_inherit in task_iterator:
            for param_name, param_obj in task_to_inherit.get_params():
                if not hasattr(task_that_inherits, param_name):
                    setattr(task_that_inherits, param_name, param_obj)

        if self.tasks_to_inherit:
            def clone_parent(_self, **kwargs):
                return _self.clone(cls=self.tasks_to_inherit[0], **kwargs)
            task_that_inherits.clone_parent = clone_parent

            def clone_parents(_self, **kwargs):
                return [
                    _self.clone(cls=task_to_inherit, **kwargs)
                    for task_to_inherit in self.tasks_to_inherit
                ]
            task_that_inherits.clone_parents = clone_parents
        elif self.kw_tasks_to_inherit:
            def clone_parents(_self, **kwargs):
                return {
                    task_name: _self.clone(cls=task_to_inherit, **kwargs)
                    for task_name, task_to_inherit in self.kw_tasks_to_inherit.items()
                }
            task_that_inherits.clone_parents = clone_parents

        return task_that_inherits
```

#### requires

Same as class:`inherits`, but also auto-defines the `requires` method.

Source code in `oryxflow/core.py`

```
class requires:
    """Same as :py:class:`inherits`, but also auto-defines the ``requires`` method."""

    def __init__(self, *tasks_to_require, **kw_tasks_to_require):
        super(requires, self).__init__()
        self.tasks_to_require = tasks_to_require
        self.kw_tasks_to_require = kw_tasks_to_require

    def __call__(self, task_that_requires):
        task_that_requires = inherits(*self.tasks_to_require, **self.kw_tasks_to_require)(task_that_requires)

        def requires(_self):
            return _self.clone_parent() if len(self.tasks_to_require) == 1 else _self.clone_parents()
        task_that_requires.requires = requires

        return task_that_requires
```

#### TaskFailure

One failed task: the task instance plus why it failed.

Source code in `oryxflow/core.py`

```
class TaskFailure:
    """One failed task: the task instance plus why it failed."""
    def __init__(self, task, exception=None, traceback=None, reason=None):
        self.task = task
        self.exception = exception      # the real run() exception, or None
        self.traceback = traceback      # formatted traceback string, or None
        self.reason = reason            # 'run error' | 'dependency failed' | 'external missing'
    def __str__(self):
        if self.exception is not None:
            return "{}: {}: {}".format(_task_label(self.task),
                                       type(self.exception).__name__, self.exception)
        return "{}: {}".format(_task_label(self.task), self.reason or "failed")
```

#### RunResult

What happened to the DAG in one build: identities, status, failure context.

Source code in `oryxflow/core.py`

```
class RunResult:
    """What happened to the DAG in one build: identities, status, failure context."""
    def __init__(self, success, ran, complete, failed, run_id=None, reasons=None,
                 warnings=None):
        self.success = success          # bool: no failures
        self.ran = ran                  # list[Task]  actually recomputed
        self.complete = complete        # list[Task]  cache hits (skipped)
        self.failed = failed            # list[TaskFailure]
        self.run_id = run_id            # id shared by this build's events
        self.reasons = reasons or {}    # {task_id: why it ran} for tasks in `ran`
        self.warnings = warnings or []  # unacknowledged code-change warnings, deduped messages

    # --- back-compat aliases ---
    @property
    def scheduling_succeeded(self):
        return self.success
    @property
    def first_exception(self):
        return next((f.exception for f in self.failed if f.exception is not None), None)
    def summary(self):
        """Return the execution summary text (``print(result.summary())``)."""
        return str(self)

    def __bool__(self):
        return self.success

    # --- queries (by task CLASS) ---
    def did_run(self, task_cls):
        return any(isinstance(t, task_cls) for t in self.ran)
    def ran_of(self, task_cls):
        return [t for t in self.ran if isinstance(t, task_cls)]
    def failure_of(self, task_cls):
        return next((f for f in self.failed if isinstance(f.task, task_cls)), None)

    def __str__(self):
        # luigi-compatible wording so the plugin SKILL.md guidance stays accurate.
        n = len(self.complete) + len(self.ran) + len(self.failed)
        def block(label, rendered, cap=None):
            lines = ["* {} {}".format(len(rendered), label) + (":" if rendered else "")]
            shown = rendered if cap is None else rendered[:cap]
            lines += ["    - {}".format(s) for s in shown]
            if cap is not None and len(rendered) > cap:
                lines.append("    ... and {} more".format(len(rendered) - cap))
            return lines
        parts = ["Scheduled {} tasks of which:".format(n)]
        parts += block("complete ones were encountered",
                       [_task_label(t) for t in self.complete], cap=10)
        parts += block("ran successfully", [_task_label(t) for t in self.ran])
        parts += block("failed", [str(f) for f in self.failed])   # TaskFailure.__str__
        smiley = ":)" if self.success else ":("
        why = ("there were no failed tasks or missing dependencies" if self.success
               else "there were failed tasks")
        parts.append("This progress looks {} because {}".format(smiley, why))
        return "\n".join(parts)
```

##### summary

```
summary()
```

Return the execution summary text (`print(result.summary())`).

Source code in `oryxflow/core.py`

```
def summary(self):
    """Return the execution summary text (``print(result.summary())``)."""
    return str(self)
```

#### MultiRunResult

Bases: `dict`

Return value of `WorkflowMulti.run()`: a `{flow_name: RunResult}` dict that also carries `.summary()`/`.success` so `print(result.summary())` works the same as for a single `Workflow` (whose `run()` returns a plain class:`RunResult`).

Source code in `oryxflow/core.py`

```
class MultiRunResult(dict):
    """Return value of ``WorkflowMulti.run()``: a ``{flow_name: RunResult}`` dict that also
    carries ``.summary()``/``.success`` so ``print(result.summary())`` works the same as for a
    single ``Workflow`` (whose ``run()`` returns a plain :py:class:`RunResult`)."""
    @property
    def success(self):
        return all(bool(r) for r in self.values())
    def __bool__(self):
        return self.success

    # --- aggregates across flows, so callers never hand-roll them ---
    @property
    def ran(self):
        return [t for r in self.values() for t in r.ran]
    @property
    def complete(self):
        return [t for r in self.values() for t in r.complete]
    @property
    def failed(self):
        return [f for r in self.values() for f in r.failed]
    @property
    def reasons(self):
        merged = {}
        for r in self.values():
            merged.update(r.reasons)
        return merged
    @property
    def warnings(self):
        # deduped across flows: each per-flow build re-warns for shared upstreams, and
        # "how many pending conditions" must not scale with the flow count
        merged = []
        for r in self.values():
            for w in r.warnings:
                if w not in merged:
                    merged.append(w)
        return merged

    def summary(self):
        """Per-flow execution summaries, each under a ``===== <flow> =====`` header."""
        return "\n\n".join(
            "===== {} =====\n{}".format(name, res.summary()) for name, res in self.items())
    def __str__(self):
        return self.summary()
```

##### summary

```
summary()
```

Per-flow execution summaries, each under a `===== <flow> =====` header.

Source code in `oryxflow/core.py`

```
def summary(self):
    """Per-flow execution summaries, each under a ``===== <flow> =====`` header."""
    return "\n\n".join(
        "===== {} =====\n{}".format(name, res.summary()) for name, res in self.items())
```

#### flatten

```
flatten(struct)
```

Create a flat list of all items in a structured object (dicts, lists, items)::

```
>>> sorted(flatten({'a': 'foo', 'b': 'bar'}))
['bar', 'foo']
>>> sorted(flatten(['foo', ['bar', 'troll']]))
['bar', 'foo', 'troll']
>>> flatten('foo')
['foo']
>>> flatten(42)
[42]
```

Source code in `oryxflow/core.py`

```
def flatten(struct):
    """
    Create a flat list of all items in a structured object (dicts, lists, items)::

        >>> sorted(flatten({'a': 'foo', 'b': 'bar'}))
        ['bar', 'foo']
        >>> sorted(flatten(['foo', ['bar', 'troll']]))
        ['bar', 'foo', 'troll']
        >>> flatten('foo')
        ['foo']
        >>> flatten(42)
        [42]
    """
    if struct is None:
        return []
    flat = []
    if isinstance(struct, dict):
        for _, result in struct.items():
            flat += flatten(result)
        return flat
    if isinstance(struct, str):
        return [struct]

    try:
        iterator = iter(struct)
    except TypeError:
        return [struct]

    for result in iterator:
        flat += flatten(result)
    return flat
```

#### getpaths

```
getpaths(struct)
```

Map all Tasks in a structured object to their `.output()`.

Source code in `oryxflow/core.py`

```
def getpaths(struct):
    """Map all Tasks in a structured object to their ``.output()``."""
    if isinstance(struct, Task):
        return struct.output()
    elif isinstance(struct, dict):
        return struct.__class__((k, getpaths(v)) for k, v in struct.items())
    elif isinstance(struct, (list, tuple)):
        return struct.__class__(getpaths(r) for r in struct)
    else:
        try:
            return [getpaths(r) for r in struct]
        except TypeError:
            raise Exception('Cannot map %s to Task/dict/list' % str(struct))
```

#### task_id_str

```
task_id_str(task_family, params)
```

Return a canonical, deterministic string identifying a task.

The id is `{family}_{param_summary}_{md5(sorted_json)[:10]}` so that `task_id.split('_')[0]` yields the task family (the directory convention).

:param task_family: the task family (class name) :param params: dict mapping parameter names to serialized (str) values

Source code in `oryxflow/core.py`

```
def task_id_str(task_family, params):
    """
    Return a canonical, deterministic string identifying a task.

    The id is ``{family}_{param_summary}_{md5(sorted_json)[:10]}`` so that
    ``task_id.split('_')[0]`` yields the task family (the directory convention).

    :param task_family: the task family (class name)
    :param params: dict mapping parameter names to serialized (str) values
    """
    param_str = json.dumps(params, separators=(',', ':'), sort_keys=True)
    param_hash = hashlib.md5(param_str.encode('utf-8')).hexdigest()

    param_summary = '_'.join(p[:TASK_ID_TRUNCATE_PARAMS]
                             for p in (params[p] for p in sorted(params)[:TASK_ID_INCLUDE_PARAMS]))
    param_summary = TASK_ID_INVALID_CHAR_REGEX.sub('_', param_summary)

    return '{}_{}_{}'.format(task_family, param_summary, param_hash[:TASK_ID_TRUNCATE_HASH])
```

#### dfs_paths

```
dfs_paths(start_task, goal_task_family, path=None)
```

Yield tasks on every dependency path from `start_task` to `goal_task_family`.

Source code in `oryxflow/core.py`

```
def dfs_paths(start_task, goal_task_family, path=None):
    """Yield tasks on every dependency path from ``start_task`` to ``goal_task_family``."""
    if path is None:
        path = [start_task]
    if start_task.task_family == goal_task_family or goal_task_family is None:
        for item in path:
            yield item
    for next_task in _get_task_requires(start_task) - set(path):
        for t in dfs_paths(next_task, goal_task_family, path + [next_task]):
            yield t
```

#### find_deps

```
find_deps(task, upstream_task_family)
```

Return the set of all tasks on all paths between `task` and `upstream_task_family`.

Source code in `oryxflow/core.py`

```
def find_deps(task, upstream_task_family):
    """Return the set of all tasks on all paths between ``task`` and ``upstream_task_family``."""
    return {t for t in dfs_paths(task, upstream_task_family)}
```

#### build

```
build(tasks, workers=1, detailed_summary=False, flow=None, **ignored)
```

Run `tasks` and their dependencies sequentially, in dependency order.

All state is local to this call, so a task's `run()` may itself call `oryxflow.run` / `flow.run` (the flow-within-a-flow pattern) without corrupting the outer build.

External tasks (`external=True` or `run is None`) are never executed; if still incomplete after their dependencies, they are marked failed.

:param flow: flow name when launched via `Workflow`/`WorkflowMulti`; stamped into this build's event envelopes. :returns: a class:`RunResult` with `.ran`/`.complete`/`.failed` task identities, `.success`, `.run_id`, per-task `.reasons` and code-change `.warnings`.

Source code in `oryxflow/core.py`

```
def build(tasks, workers=1, detailed_summary=False, flow=None, **ignored):
    """
    Run ``tasks`` and their dependencies sequentially, in dependency order.

    All state is local to this call, so a task's ``run()`` may itself call ``oryxflow.run`` /
    ``flow.run`` (the flow-within-a-flow pattern) without corrupting the outer build.

    External tasks (``external=True`` or ``run is None``) are never executed; if still
    incomplete after their dependencies, they are marked failed.

    :param flow: flow name when launched via ``Workflow``/``WorkflowMulti``; stamped into
        this build's event envelopes.
    :returns: a :py:class:`RunResult` with ``.ran``/``.complete``/``.failed`` task identities,
        ``.success``, ``.run_id``, per-task ``.reasons`` and code-change ``.warnings``.
    """
    import uuid
    from oryxflow import events as _events
    from oryxflow import state as _state
    from oryxflow import codehash as _codehash
    from oryxflow import codecheck as _codecheck
    from oryxflow import log as _log
    from oryxflow import settings as _settings

    if not isinstance(tasks, (list,)):
        tasks = [tasks]

    visited = {}          # task_id -> bool (success)
    ran = []              # tasks executed this build
    already_complete = []  # tasks already complete
    failed = []           # TaskFailure for tasks that failed (run error, failed dep, missing external)
    reasons = {}          # task_id -> why it ran

    run_id = uuid.uuid4().hex
    git_sha, git_dirty = _git_info()

    def _emit(event_type, payload, msg=None, level='info'):
        # single call site per engine fact: the event and the human loguru line
        # come from the same data and can't disagree
        _events.append(event_type, payload, run_id=run_id, flow=flow)
        if msg is not None:
            getattr(logger, level)("{}", msg)

    _advisor = _codecheck.Advisor(_emit)
    warned = _advisor.warned   # unacknowledged code-change warning strings (RunResult)

    def _emit_failed(task, error, tb, duration):
        _emit('task_failed',
              {'task_id': task.task_id, 'family': task.task_family,
               'params': task.to_str_params(only_significant=False),
               'error': error, 'traceback': tb[-4096:] if tb else None,
               'duration_s': duration})

    def _process(task):
        tid = task.task_id
        if tid in visited:
            return visited[tid]

        if task.complete():
            _advisor.advise(task)
            logger.debug("task skipped (already complete): {}", tid)
            already_complete.append(task)
            visited[tid] = True
            return True

        # Process dependencies first.
        dep_ok = True
        for dep in flatten(task.requires()):
            if not _process(dep):
                dep_ok = False
        if not dep_ok:
            logger.error("task failed (dependency failed): {}", tid)
            failed.append(TaskFailure(task, reason="dependency failed"))
            _emit_failed(task, 'dependency failed', None, None)
            visited[tid] = False
            return False

        # External tasks are never executed; their output must come from elsewhere.
        is_external = getattr(task, 'external', False) or getattr(task, 'run', None) is None
        if is_external:
            if task.complete():
                already_complete.append(task)
                visited[tid] = True
                return True
            logger.warning("external task missing output: {}", tid)
            failed.append(TaskFailure(task, reason="external missing output"))
            _emit_failed(task, 'external missing output', None, None)
            visited[tid] = False
            return False

        reason = _advisor.reason_for(task, ran, reasons)
        logger.info("task start: {} ({})", task.task_family, tid)
        t0 = time.perf_counter()
        try:
            result = task.run()
            if inspect.isgenerator(result):
                if not _drive_generator(result):
                    failed.append(TaskFailure(task, reason="dependency failed"))
                    _emit_failed(task, 'dependency failed', None,
                                 time.perf_counter() - t0)
                    visited[tid] = False
                    return False
        except Exception as e:
            logger.opt(exception=True).error("task failed: {}", tid)
            tb = traceback.format_exc()
            failed.append(TaskFailure(task, exception=e, traceback=tb,
                                      reason="run error"))
            _emit_failed(task, '{}: {}'.format(type(e).__name__, e), tb,
                         time.perf_counter() - t0)
            visited[tid] = False
            return False

        duration = time.perf_counter() - t0
        fp, dirpath = _codecheck.code_state(task)
        hashes = _codecheck.hashes_of(task) if (fp is not None
                                                or getattr(_settings, 'events', True)) else {}
        if fp is not None:
            try:
                # fresh output_id: this task actually rematerialized, downstream
                # dep_state folds must move
                _state.put_record(dirpath, tid,
                                  _codecheck.make_record(task, hashes, uuid.uuid4().hex[:16],
                                                         duration=duration))
            except Exception as e:
                logger.debug("state record write failed for {}: {}", tid, e)
        try:
            from oryxflow import __version__ as _version
        except Exception:
            _version = None
        _emit('task_ran',
              {'task_id': tid, 'family': task.task_family,
               'params': task.to_str_params(only_significant=False),
               'code_version': task.code_version, 'fingerprint': fp,
               'auto': task.code_version is None and _settings.code_version_auto,
               'source_hashes': hashes, 'reason': reason, 'duration_s': duration,
               'git_sha': git_sha, 'git_dirty': git_dirty,
               'oryxflow_version': _version,
               'dirpath': str(dirpath if dirpath is not None else _settings.dirpath)},
              msg="task complete: {} in {:.3f}s".format(tid, duration))
        _code_warned.pop(tid, None)   # condition resolved; a future recurrence re-warns
        ran.append(task)
        reasons[tid] = reason
        visited[tid] = True
        return True

    def _drive_generator(gen):
        # Drive a generator-style run() that yields tasks (eg TaskAggregator) or dynamic
        # requirements, processing each yielded batch before resuming.
        try:
            next_requires = next(gen)
            while True:
                logger.debug("generator yielded {} requires", len(flatten(next_requires)))
                ok = True
                for req in flatten(next_requires):
                    if not _process(req):
                        ok = False
                if not ok:
                    gen.close()
                    return False
                next_requires = gen.send(getpaths(next_requires))
        except StopIteration:
            return True

    _emit('run_started',
          {'tasks': sorted({t.task_family for t in tasks})},
          msg="run started: {} (run_id={})".format(
              ', '.join(sorted({t.task_family for t in tasks})), run_id))

    def _capture_task_log(level, message, context):
        extra = {}
        for k, v in list(context.items())[:20]:
            extra[k] = v if isinstance(v, (int, float, bool, type(None))) else str(v)[:512]
        _events.append('task_log',
                       {'task_id': context.get('task_id'),
                        'family': context.get('task_family'),
                        'level': level, 'message': str(message)[:4096], 'extra': extra},
                       run_id=run_id, flow=flow)

    previous_capture = _log.set_task_log_capture(_capture_task_log)
    _codehash.freeze()   # code can't change mid-build: skip per-complete() mtime re-stats
    try:
        for task in tasks:
            _process(task)
    finally:
        _codehash.unfreeze()
        _log.set_task_log_capture(previous_capture)

    success = len(failed) == 0
    _emit('run_finished',
          {'counts': {'ran': len(ran), 'complete': len(already_complete),
                      'failed': len(failed)}, 'success': success},
          msg="run finished: {} ran, {} complete, {} failed, success={}".format(
              len(ran), len(already_complete), len(failed), success))
    try:
        _events.flush()   # events are written async; the run's story is durable on return
    except Exception:
        pass
    result = RunResult(success, ran, already_complete, failed,
                       run_id=run_id, reasons=reasons, warnings=warned)
    if detailed_summary:                       # gated by execution_summary
        logger.info("run summary:\n{}", result.summary())
    return result
```

## Parameters — `oryxflow.parameter`

### oryxflow.parameter

Self-contained, trimmed set of parameter types used by oryxflow.

There is no command-line / config-file value resolution, parameter visibility, `date_interval`, freezing (`FrozenOrderedDict`) or `jsonschema` support. Values are resolved from the constructor argument or the `default` only.

Only the parameter types oryxflow actually uses are kept: `Parameter`, `IntParameter`, `FloatParameter`, `BoolParameter`, `DateParameter`, `DictParameter`, `ListParameter`, `ChoiceParameter` and `EnumParameter`.

#### ParameterException

Bases: `Exception`

Base parameter exception.

Source code in `oryxflow/parameter.py`

```
class ParameterException(Exception):
    """Base parameter exception."""
    pass
```

#### MissingParameterException

Bases: `ParameterException`

Raised when a required parameter has no value and no default.

Source code in `oryxflow/parameter.py`

```
class MissingParameterException(ParameterException):
    """Raised when a required parameter has no value and no default."""
    pass
```

#### UnknownParameterException

Bases: `ParameterException`

Raised when an unknown parameter is supplied to a task.

Source code in `oryxflow/parameter.py`

```
class UnknownParameterException(ParameterException):
    """Raised when an unknown parameter is supplied to a task."""
    pass
```

#### DuplicateParameterException

Bases: `ParameterException`

Raised when a parameter is supplied both positionally and as a keyword.

Source code in `oryxflow/parameter.py`

```
class DuplicateParameterException(ParameterException):
    """Raised when a parameter is supplied both positionally and as a keyword."""
    pass
```

#### Parameter

Parameter whose value is a `str`, and the base class for the other parameter types.

Parameters are set on the Task class to parameterize tasks::

```
class MyTask(oryxflow.tasks.TaskData):
    foo = oryxflow.Parameter()
```

When a value is not provided at instantiation, the `default` is used.

Source code in `oryxflow/parameter.py`

```
class Parameter:
    """
    Parameter whose value is a ``str``, and the base class for the other parameter types.

    Parameters are set on the Task class to parameterize tasks::

        class MyTask(oryxflow.tasks.TaskData):
            foo = oryxflow.Parameter()

    When a value is not provided at instantiation, the ``default`` is used.
    """

    # Non-atomically increasing counter used to preserve declaration order (see Task.get_params).
    _counter = 0

    def __init__(self, default=_no_value, significant=True, description=None, positional=True):
        """
        :param default: the default value for this parameter. If not given, a value must be
                        specified when the task is instantiated.
        :param bool significant: ``False`` if the parameter should not be part of a task's unique
                                 identifier (eg passwords, environment markers). Default ``True``.
        :param str description: human-readable description of the parameter.
        :param bool positional: if ``True`` the parameter may be set positionally. Default ``True``.
        """
        self._default = default
        self.significant = significant
        self.description = description
        self.positional = positional

        self._counter = Parameter._counter
        Parameter._counter += 1

    def has_task_value(self):
        """Whether a default value is available."""
        return self._default != _no_value

    def task_value(self):
        """The normalized default value. Raises if no default is set."""
        if self._default == _no_value:
            raise MissingParameterException("No default specified")
        return self.normalize(self._default)

    def parse(self, x):
        """Parse an individual value from a string. Identity by default."""
        return x

    def serialize(self, x):
        """Convert the value ``x`` to a string. Opposite of :py:meth:`parse`."""
        return str(x)

    def normalize(self, x):
        """Normalize a parsed/default/constructor value. Identity by default."""
        return x
```

##### has_task_value

```
has_task_value()
```

Whether a default value is available.

Source code in `oryxflow/parameter.py`

```
def has_task_value(self):
    """Whether a default value is available."""
    return self._default != _no_value
```

##### task_value

```
task_value()
```

The normalized default value. Raises if no default is set.

Source code in `oryxflow/parameter.py`

```
def task_value(self):
    """The normalized default value. Raises if no default is set."""
    if self._default == _no_value:
        raise MissingParameterException("No default specified")
    return self.normalize(self._default)
```

##### parse

```
parse(x)
```

Parse an individual value from a string. Identity by default.

Source code in `oryxflow/parameter.py`

```
def parse(self, x):
    """Parse an individual value from a string. Identity by default."""
    return x
```

##### serialize

```
serialize(x)
```

Convert the value `x` to a string. Opposite of meth:`parse`.

Source code in `oryxflow/parameter.py`

```
def serialize(self, x):
    """Convert the value ``x`` to a string. Opposite of :py:meth:`parse`."""
    return str(x)
```

##### normalize

```
normalize(x)
```

Normalize a parsed/default/constructor value. Identity by default.

Source code in `oryxflow/parameter.py`

```
def normalize(self, x):
    """Normalize a parsed/default/constructor value. Identity by default."""
    return x
```

#### IntParameter

Bases: `Parameter`

Parameter whose value is an `int`.

Source code in `oryxflow/parameter.py`

```
class IntParameter(Parameter):
    """Parameter whose value is an ``int``."""

    def parse(self, s):
        return int(s)
```

#### FloatParameter

Bases: `Parameter`

Parameter whose value is a `float`.

Source code in `oryxflow/parameter.py`

```
class FloatParameter(Parameter):
    """Parameter whose value is a ``float``."""

    def parse(self, s):
        return float(s)
```

#### BoolParameter

Bases: `Parameter`

A Parameter whose value is a `bool`. Has an implicit default of `False`.

Source code in `oryxflow/parameter.py`

```
class BoolParameter(Parameter):
    """
    A Parameter whose value is a ``bool``. Has an implicit default of ``False``.
    """

    def __init__(self, *args, **kwargs):
        super(BoolParameter, self).__init__(*args, **kwargs)
        if self._default == _no_value:
            self._default = False

    def parse(self, val):
        """Parse a ``bool`` from the string, matching 'true'/'false' case-insensitively."""
        s = str(val).lower()
        if s == "true":
            return True
        elif s == "false":
            return False
        else:
            raise ValueError("cannot interpret '{}' as boolean".format(val))

    def normalize(self, value):
        try:
            return self.parse(value)
        except ValueError:
            return None
```

##### parse

```
parse(val)
```

Parse a `bool` from the string, matching 'true'/'false' case-insensitively.

Source code in `oryxflow/parameter.py`

```
def parse(self, val):
    """Parse a ``bool`` from the string, matching 'true'/'false' case-insensitively."""
    s = str(val).lower()
    if s == "true":
        return True
    elif s == "false":
        return False
    else:
        raise ValueError("cannot interpret '{}' as boolean".format(val))
```

#### DateParameter

Bases: `Parameter`

Parameter whose value is a class:`~datetime.date`, formatted `YYYY-MM-DD`.

Source code in `oryxflow/parameter.py`

```
class DateParameter(Parameter):
    """
    Parameter whose value is a :py:class:`~datetime.date`, formatted ``YYYY-MM-DD``.
    """

    date_format = '%Y-%m-%d'

    def parse(self, s):
        return datetime.datetime.strptime(s, self.date_format).date()

    def serialize(self, dt):
        if dt is None:
            return str(dt)
        return dt.strftime(self.date_format)

    def normalize(self, value):
        if value is None:
            return None
        if isinstance(value, datetime.datetime):
            value = value.date()
        return value
```

#### DictParameter

Bases: `Parameter`

Parameter whose value is a `dict`.

The value is stored as-is (no freezing); it is serialized with sorted keys so that the task id stays deterministic.

Source code in `oryxflow/parameter.py`

```
class DictParameter(Parameter):
    """
    Parameter whose value is a ``dict``.

    The value is stored as-is (no freezing); it is serialized with sorted keys so that the
    task id stays deterministic.
    """

    def parse(self, source):
        if not isinstance(source, str):
            return source
        return json.loads(source)

    def serialize(self, x):
        return json.dumps(x, sort_keys=True)
```

#### ListParameter

Bases: `Parameter`

Parameter whose value is a `list`.

The value is stored as-is (no freezing); it is serialized as JSON so that the task id stays deterministic.

Source code in `oryxflow/parameter.py`

```
class ListParameter(Parameter):
    """
    Parameter whose value is a ``list``.

    The value is stored as-is (no freezing); it is serialized as JSON so that the task id
    stays deterministic.
    """

    def parse(self, x):
        if not isinstance(x, str):
            return x
        i = json.loads(x)
        if i is None:
            return None
        return list(i)

    def serialize(self, x):
        return json.dumps(x)
```

#### ChoiceParameter

Bases: `Parameter`

A string-valued parameter restricted to a fixed set of `choices`::

```
class MyTask(oryxflow.tasks.TaskData):
    model = oryxflow.ChoiceParameter(choices=['rf', 'lgbm'], default='rf')
```

Values stay plain strings (no `enum.Enum` ceremony, unlike :class:`EnumParameter`); a value outside `choices` raises immediately at task construction (or when the default is resolved), so typos fail fast instead of dying deep in downstream code.

Source code in `oryxflow/parameter.py`

```
class ChoiceParameter(Parameter):
    """
    A string-valued parameter restricted to a fixed set of ``choices``::

        class MyTask(oryxflow.tasks.TaskData):
            model = oryxflow.ChoiceParameter(choices=['rf', 'lgbm'], default='rf')

    Values stay plain strings (no ``enum.Enum`` ceremony, unlike
    :class:`EnumParameter`); a value outside ``choices`` raises immediately at task
    construction (or when the default is resolved), so typos fail fast instead of
    dying deep in downstream code.
    """

    def __init__(self, *args, **kwargs):
        if 'choices' not in kwargs:
            raise ParameterException('A choices list must be specified.')
        choices = kwargs.pop('choices')
        try:
            self._choices = list(choices)
        except TypeError:
            raise ParameterException('choices must be an iterable of values.')
        if not self._choices:
            raise ParameterException('choices must not be empty.')
        super(ChoiceParameter, self).__init__(*args, **kwargs)

    def normalize(self, x):
        if x not in self._choices:
            raise ValueError(
                "'{}' is not a valid choice - must be one of {}".format(x, self._choices))
        return x
```

#### EnumParameter

Bases: `Parameter`

A parameter whose value is an :class:`~enum.Enum`. Pass the enum class via `enum=`::

```
class Model(enum.Enum):
    Honda = 1
    Volvo = 2

class MyTask(oryxflow.tasks.TaskData):
    my_param = oryxflow.EnumParameter(enum=Model)
```

Source code in `oryxflow/parameter.py`

```
class EnumParameter(Parameter):
    """
    A parameter whose value is an :class:`~enum.Enum`. Pass the enum class via ``enum=``::

        class Model(enum.Enum):
            Honda = 1
            Volvo = 2

        class MyTask(oryxflow.tasks.TaskData):
            my_param = oryxflow.EnumParameter(enum=Model)
    """

    def __init__(self, *args, **kwargs):
        if 'enum' not in kwargs:
            raise ParameterException('An enum class must be specified.')
        self._enum = kwargs.pop('enum')
        super(EnumParameter, self).__init__(*args, **kwargs)

    def parse(self, s):
        try:
            return self._enum[s]
        except KeyError:
            raise ValueError('Invalid enum value - could not be parsed')

    def serialize(self, e):
        return e.name
```

## Tasks — `oryxflow.tasks`

### oryxflow.tasks

#### TaskData

Bases: `Task`

Task which has data as input and output

Attributes:

| Name           | Type   | Description                                                                                                                                                                                            |
| -------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `target_class` | `obj`  | target data format                                                                                                                                                                                     |
| `target_ext`   | `str`  | file extension                                                                                                                                                                                         |
| `persists`     | `list` | list of strings naming the outputs this task saves. Declare it on your task class, e.g. persists = ['x', 'y']. persist (singular) is a backwards-compatible alias for the same thing; prefer persists. |
| `data`         | `dict` | data container for all outputs                                                                                                                                                                         |

Source code in `oryxflow/tasks/__init__.py`

```
class TaskData(core.Task):
    """
    Task which has data as input and output

    Attributes:
        target_class (obj): target data format
        target_ext (str): file extension
        persists (list): list of strings naming the outputs this task saves.
            Declare it on your task class, e.g. ``persists = ['x', 'y']``.
            ``persist`` (singular) is a backwards-compatible alias for the same
            thing; prefer ``persists``.
        data (dict): data container for all outputs

    """
    target_class = oryxflow.targets.DataTarget
    target_ext = 'ext'
    # canonical internal attribute; users declare it as `persists` (see __init__)
    persist = ['data']
    metadata = None
    # keep outputs of previous code_versions at readable paths (.../Task/v1/...) instead
    # of overwriting in place; only takes effect when code_version is set
    keep_versions = False

    def __init__(self, *args, path=None, flows=None, **kwargs):
        kwargs_ = {k: v for k, v in kwargs.items(
        ) if k in self.get_param_names(include_significant=True)}
        super().__init__(*args, **kwargs_)

        # Check if Child Has Path Var
        self.path = getattr(self, 'path', path)

        # `persists` is the user-facing name; fold it into the internal `persist`.
        # (engine code reads `self.persist` throughout)
        self.persist = getattr(self, 'persists', self.persist)

        # Flow
        self.flows = flows

    @classmethod
    def get_param_values(cls, params, args, kwargs):
        kwargs_ = {k: v for k, v in kwargs.items(
        ) if k in cls.get_param_names(include_significant=True)}
        return super(TaskData, cls).get_param_values(params, args, kwargs_)

    def reset(self, confirm=False):
        """
        Reset a task, eg by deleting output file
        """
        return self.invalidate(confirm)

    def invalidate(self, confirm=False):
        """
        Reset a task, eg by deleting output file
        """
        if confirm:
            c = input(
                'Confirm invalidating task: {} (y/n). PS You can disable this message by passing confirm=False'.format(
                    self.__class__.__qualname__))
        else:
            c = 'y'
        if c == 'y':  # and self.complete():
            if self.persist == ['data']:  # 1 data shortcut
                self.output().invalidate()
            else:
                [t.invalidate() for t in self.output().values()]
            self._invalidate_meta()
            logger.debug("invalidated {}", self.task_id)
        return True

    def _invalidate_meta(self):
        # Metadata (saveMeta/saveMetaJson) lives outside output(), so delete it here too.
        meta_base = self._getpath('meta')
        for ext in ('.pickle', '.json'):
            path = self._make_path_cloud_compatible(meta_base.with_suffix(ext))
            try:
                path.unlink()
                logger.debug("invalidated meta {}", path)
            except FileNotFoundError:
                pass  # no metadata was saved for this format


    def complete(self, cascade=True):
        """
        Check if a task is complete: output exists AND the stored code fingerprint
        matches the current one (``_code_ok`` -- a ``code_version`` bump makes the
        task incomplete and forces a rerun; authoritative, unlike the warn-only AST
        source-hash advisory). With ``check_dependencies``, cascades upstream.
        """
        complete = super().complete()
        if complete and not getattr(self, 'external', False):
            complete = self._code_ok()
        if oryxflow.settings.check_dependencies and cascade and not getattr(self, 'external', False):
            complete = complete and all(
                [t.complete() for t in core.flatten(self.requires())])
        return complete

    def _code_ok(self):
        # record-based, two-dimensional completeness: outputs count as complete only while
        # (a) the task's OWN code identity is unchanged -- the explicit code_version when
        # pinned, the stored source hashes when auto (mode-aware, see _own_code_ok) -- and
        # (b) no dependency has rematerialized since this record (_dep_state folds dep
        # output_ids). Both dimensions live in every record, so pinning/unpinning a task
        # with genuinely unchanged code "just resumes" instead of recomputing, and never
        # ripples downstream. Inert (True) when no code identity applies here or upstream,
        # and grandfathering (no record yet) also passes -- build() stamps the baseline.
        fp = self._code_fingerprint
        if fp is None:
            return True
        from oryxflow import state, codehash
        rec = state.get_record(self._resolved_dirpath(), self.task_id)
        if rec is None:
            return True          # grandfathered; build() stamps it
        if rec.get('v') != state.RECORD_V or rec.get('py') != codehash.PY_TAG:
            # record schema or interpreter changed -> not comparable; treat as complete
            # (grandfather trust level), build()'s advisory sweep re-stamps
            return True
        if not self._own_code_ok(rec):
            return False
        return rec.get('dep_state') == self._dep_state()

    def _own_code_ok(self, rec):
        # own dimension, mode-aware: a pin (code_version) is THE version while present;
        # auto compares the source hashes as of the last materialization. Because the
        # hashes are stored on every stamp regardless of mode, removing a pin "just
        # resumes" (no recompute when the code is genuinely unchanged) yet an edit made
        # while pinned-and-unbumped is caught the moment the pin comes off -- and pinning
        # a task in the same change that edits its code forces a rerun instead of
        # blessing the stale output.
        from oryxflow import codehash
        if self.code_version is not None:
            if rec.get('code_version') == self.code_version:
                return True
            if rec.get('code_version') is None and settings.code_version_auto:
                # opting in: free iff the code really is what produced the output
                return rec.get('source_hashes') == codehash.task_hashes(self)
            return False         # pin bump (or unverifiable flip) -> recompute
        if not settings.code_version_auto:
            return True          # own logic untracked in explicit-only mode
        if rec.get('source_hashes') == codehash.task_hashes(self):
            return True
        # expensive-recompute guard: don't silently burn a long run on a code change --
        # stay complete; build()'s advisory warns with the exits (reset/accept/pin)
        thresh = getattr(settings, 'code_version_auto_expensive_s', None)
        if thresh and (rec.get('duration_s') or 0) > thresh:
            return True
        return False

    def _dep_state(self):
        # folded output identity of the direct deps: each dep's record output_id, a fresh
        # id per actual materialization that re-stamps and accept_code PRESERVE. Downstream
        # therefore reruns when an upstream actually recomputed -- and only then: mode
        # toggles and accepts upstream don't ripple, while a reset+rerun upstream
        # propagates even across separate builds.
        from oryxflow import state
        ids = []
        for d in self.deps():
            dirpath = d._resolved_dirpath() if hasattr(d, '_resolved_dirpath') \
                else settings.dirpath
            rec = state.get_record(dirpath, d.task_id)
            ids.append((rec or {}).get('output_id') or '')
        return hashlib.md5('|'.join(sorted(ids)).encode('utf-8')).hexdigest()[:16]

    def _resolved_dirpath(self):
        # the data directory this task's artifacts (and code-invalidation records) live in
        if self.path is not None:
            return pathlib.Path(self.path)
        return settings.dirpath

    # Private Get Path Function
    def _getpath(self, k, subdir=True):
        # Get Output dir
        dirpath = self._resolved_dirpath()

        # Add Group
        if hasattr(self, 'task_group'):
            dirpath = dirpath / f"/group={getattr(self, 'task_group')}"

        # Get Path
        tidroot = getattr(self, 'target_dir', self.task_id.split('_')[0])
        if getattr(self, 'keep_versions', False) and self.code_version is not None:
            tidroot = '{}/v{}'.format(
                tidroot, core.TASK_ID_INVALID_CHAR_REGEX.sub('_', str(self.code_version)))
        fname = '{}-{}'.format(self.task_id, k) if (settings.save_with_param and getattr(
            self, 'save_attrib', True)) else '{}'.format(k)
        fname += '.{}'.format(self.target_ext)
        if subdir:
            path = dirpath / tidroot / fname
        else:
            path = dirpath / fname

        # use cloud storage
        if settings.cloud_fs_enabled:
            from pathlib import PurePosixPath  # needed on windows
            path = f'{settings.cloud_fs_prefix}/{PurePosixPath(path)}'

        return path

    def output(self):
        """
        Output target(s) this task produces
        """
        save_ = getattr(self, 'persist', [])
        output = dict([(k, self.target_class(self._getpath(k)))
                       for k in save_])
        if self.persist == ['data']:  # 1 data shortcut
            output = output['data']
        return output

    def inputLoad(self, keys=None, task=None, cached=False, as_dict=False):
        """
        Load all or several outputs from task

        Args:
            keys (list): list of data to load
            task (str): if requires multiple tasks load that task 'input1' for eg `def requires: {'input1':Task1(), 'input2':Task2()}`
            cached (bool): cache data in memory
            as_dict (bool): if the inputs were saved as a dictionary. use this to return them as dictionary. 
        Returns: list or dict of all task output
        """

        if task is not None:
            input = self.input()[task]
        else:
            input = self.input()

        requires = self.requires()
        type_of_requires = type(requires)

        if isinstance(input, dict):
            keys = input.keys() if keys is None else keys
            data = {}
            for k, v in input.items():
                if k in keys:
                    if type(v) == dict:
                        if as_dict:
                            data[k] = {k: v.load(cached) for k, v in v.items()}
                        else:
                            data[k] = [v.load(cached) for k, v in v.items()]
                    else:
                        data[k] = v.load(cached)
            # Return DF if Single Key
            if isinstance(keys, str) and not as_dict:
                return data[keys]
            # Convert to list if dependecy is Single
            if (type_of_requires != dict or task is not None) and not as_dict:
                data = list(data.values())
        elif isinstance(input, list):
            data = []
            for _target in input:
                if isinstance(_target, dict):
                    if as_dict:
                        data.append({k: v.load(cached)
                                     for k, v in _target.items()})
                    else:
                        data.append([v.load(cached)
                                     for _, v in _target.items()])
                else:
                    data.append(_target.load(cached))
        else:
            data = input.load()

        logger.debug("loaded input for {} keys={}", self.task_id,
                     list(keys) if keys is not None else None)
        return data

    def inputLoadConcat(self, keys=None, tag=True, tagkeys=None, as_dict=False,
                        concat_fn=None, cached=False):
        """Load every dependency and concatenate into one DataFrame. Works for the dict form of
        requires() ({key: Task(...)}) and the list/positional form. By default each dependency's
        significant params are added as columns. concat_fn(identifier, params, df)->df overrides."""
        requires = self.requires()
        if isinstance(requires, dict):
            items = list(requires.items())        # (key, task)
        elif isinstance(requires, (list, tuple)):
            items = list(enumerate(requires))     # (index, task)
        else:
            items = [(None, requires)]            # single dep
        def _gen():
            for ident, dep in items:
                data = self.inputLoad(keys=keys, task=ident, as_dict=as_dict, cached=cached)
                params = {n: getattr(dep, n) for n in dep.get_param_names()} if tag else {}
                yield ident, params, data
        import oryxflow.utils
        return oryxflow.utils.concat_iter(_gen(), concat_fn=concat_fn, keys=tagkeys)

    def outputLoad(self, keys=None, as_dict=False, cached=False):
        """
        Load all or several outputs from task

        Args:
            keys (list): list of data to load
            as_dict (bool): cache data in memory
            cached (bool): cache data in memory

        Returns: list or dict of all task output
        """
        if not self.complete(cascade=False):
            raise RuntimeError(
                f'Cannot load {self.__class__}, task not complete, run flow first')

        # Check Keys is not empty
        keys = self.persist if keys is None else keys
        # Not List
        if type(keys) is not list:
            if not keys in self.persist:
                raise IndexError('Key name does not match')
        else:
            for key in keys:
                if not key in self.persist:
                    raise IndexError('Key name does not match')

        logger.debug("loaded output for {} keys={}", self.task_id,
                     keys if isinstance(keys, list) else [keys])

        if self.persist == ['data']:  # 1 data shortcut
            persist_data = self.output().load()
            return persist_data

        # Get Data
        data = {k: v.load(cached)
                for k, v in self.output().items() if k in keys}

        # Return As List
        if not as_dict:
            data = list(data.values())
        # If Keys is not a list
        if type(keys) is not list:
            data = data[0]

        # Return
        return data

    def save(self, data, from_list=False, **kwargs):
        """
        Persist data to target

        Args:
            data (dict): data to save. keys are the self.persist keys and values is data

        """

        if self.persist == ['data']:  # 1 data shortcut
            self.output().save(data, **kwargs)
        else:
            targets = self.output()
            if from_list:
                data = dict(zip(self.persist, data))
            if not set(data.keys()) == set(targets.keys()):
                raise ValueError(
                    'Save dictionary needs to consistent with Task.persist')
            for k, v in data.items():
                targets[k].save(v, **kwargs)
        logger.debug("saved {} keys={}", self.task_id, list(self.persist))

    def _get_meta_path_with_format(self, task, format='pickle'):
        """Get metadata path for a given task and format"""
        if format == 'pickle':
            return self._get_meta_path(task)
        else:  # json
            return task._getpath('meta').with_suffix('.json')

    def _make_path_cloud_compatible(self, path):
        """Convert path to cloud-compatible path if cloud storage is enabled"""
        if settings.cloud_fs_enabled:
            import upath
            return upath.UPath(path)
        return pathlib.Path(path)

    def _save_meta_internal(self, data, format='pickle'):
        """Internal method to save metadata in specified format"""
        self.metadata = data
        if format == 'pickle':
            path = self._get_meta_path(self)
            path = self._make_path_cloud_compatible(path)
            with path.open("wb") as fh:
                pickle.dump(data, fh)
        else:  # json
            path = self._getpath('meta').with_suffix('.json')
            path = self._make_path_cloud_compatible(path)
            path.parent.mkdir(exist_ok=True, parents=True)
            with path.open("w") as fh:
                json.dump(data, fh)

    def _load_meta_from_task(self, task, format='pickle'):
        """Load metadata from a single task"""
        if format == 'pickle':
            path = self._get_meta_path(task)
            path = self._make_path_cloud_compatible(path)
            with path.open("rb") as fh:
                return pickle.load(fh)
        else:  # json
            path = task._getpath('meta').with_suffix('.json')
            path = self._make_path_cloud_compatible(path)
            with path.open("r") as fh:
                return json.load(fh)

    def _input_load_meta_internal(self, key=None, format='pickle'):
        """Internal method to load metadata from input tasks"""
        inputs = self.requires()

        if key is not None:
            return self._load_meta_from_task(inputs[key], format)
        elif isinstance(inputs, dict):
            return {k: self._load_meta_from_task(v, format) for k, v in inputs.items()}
        elif isinstance(inputs, list):
            return [self._load_meta_from_task(task, format) for task in inputs]
        else:
            return self._load_meta_from_task(inputs, format)

    def metaSave(self, data):
        self._save_meta_internal(data, format='pickle')

    def saveMeta(self, data):
        self.metaSave(data)

    def saveMetaJson(self, data):
        self._save_meta_internal(data, format='json')

    def metaLoad(self, key=None):
        return self._input_load_meta_internal(key, format='pickle')

    def inputLoadMetaJson(self, key=None):
        return self._input_load_meta_internal(key, format='json')

    def outputLoadMeta(self):
        if not self.complete(cascade=False):
            raise RuntimeError(
                'Cannot load, task not complete, run flow first')
        try:
            return self._load_meta_from_task(self, format='pickle')
        except FileNotFoundError:
            raise RuntimeError(
                f"No metadata to load for task {self.task_family}")

    def outputLoadMetaJson(self):
        if not self.complete(cascade=False):
            raise RuntimeError(
                'Cannot load, task not complete, run flow first')
        try:
            return self._load_meta_from_task(self, format='json')
        except FileNotFoundError:
            raise RuntimeError(
                f"No metadata to load for task {self.task_family}")

    def outputLoadAllMeta(self):
        if not self.complete(cascade=False):
            raise RuntimeError(
                'Cannot load, task not complete, run flow first')
        tasks = oryxflow.taskflow_upstream(self, only_complete=True)
        meta = []
        for task in tasks:
            try:
                meta.append(task.outputLoadMeta())
            except:
                tasks.remove(task)
        tasks = [task.task_family for task in tasks]
        return dict(zip(tasks, meta))

    def _get_meta_path(self, task):
        # Get Meta Path
        meta_path = task._getpath('meta').with_suffix('.pickle')
        meta_path.parent.mkdir(exist_ok=True, parents=True)
        return meta_path
```

##### reset

```
reset(confirm=False)
```

Reset a task, eg by deleting output file

Source code in `oryxflow/tasks/__init__.py`

```
def reset(self, confirm=False):
    """
    Reset a task, eg by deleting output file
    """
    return self.invalidate(confirm)
```

##### invalidate

```
invalidate(confirm=False)
```

Reset a task, eg by deleting output file

Source code in `oryxflow/tasks/__init__.py`

```
def invalidate(self, confirm=False):
    """
    Reset a task, eg by deleting output file
    """
    if confirm:
        c = input(
            'Confirm invalidating task: {} (y/n). PS You can disable this message by passing confirm=False'.format(
                self.__class__.__qualname__))
    else:
        c = 'y'
    if c == 'y':  # and self.complete():
        if self.persist == ['data']:  # 1 data shortcut
            self.output().invalidate()
        else:
            [t.invalidate() for t in self.output().values()]
        self._invalidate_meta()
        logger.debug("invalidated {}", self.task_id)
    return True
```

##### complete

```
complete(cascade=True)
```

Check if a task is complete: output exists AND the stored code fingerprint matches the current one (`_code_ok` -- a `code_version` bump makes the task incomplete and forces a rerun; authoritative, unlike the warn-only AST source-hash advisory). With `check_dependencies`, cascades upstream.

Source code in `oryxflow/tasks/__init__.py`

```
def complete(self, cascade=True):
    """
    Check if a task is complete: output exists AND the stored code fingerprint
    matches the current one (``_code_ok`` -- a ``code_version`` bump makes the
    task incomplete and forces a rerun; authoritative, unlike the warn-only AST
    source-hash advisory). With ``check_dependencies``, cascades upstream.
    """
    complete = super().complete()
    if complete and not getattr(self, 'external', False):
        complete = self._code_ok()
    if oryxflow.settings.check_dependencies and cascade and not getattr(self, 'external', False):
        complete = complete and all(
            [t.complete() for t in core.flatten(self.requires())])
    return complete
```

##### output

```
output()
```

Output target(s) this task produces

Source code in `oryxflow/tasks/__init__.py`

```
def output(self):
    """
    Output target(s) this task produces
    """
    save_ = getattr(self, 'persist', [])
    output = dict([(k, self.target_class(self._getpath(k)))
                   for k in save_])
    if self.persist == ['data']:  # 1 data shortcut
        output = output['data']
    return output
```

##### inputLoad

```
inputLoad(keys=None, task=None, cached=False, as_dict=False)
```

Load all or several outputs from task

Parameters:

| Name      | Type   | Description                                                                                                  | Default |
| --------- | ------ | ------------------------------------------------------------------------------------------------------------ | ------- |
| `keys`    | `list` | list of data to load                                                                                         | `None`  |
| `task`    | `str`  | if requires multiple tasks load that task 'input1' for eg def requires: {'input1':Task1(), 'input2':Task2()} | `None`  |
| `cached`  | `bool` | cache data in memory                                                                                         | `False` |
| `as_dict` | `bool` | if the inputs were saved as a dictionary. use this to return them as dictionary.                             | `False` |

Returns: list or dict of all task output

Source code in `oryxflow/tasks/__init__.py`

```
def inputLoad(self, keys=None, task=None, cached=False, as_dict=False):
    """
    Load all or several outputs from task

    Args:
        keys (list): list of data to load
        task (str): if requires multiple tasks load that task 'input1' for eg `def requires: {'input1':Task1(), 'input2':Task2()}`
        cached (bool): cache data in memory
        as_dict (bool): if the inputs were saved as a dictionary. use this to return them as dictionary. 
    Returns: list or dict of all task output
    """

    if task is not None:
        input = self.input()[task]
    else:
        input = self.input()

    requires = self.requires()
    type_of_requires = type(requires)

    if isinstance(input, dict):
        keys = input.keys() if keys is None else keys
        data = {}
        for k, v in input.items():
            if k in keys:
                if type(v) == dict:
                    if as_dict:
                        data[k] = {k: v.load(cached) for k, v in v.items()}
                    else:
                        data[k] = [v.load(cached) for k, v in v.items()]
                else:
                    data[k] = v.load(cached)
        # Return DF if Single Key
        if isinstance(keys, str) and not as_dict:
            return data[keys]
        # Convert to list if dependecy is Single
        if (type_of_requires != dict or task is not None) and not as_dict:
            data = list(data.values())
    elif isinstance(input, list):
        data = []
        for _target in input:
            if isinstance(_target, dict):
                if as_dict:
                    data.append({k: v.load(cached)
                                 for k, v in _target.items()})
                else:
                    data.append([v.load(cached)
                                 for _, v in _target.items()])
            else:
                data.append(_target.load(cached))
    else:
        data = input.load()

    logger.debug("loaded input for {} keys={}", self.task_id,
                 list(keys) if keys is not None else None)
    return data
```

##### inputLoadConcat

```
inputLoadConcat(keys=None, tag=True, tagkeys=None, as_dict=False, concat_fn=None, cached=False)
```

Load every dependency and concatenate into one DataFrame. Works for the dict form of requires() ({key: Task(...)}) and the list/positional form. By default each dependency's significant params are added as columns. concat_fn(identifier, params, df)->df overrides.

Source code in `oryxflow/tasks/__init__.py`

```
def inputLoadConcat(self, keys=None, tag=True, tagkeys=None, as_dict=False,
                    concat_fn=None, cached=False):
    """Load every dependency and concatenate into one DataFrame. Works for the dict form of
    requires() ({key: Task(...)}) and the list/positional form. By default each dependency's
    significant params are added as columns. concat_fn(identifier, params, df)->df overrides."""
    requires = self.requires()
    if isinstance(requires, dict):
        items = list(requires.items())        # (key, task)
    elif isinstance(requires, (list, tuple)):
        items = list(enumerate(requires))     # (index, task)
    else:
        items = [(None, requires)]            # single dep
    def _gen():
        for ident, dep in items:
            data = self.inputLoad(keys=keys, task=ident, as_dict=as_dict, cached=cached)
            params = {n: getattr(dep, n) for n in dep.get_param_names()} if tag else {}
            yield ident, params, data
    import oryxflow.utils
    return oryxflow.utils.concat_iter(_gen(), concat_fn=concat_fn, keys=tagkeys)
```

##### outputLoad

```
outputLoad(keys=None, as_dict=False, cached=False)
```

Load all or several outputs from task

Parameters:

| Name      | Type   | Description          | Default |
| --------- | ------ | -------------------- | ------- |
| `keys`    | `list` | list of data to load | `None`  |
| `as_dict` | `bool` | cache data in memory | `False` |
| `cached`  | `bool` | cache data in memory | `False` |

Returns: list or dict of all task output

Source code in `oryxflow/tasks/__init__.py`

```
def outputLoad(self, keys=None, as_dict=False, cached=False):
    """
    Load all or several outputs from task

    Args:
        keys (list): list of data to load
        as_dict (bool): cache data in memory
        cached (bool): cache data in memory

    Returns: list or dict of all task output
    """
    if not self.complete(cascade=False):
        raise RuntimeError(
            f'Cannot load {self.__class__}, task not complete, run flow first')

    # Check Keys is not empty
    keys = self.persist if keys is None else keys
    # Not List
    if type(keys) is not list:
        if not keys in self.persist:
            raise IndexError('Key name does not match')
    else:
        for key in keys:
            if not key in self.persist:
                raise IndexError('Key name does not match')

    logger.debug("loaded output for {} keys={}", self.task_id,
                 keys if isinstance(keys, list) else [keys])

    if self.persist == ['data']:  # 1 data shortcut
        persist_data = self.output().load()
        return persist_data

    # Get Data
    data = {k: v.load(cached)
            for k, v in self.output().items() if k in keys}

    # Return As List
    if not as_dict:
        data = list(data.values())
    # If Keys is not a list
    if type(keys) is not list:
        data = data[0]

    # Return
    return data
```

##### save

```
save(data, from_list=False, **kwargs)
```

Persist data to target

Parameters:

| Name   | Type   | Description                                                     | Default    |
| ------ | ------ | --------------------------------------------------------------- | ---------- |
| `data` | `dict` | data to save. keys are the self.persist keys and values is data | *required* |

Source code in `oryxflow/tasks/__init__.py`

```
def save(self, data, from_list=False, **kwargs):
    """
    Persist data to target

    Args:
        data (dict): data to save. keys are the self.persist keys and values is data

    """

    if self.persist == ['data']:  # 1 data shortcut
        self.output().save(data, **kwargs)
    else:
        targets = self.output()
        if from_list:
            data = dict(zip(self.persist, data))
        if not set(data.keys()) == set(targets.keys()):
            raise ValueError(
                'Save dictionary needs to consistent with Task.persist')
        for k, v in data.items():
            targets[k].save(v, **kwargs)
    logger.debug("saved {} keys={}", self.task_id, list(self.persist))
```

#### TaskCache

Bases: `TaskData`

Task which saves to cache

Source code in `oryxflow/tasks/__init__.py`

```
class TaskCache(TaskData):
    """
    Task which saves to cache
    """
    target_class = oryxflow.targets.CacheTarget
    target_ext = 'cache'
```

#### TaskCachePandas

Bases: `TaskData`

Task which saves to cache pandas dataframes

Source code in `oryxflow/tasks/__init__.py`

```
class TaskCachePandas(TaskData):
    """
    Task which saves to cache pandas dataframes
    """
    target_class = oryxflow.targets.PdCacheTarget
    target_ext = 'cache'
```

#### TaskJson

Bases: `TaskData`

Task which saves to json

Source code in `oryxflow/tasks/__init__.py`

```
class TaskJson(TaskData):
    """
    Task which saves to json
    """
    target_class = oryxflow.targets.JsonTarget
    target_ext = 'json'
```

#### TaskPickle

Bases: `TaskData`

Task which saves to pickle

Source code in `oryxflow/tasks/__init__.py`

```
class TaskPickle(TaskData):
    """
    Task which saves to pickle
    """
    target_class = oryxflow.targets.PickleTarget
    target_ext = 'pkl'
```

#### TaskCSVPandas

Bases: `TaskData`

Task which saves to CSV

Source code in `oryxflow/tasks/__init__.py`

```
class TaskCSVPandas(TaskData):
    """
    Task which saves to CSV
    """
    target_class = oryxflow.targets.CSVPandasTarget
    target_ext = 'csv'
```

#### TaskCSVGZPandas

Bases: `TaskData`

Task which saves to CSV

Source code in `oryxflow/tasks/__init__.py`

```
class TaskCSVGZPandas(TaskData):
    """
    Task which saves to CSV
    """
    target_class = oryxflow.targets.CSVGZPandasTarget
    target_ext = 'csv.gz'
```

#### TaskExcelPandasSingle

Bases: `TaskData`

Task which saves each persist key as a separate Excel file

Source code in `oryxflow/tasks/__init__.py`

```
class TaskExcelPandasSingle(TaskData):
    """
    Task which saves each persist key as a separate Excel file
    """
    target_class = oryxflow.targets.ExcelPandasTarget
    target_ext = 'xlsx'
```

#### TaskExcelPandas

Bases: `TaskData`

Task which saves multiple dataframes as sheets in a single Excel file

Source code in `oryxflow/tasks/__init__.py`

```
class TaskExcelPandas(TaskData):
    """
    Task which saves multiple dataframes as sheets in a single Excel file
    """
    target_class = oryxflow.targets.ExcelPandasSheetsTarget
    target_ext = 'xlsx'

    def output(self):
        return self.target_class(self._getpath('data'))

    def save(self, data, from_list=False, **kwargs):
        if self.persist == ['data']:
            data = {'data': data}
        else:
            if from_list:
                data = dict(zip(self.persist, data))
            if not set(data.keys()) == set(self.persist):
                raise ValueError(
                    'Save dictionary needs to be consistent with Task.persist')
        self.output().save(data, **kwargs)

    def outputLoad(self, keys=None, as_dict=False, cached=False):
        if not self.complete(cascade=False):
            raise RuntimeError(
                f'Cannot load {self.__class__}, task not complete, run flow first')

        if self.persist == ['data']:
            return self.output().load(keys='data', cached=cached)

        if keys is not None:
            # Validate keys
            check_keys = [keys] if isinstance(keys, str) else keys
            for key in check_keys:
                if key not in self.persist:
                    raise IndexError('Key name does not match')

        data = self.output().load(keys=keys, cached=cached)

        # keys=str: target returns single df directly
        if isinstance(keys, str):
            return data

        # keys=None or keys=list: target returns dict
        if as_dict:
            return data
        return list(data.values())

    def invalidate(self, confirm=False):
        if confirm:
            c = input(
                'Confirm invalidating task: {} (y/n). PS You can disable this message by passing confirm=False'.format(
                    self.__class__.__qualname__))
        else:
            c = 'y'
        if c == 'y':
            self.output().invalidate()
        return True
```

#### TaskPqPandas

Bases: `TaskData`

Task which saves to parquet

Source code in `oryxflow/tasks/__init__.py`

```
class TaskPqPandas(TaskData):
    """
    Task which saves to parquet
    """
    target_class = oryxflow.targets.PqPandasTarget
    target_ext = 'parquet'
```

#### TaskMarkdown

Bases: `TaskData`

Task which saves to markdown and HTML

Source code in `oryxflow/tasks/__init__.py`

```
class TaskMarkdown(TaskData):
    """
    Task which saves to markdown and HTML
    """
    target_class = oryxflow.targets.MarkdownTarget
    target_ext = 'md'
```

#### TaskAggregator

Bases: `Task`

Task which groups other tasks, without saving an output of its own

Declare the group in `requires()` (or with `@oryxflow.requires`) and leave `run()` empty. The group is complete when every task it requires is complete.

example::

```
@oryxflow.requires({'ols': TaskTrainOLS, 'gbm': TaskTrainGBM})
class TaskTrainAll(oryxflow.tasks.TaskAggregator):
    pass

oryxflow.Workflow(TaskTrainAll).run()
```

For a one-off group that needs no task of its own, pass a list instead: `oryxflow.run([TaskTrainOLS(), TaskTrainGBM()])`.

Source code in `oryxflow/tasks/__init__.py`

```
class TaskAggregator(core.Task):
    """
    Task which groups other tasks, without saving an output of its own

    Declare the group in ``requires()`` (or with ``@oryxflow.requires``) and leave
    ``run()`` empty. The group is complete when every task it requires is complete.

    example::

        @oryxflow.requires({'ols': TaskTrainOLS, 'gbm': TaskTrainGBM})
        class TaskTrainAll(oryxflow.tasks.TaskAggregator):
            pass

        oryxflow.Workflow(TaskTrainAll).run()

    For a one-off group that needs no task of its own, pass a list instead:
    ``oryxflow.run([TaskTrainOLS(), TaskTrainGBM()])``.
    """

    def __init__(self, *args, path=None, flows=None, **kwargs):
        # `path`/`flows` are set by Workflow, are not Parameters, and must not reach
        # the parameter machinery -- same absorption TaskData does.
        kwargs_ = {k: v for k, v in kwargs.items(
        ) if k in self.get_param_names(include_significant=True)}
        super().__init__(*args, **kwargs_)
        if inspect.isgeneratorfunction(type(self).run):
            raise RuntimeError(
                '{}: TaskAggregator no longer yields tasks from run(). Declare the group in '
                'requires() (or @oryxflow.requires) and leave run() empty.'.format(self.task_family))
        self.path = getattr(self, 'path', path)
        self.flows = flows

    @classmethod
    def get_param_values(cls, params, args, kwargs):
        kwargs_ = {k: v for k, v in kwargs.items(
        ) if k in cls.get_param_names(include_significant=True)}
        return super(TaskAggregator, cls).get_param_values(params, args, kwargs_)

    def run(self):
        pass

    def reset(self, confirm=False):
        return self.invalidate(confirm=confirm)

    def invalidate(self, confirm=False):
        for t in self.deps():
            t.invalidate(confirm)
        return True

    def complete(self, cascade=True):
        return all(t.complete(cascade) for t in self.deps())

    def output(self):
        return [t.output() for t in self.deps()]

    def outputLoad(self, keys=None, as_dict=False, cached=False):
        return [t.outputLoad(keys, as_dict, cached) for t in self.deps()]
```

## Targets — `oryxflow.targets`

### oryxflow.targets

#### CacheTarget

Bases: `LocalTarget`

Saves to in-memory cache, loads to python object

Source code in `oryxflow/targets/__init__.py`

```
class CacheTarget(core.LocalTarget):
    """
    Saves to in-memory cache, loads to python object

    """
    def __init__(self, path=None):
        super().__init__(path)
        # store as pathlib.Path so cache keys / outputPath match file-based targets
        self.path = pathlib.Path(path)

    def exists(self):
        return self.path in cache

    def invalidate(self):
        if self.path in cache:
            cache.pop(self.path)

    def load(self, cached=True):
        """
        Load from in-memory cache

        Returns: python object

        """
        if self.exists():
            return cache.get(self.path)
        else:
            raise RuntimeError('Target does not exist, make sure task is complete')


    def save(self, df):
        """
        Save dataframe to in-memory cache

        Args:
            df (obj): pandas dataframe

        Returns: filename

        """
        cache[self.path] = df
        return self.path
```

##### load

```
load(cached=True)
```

Load from in-memory cache

Returns: python object

Source code in `oryxflow/targets/__init__.py`

```
def load(self, cached=True):
    """
    Load from in-memory cache

    Returns: python object

    """
    if self.exists():
        return cache.get(self.path)
    else:
        raise RuntimeError('Target does not exist, make sure task is complete')
```

##### save

```
save(df)
```

Save dataframe to in-memory cache

Parameters:

| Name | Type  | Description      | Default    |
| ---- | ----- | ---------------- | ---------- |
| `df` | `obj` | pandas dataframe | *required* |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, df):
    """
    Save dataframe to in-memory cache

    Args:
        df (obj): pandas dataframe

    Returns: filename

    """
    cache[self.path] = df
    return self.path
```

#### DataTarget

Bases: `_LocalPathTarget`

Local target which saves in-memory data (eg dataframes) to persistent storage (eg files) and loads from storage to memory

This is an abstract class that you should extend.

Source code in `oryxflow/targets/__init__.py`

```
class DataTarget(_LocalPathTarget):
    """
    Local target which saves in-memory data (eg dataframes) to persistent storage (eg files) and loads from storage to memory

    This is an abstract class that you should extend.

    """
    def load(self, fun, cached=False, **kwargs):
        """
        Runs a function to load data from storage into memory

        Args:
            fun (function): loading function
            cached (bool): keep data cached in memory
            **kwargs: arguments to pass to `fun`

        Returns: data object

        """
        if self.exists():
            if not cached or not settings.cached or self.path not in cache:
                opts = {**{},**kwargs}
                df = fun(self.path, **opts)
                if cached or settings.cached:
                    cache[self.path] = df
                return df
            else:
                return cache.get(self.path)
        else:
            raise RuntimeError('Target does not exist, make sure task is complete')

    def save(self, df, fun, **kwargs):
        """
        Runs a function to save data from memory into storage

        Args:
            df (obj): data to save
            fun (function): saving function
            **kwargs: arguments to pass to `fun`

        Returns: filename

        """
        fun = getattr(df, fun)
        (self.path).parent.mkdir(parents=True, exist_ok=True)
        fun(self.path, **kwargs)
        return self.path
```

##### load

```
load(fun, cached=False, **kwargs)
```

Runs a function to load data from storage into memory

Parameters:

| Name       | Type       | Description                | Default    |
| ---------- | ---------- | -------------------------- | ---------- |
| `fun`      | `function` | loading function           | *required* |
| `cached`   | `bool`     | keep data cached in memory | `False`    |
| `**kwargs` |            | arguments to pass to fun   | `{}`       |

Returns: data object

Source code in `oryxflow/targets/__init__.py`

```
def load(self, fun, cached=False, **kwargs):
    """
    Runs a function to load data from storage into memory

    Args:
        fun (function): loading function
        cached (bool): keep data cached in memory
        **kwargs: arguments to pass to `fun`

    Returns: data object

    """
    if self.exists():
        if not cached or not settings.cached or self.path not in cache:
            opts = {**{},**kwargs}
            df = fun(self.path, **opts)
            if cached or settings.cached:
                cache[self.path] = df
            return df
        else:
            return cache.get(self.path)
    else:
        raise RuntimeError('Target does not exist, make sure task is complete')
```

##### save

```
save(df, fun, **kwargs)
```

Runs a function to save data from memory into storage

Parameters:

| Name       | Type       | Description              | Default    |
| ---------- | ---------- | ------------------------ | ---------- |
| `df`       | `obj`      | data to save             | *required* |
| `fun`      | `function` | saving function          | *required* |
| `**kwargs` |            | arguments to pass to fun | `{}`       |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, df, fun, **kwargs):
    """
    Runs a function to save data from memory into storage

    Args:
        df (obj): data to save
        fun (function): saving function
        **kwargs: arguments to pass to `fun`

    Returns: filename

    """
    fun = getattr(df, fun)
    (self.path).parent.mkdir(parents=True, exist_ok=True)
    fun(self.path, **kwargs)
    return self.path
```

#### CSVPandasTarget

Bases: `DataTarget`

Saves to CSV, loads to pandas dataframe

Source code in `oryxflow/targets/__init__.py`

```
class CSVPandasTarget(DataTarget):
    """
    Saves to CSV, loads to pandas dataframe

    """
    def load(self, cached=False, **kwargs):
        """
        Load from csv to pandas dataframe

        Args:
            cached (bool): keep data cached in memory
            **kwargs: arguments to pass to pd.read_csv

        Returns: pandas dataframe

        """
        return super().load(pd.read_csv, cached, **kwargs)


    def save(self, df, **kwargs):
        """
        Save dataframe to csv

        Args:
            df (obj): pandas dataframe
            **kwargs (dict): additional arguments to pass to df.to_csv

        Returns: filename

        """
        opts = {**{'index':False},**kwargs}
        return super().save(df, 'to_csv', **opts)
```

##### load

```
load(cached=False, **kwargs)
```

Load from csv to pandas dataframe

Parameters:

| Name       | Type   | Description                      | Default |
| ---------- | ------ | -------------------------------- | ------- |
| `cached`   | `bool` | keep data cached in memory       | `False` |
| `**kwargs` |        | arguments to pass to pd.read_csv | `{}`    |

Returns: pandas dataframe

Source code in `oryxflow/targets/__init__.py`

```
def load(self, cached=False, **kwargs):
    """
    Load from csv to pandas dataframe

    Args:
        cached (bool): keep data cached in memory
        **kwargs: arguments to pass to pd.read_csv

    Returns: pandas dataframe

    """
    return super().load(pd.read_csv, cached, **kwargs)
```

##### save

```
save(df, **kwargs)
```

Save dataframe to csv

Parameters:

| Name       | Type   | Description                               | Default    |
| ---------- | ------ | ----------------------------------------- | ---------- |
| `df`       | `obj`  | pandas dataframe                          | *required* |
| `**kwargs` | `dict` | additional arguments to pass to df.to_csv | `{}`       |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, df, **kwargs):
    """
    Save dataframe to csv

    Args:
        df (obj): pandas dataframe
        **kwargs (dict): additional arguments to pass to df.to_csv

    Returns: filename

    """
    opts = {**{'index':False},**kwargs}
    return super().save(df, 'to_csv', **opts)
```

#### CSVGZPandasTarget

Bases: `CSVPandasTarget`

Saves to CSV gzip, loads to pandas dataframe

Source code in `oryxflow/targets/__init__.py`

```
class CSVGZPandasTarget(CSVPandasTarget):
    """
    Saves to CSV gzip, loads to pandas dataframe

    """


    def save(self, df, **kwargs):
        """
        Save dataframe to csv gzip

        Args:
            df (obj): pandas dataframe
            **kwargs (dict): additional arguments to pass to df.to_csv

        Returns: filename

        """
        opts = {**{'index':False, 'compression':'gzip'},**kwargs}
        return super().save(df, 'to_csv', **opts)
```

##### save

```
save(df, **kwargs)
```

Save dataframe to csv gzip

Parameters:

| Name       | Type   | Description                               | Default    |
| ---------- | ------ | ----------------------------------------- | ---------- |
| `df`       | `obj`  | pandas dataframe                          | *required* |
| `**kwargs` | `dict` | additional arguments to pass to df.to_csv | `{}`       |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, df, **kwargs):
    """
    Save dataframe to csv gzip

    Args:
        df (obj): pandas dataframe
        **kwargs (dict): additional arguments to pass to df.to_csv

    Returns: filename

    """
    opts = {**{'index':False, 'compression':'gzip'},**kwargs}
    return super().save(df, 'to_csv', **opts)
```

#### ExcelPandasTarget

Bases: `DataTarget`

Saves to Excel, loads to pandas dataframe

Source code in `oryxflow/targets/__init__.py`

```
class ExcelPandasTarget(DataTarget):
    """
    Saves to Excel, loads to pandas dataframe

    """
    def load(self, cached=False, **kwargs):
        """
        Load from Excel to pandas dataframe

        Args:
            cached (bool): keep data cached in memory
            **kwargs: arguments to pass to pd.read_csv

        Returns: pandas dataframe

        """
        return super().load(pd.read_excel, cached, **kwargs)


    def save(self, df, **kwargs):
        """
        Save dataframe to Excel

        Args:
            df (obj): pandas dataframe
            **kwargs (dict): additional arguments to pass to df.to_csv

        Returns: filename

        """
        opts = {**{'index':False},**kwargs}
        return super().save(df, 'to_excel', **opts)
```

##### load

```
load(cached=False, **kwargs)
```

Load from Excel to pandas dataframe

Parameters:

| Name       | Type   | Description                      | Default |
| ---------- | ------ | -------------------------------- | ------- |
| `cached`   | `bool` | keep data cached in memory       | `False` |
| `**kwargs` |        | arguments to pass to pd.read_csv | `{}`    |

Returns: pandas dataframe

Source code in `oryxflow/targets/__init__.py`

```
def load(self, cached=False, **kwargs):
    """
    Load from Excel to pandas dataframe

    Args:
        cached (bool): keep data cached in memory
        **kwargs: arguments to pass to pd.read_csv

    Returns: pandas dataframe

    """
    return super().load(pd.read_excel, cached, **kwargs)
```

##### save

```
save(df, **kwargs)
```

Save dataframe to Excel

Parameters:

| Name       | Type   | Description                               | Default    |
| ---------- | ------ | ----------------------------------------- | ---------- |
| `df`       | `obj`  | pandas dataframe                          | *required* |
| `**kwargs` | `dict` | additional arguments to pass to df.to_csv | `{}`       |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, df, **kwargs):
    """
    Save dataframe to Excel

    Args:
        df (obj): pandas dataframe
        **kwargs (dict): additional arguments to pass to df.to_csv

    Returns: filename

    """
    opts = {**{'index':False},**kwargs}
    return super().save(df, 'to_excel', **opts)
```

#### ExcelPandasSheetsTarget

Bases: `_LocalPathTarget`

Saves dict of dataframes as sheets in a single Excel file, loads selectively by sheet

Source code in `oryxflow/targets/__init__.py`

```
class ExcelPandasSheetsTarget(_LocalPathTarget):
    """
    Saves dict of dataframes as sheets in a single Excel file, loads selectively by sheet

    """
    def load(self, keys=None, cached=False, **kwargs):
        """
        Load sheets from Excel file

        Args:
            keys (str/list): sheet name(s) to load. None loads all sheets
            cached (bool): keep data cached in memory
            **kwargs: arguments to pass to pd.read_excel

        Returns: dict of dataframes, single dataframe, or filtered dict

        """
        if self.exists():
            if not cached or not settings.cached or self.path not in cache:
                sheet_name = keys if keys is not None else None
                data = pd.read_excel(self.path, sheet_name=sheet_name, **kwargs)
                if cached or settings.cached:
                    cache[self.path] = data
                return data
            else:
                data = cache.get(self.path)
                if keys is None:
                    return data
                if isinstance(keys, str):
                    return data[keys]
                return {k: v for k, v in data.items() if k in keys}
        else:
            raise RuntimeError('Target does not exist, make sure task is complete')

    def save(self, data, **kwargs):
        """
        Save dict of dataframes as sheets in a single Excel file

        Args:
            data (dict): {sheet_name: dataframe}
            kwargs: additional arguments to pass to df.to_excel

        Returns: filename

        """
        opts = {**{'index': False}, **kwargs}
        (self.path).parent.mkdir(parents=True, exist_ok=True)
        with pd.ExcelWriter(self.path, engine='openpyxl') as writer:
            for sheet_name, df in data.items():
                df.to_excel(writer, sheet_name=sheet_name, **opts)
        return self.path
```

##### load

```
load(keys=None, cached=False, **kwargs)
```

Load sheets from Excel file

Parameters:

| Name       | Type         | Description                                  | Default |
| ---------- | ------------ | -------------------------------------------- | ------- |
| `keys`     | `str / list` | sheet name(s) to load. None loads all sheets | `None`  |
| `cached`   | `bool`       | keep data cached in memory                   | `False` |
| `**kwargs` |              | arguments to pass to pd.read_excel           | `{}`    |

Returns: dict of dataframes, single dataframe, or filtered dict

Source code in `oryxflow/targets/__init__.py`

```
def load(self, keys=None, cached=False, **kwargs):
    """
    Load sheets from Excel file

    Args:
        keys (str/list): sheet name(s) to load. None loads all sheets
        cached (bool): keep data cached in memory
        **kwargs: arguments to pass to pd.read_excel

    Returns: dict of dataframes, single dataframe, or filtered dict

    """
    if self.exists():
        if not cached or not settings.cached or self.path not in cache:
            sheet_name = keys if keys is not None else None
            data = pd.read_excel(self.path, sheet_name=sheet_name, **kwargs)
            if cached or settings.cached:
                cache[self.path] = data
            return data
        else:
            data = cache.get(self.path)
            if keys is None:
                return data
            if isinstance(keys, str):
                return data[keys]
            return {k: v for k, v in data.items() if k in keys}
    else:
        raise RuntimeError('Target does not exist, make sure task is complete')
```

##### save

```
save(data, **kwargs)
```

Save dict of dataframes as sheets in a single Excel file

Parameters:

| Name     | Type   | Description                                 | Default    |
| -------- | ------ | ------------------------------------------- | ---------- |
| `data`   | `dict` | {sheet_name: dataframe}                     | *required* |
| `kwargs` |        | additional arguments to pass to df.to_excel | `{}`       |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, data, **kwargs):
    """
    Save dict of dataframes as sheets in a single Excel file

    Args:
        data (dict): {sheet_name: dataframe}
        kwargs: additional arguments to pass to df.to_excel

    Returns: filename

    """
    opts = {**{'index': False}, **kwargs}
    (self.path).parent.mkdir(parents=True, exist_ok=True)
    with pd.ExcelWriter(self.path, engine='openpyxl') as writer:
        for sheet_name, df in data.items():
            df.to_excel(writer, sheet_name=sheet_name, **opts)
    return self.path
```

#### PqPandasTarget

Bases: `DataTarget`

Saves to parquet, loads to pandas dataframe

Source code in `oryxflow/targets/__init__.py`

```
class PqPandasTarget(DataTarget):
    """
    Saves to parquet, loads to pandas dataframe

    """
    def load(self, cached=False, **kwargs):
        """
        Load from parquet to pandas dataframe

        Args:
            cached (bool): keep data cached in memory
            **kwargs: arguments to pass to pd.read_parquet

        Returns: pandas dataframe

        """
        return super().load(pd.read_parquet, cached, **kwargs)


    def save(self, df, **kwargs):
        """
        Save dataframe to parquet

        Args:
            df (obj): pandas dataframe
            **kwargs (dict): additional arguments to pass to df.to_parquet

        Returns: filename

        """
        opts = {**{'compression':'gzip','engine':'pyarrow'},**kwargs}
        return super().save(df, 'to_parquet', **opts)
```

##### load

```
load(cached=False, **kwargs)
```

Load from parquet to pandas dataframe

Parameters:

| Name       | Type   | Description                          | Default |
| ---------- | ------ | ------------------------------------ | ------- |
| `cached`   | `bool` | keep data cached in memory           | `False` |
| `**kwargs` |        | arguments to pass to pd.read_parquet | `{}`    |

Returns: pandas dataframe

Source code in `oryxflow/targets/__init__.py`

```
def load(self, cached=False, **kwargs):
    """
    Load from parquet to pandas dataframe

    Args:
        cached (bool): keep data cached in memory
        **kwargs: arguments to pass to pd.read_parquet

    Returns: pandas dataframe

    """
    return super().load(pd.read_parquet, cached, **kwargs)
```

##### save

```
save(df, **kwargs)
```

Save dataframe to parquet

Parameters:

| Name       | Type   | Description                                   | Default    |
| ---------- | ------ | --------------------------------------------- | ---------- |
| `df`       | `obj`  | pandas dataframe                              | *required* |
| `**kwargs` | `dict` | additional arguments to pass to df.to_parquet | `{}`       |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, df, **kwargs):
    """
    Save dataframe to parquet

    Args:
        df (obj): pandas dataframe
        **kwargs (dict): additional arguments to pass to df.to_parquet

    Returns: filename

    """
    opts = {**{'compression':'gzip','engine':'pyarrow'},**kwargs}
    return super().save(df, 'to_parquet', **opts)
```

#### JsonTarget

Bases: `DataTarget`

Saves to json, loads to dict

Source code in `oryxflow/targets/__init__.py`

```
class JsonTarget(DataTarget):
    """
    Saves to json, loads to dict

    """
    def load(self, cached=False, **kwargs):
        """
        Load from json to dict

        Args:
            cached (bool): keep data cached in memory
            **kwargs: arguments to pass to json.load

        Returns: dict

        """
        def read_json(path, **opts):
            with path.open('r') as fhandle:
                df = json.load(fhandle)
            return df['data']
        return super().load(read_json, cached, **kwargs)


    def save(self, dict_, **kwargs):
        """
        Save dict to json

        Args:
            dict_ (dict): python dict
            **kwargs (dict): additional arguments to pass to json.dump

        Returns: filename

        """
        def write_json(path, _dict_, **opts):
            with path.open('w') as fhandle:
                json.dump(_dict_, fhandle, **opts)
        opts = {**{'indent':4},**kwargs}
        write_json(self.path, {'data':dict_}, **opts)
        return self.path
```

##### load

```
load(cached=False, **kwargs)
```

Load from json to dict

Parameters:

| Name       | Type   | Description                    | Default |
| ---------- | ------ | ------------------------------ | ------- |
| `cached`   | `bool` | keep data cached in memory     | `False` |
| `**kwargs` |        | arguments to pass to json.load | `{}`    |

Returns: dict

Source code in `oryxflow/targets/__init__.py`

```
def load(self, cached=False, **kwargs):
    """
    Load from json to dict

    Args:
        cached (bool): keep data cached in memory
        **kwargs: arguments to pass to json.load

    Returns: dict

    """
    def read_json(path, **opts):
        with path.open('r') as fhandle:
            df = json.load(fhandle)
        return df['data']
    return super().load(read_json, cached, **kwargs)
```

##### save

```
save(dict_, **kwargs)
```

Save dict to json

Parameters:

| Name       | Type   | Description                               | Default    |
| ---------- | ------ | ----------------------------------------- | ---------- |
| `dict_`    | `dict` | python dict                               | *required* |
| `**kwargs` | `dict` | additional arguments to pass to json.dump | `{}`       |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, dict_, **kwargs):
    """
    Save dict to json

    Args:
        dict_ (dict): python dict
        **kwargs (dict): additional arguments to pass to json.dump

    Returns: filename

    """
    def write_json(path, _dict_, **opts):
        with path.open('w') as fhandle:
            json.dump(_dict_, fhandle, **opts)
    opts = {**{'indent':4},**kwargs}
    write_json(self.path, {'data':dict_}, **opts)
    return self.path
```

#### MarkdownTarget

Bases: `DataTarget`

Saves to markdown (.md) and HTML (.html), loads markdown string

Source code in `oryxflow/targets/__init__.py`

```
class MarkdownTarget(DataTarget):
    """
    Saves to markdown (.md) and HTML (.html), loads markdown string

    """
    def load(self, cached=False, **kwargs):
        """
        Load from markdown file to string

        Args:
            cached (bool): keep data cached in memory
            **kwargs: arguments to pass to read function

        Returns: markdown string

        """
        def read_md(path, **opts):
            with path.open('r', encoding='utf-8') as fhandle:
                return fhandle.read()
        return super().load(read_md, cached, **kwargs)

    def save(self, md_string, **kwargs):
        """
        Save markdown string to .md and .html files

        Args:
            md_string (str): markdown string
            **kwargs (dict): additional arguments to pass to markdown.markdown

        Returns: filename

        """
        (self.path).parent.mkdir(parents=True, exist_ok=True)
        with self.path.open('w', encoding='utf-8') as fhandle:
            fhandle.write(md_string)
        html_path = self.path.with_suffix('.html')
        html_body = markdown.markdown(md_string, extensions=['tables'], **kwargs)
        html_string = (
            '<!DOCTYPE html>\n'
            '<html>\n'
            '<head>\n'
            '<meta charset="utf-8">\n'
            '<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/github-markdown-css/5.8.1/github-markdown.css">\n'
            '</head>\n'
            '<body>\n'
            '<article class="markdown-body">\n'
            f'{html_body}\n'
            '</article>\n'
            '</body>\n'
            '</html>\n'
        )
        with html_path.open('w', encoding='utf-8') as fhandle:
            fhandle.write(html_string)
        return self.path

    def invalidate(self):
        html_path = self.path.with_suffix('.html')
        if html_path.exists():
            html_path.unlink()
        if self.exists():
            self.path.unlink()
        return not self.exists()
```

##### load

```
load(cached=False, **kwargs)
```

Load from markdown file to string

Parameters:

| Name       | Type   | Description                        | Default |
| ---------- | ------ | ---------------------------------- | ------- |
| `cached`   | `bool` | keep data cached in memory         | `False` |
| `**kwargs` |        | arguments to pass to read function | `{}`    |

Returns: markdown string

Source code in `oryxflow/targets/__init__.py`

```
def load(self, cached=False, **kwargs):
    """
    Load from markdown file to string

    Args:
        cached (bool): keep data cached in memory
        **kwargs: arguments to pass to read function

    Returns: markdown string

    """
    def read_md(path, **opts):
        with path.open('r', encoding='utf-8') as fhandle:
            return fhandle.read()
    return super().load(read_md, cached, **kwargs)
```

##### save

```
save(md_string, **kwargs)
```

Save markdown string to .md and .html files

Parameters:

| Name        | Type   | Description                                       | Default    |
| ----------- | ------ | ------------------------------------------------- | ---------- |
| `md_string` | `str`  | markdown string                                   | *required* |
| `**kwargs`  | `dict` | additional arguments to pass to markdown.markdown | `{}`       |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, md_string, **kwargs):
    """
    Save markdown string to .md and .html files

    Args:
        md_string (str): markdown string
        **kwargs (dict): additional arguments to pass to markdown.markdown

    Returns: filename

    """
    (self.path).parent.mkdir(parents=True, exist_ok=True)
    with self.path.open('w', encoding='utf-8') as fhandle:
        fhandle.write(md_string)
    html_path = self.path.with_suffix('.html')
    html_body = markdown.markdown(md_string, extensions=['tables'], **kwargs)
    html_string = (
        '<!DOCTYPE html>\n'
        '<html>\n'
        '<head>\n'
        '<meta charset="utf-8">\n'
        '<link rel="stylesheet" href="https://cdnjs.cloudflare.com/ajax/libs/github-markdown-css/5.8.1/github-markdown.css">\n'
        '</head>\n'
        '<body>\n'
        '<article class="markdown-body">\n'
        f'{html_body}\n'
        '</article>\n'
        '</body>\n'
        '</html>\n'
    )
    with html_path.open('w', encoding='utf-8') as fhandle:
        fhandle.write(html_string)
    return self.path
```

#### PickleTarget

Bases: `DataTarget`

Saves to pickle, loads to python obj

Source code in `oryxflow/targets/__init__.py`

```
class PickleTarget(DataTarget):
    """
    Saves to pickle, loads to python obj

    """
    def load(self, cached=False, **kwargs):
        """
        Load from pickle to obj

        Args:
            cached (bool): keep data cached in memory
            **kwargs: arguments to pass to pickle.load

        Returns: dict

        """
        def funload(x):
            with x.open("rb" ) as fhandle:
                data = pickle.load(fhandle)
            return data
        return super().load(funload, cached, **kwargs)


    def save(self, obj, **kwargs):
        """
        Save obj to pickle

        Args:
            obj (obj): python object
            **kwargs (dict): additional arguments to pass to pickle.dump

        Returns: filename

        """
        with self.path.open("wb") as fhandle:
            pickle.dump(obj, fhandle, **kwargs)
        return self.path
```

##### load

```
load(cached=False, **kwargs)
```

Load from pickle to obj

Parameters:

| Name       | Type   | Description                      | Default |
| ---------- | ------ | -------------------------------- | ------- |
| `cached`   | `bool` | keep data cached in memory       | `False` |
| `**kwargs` |        | arguments to pass to pickle.load | `{}`    |

Returns: dict

Source code in `oryxflow/targets/__init__.py`

```
def load(self, cached=False, **kwargs):
    """
    Load from pickle to obj

    Args:
        cached (bool): keep data cached in memory
        **kwargs: arguments to pass to pickle.load

    Returns: dict

    """
    def funload(x):
        with x.open("rb" ) as fhandle:
            data = pickle.load(fhandle)
        return data
    return super().load(funload, cached, **kwargs)
```

##### save

```
save(obj, **kwargs)
```

Save obj to pickle

Parameters:

| Name       | Type   | Description                                 | Default    |
| ---------- | ------ | ------------------------------------------- | ---------- |
| `obj`      | `obj`  | python object                               | *required* |
| `**kwargs` | `dict` | additional arguments to pass to pickle.dump | `{}`       |

Returns: filename

Source code in `oryxflow/targets/__init__.py`

```
def save(self, obj, **kwargs):
    """
    Save obj to pickle

    Args:
        obj (obj): python object
        **kwargs (dict): additional arguments to pass to pickle.dump

    Returns: filename

    """
    with self.path.open("wb") as fhandle:
        pickle.dump(obj, fhandle, **kwargs)
    return self.path
```

# Changelog

Every release and what changed in it. Newest first; versions follow calver (`YY.M.D`) and match `oryxflow.__version__`.

All notable changes to **oryxflow** are recorded here. This file is read by humans *and* by AI coding agents diagnosing regressions after an upgrade, so the format is load-bearing:

- Newest first. One `## [version] - YYYY-MM-DD` heading per release; version is calver `YY.M.D` matching `setup.py` / `oryxflow.__version__`. Unreleased work goes under `## [Unreleased]`.
- Group bullets under `### Added` / `### Changed` / `### Deprecated` / `### Removed` / `### Fixed` / `### Security` (Keep a Changelog: https://keepachangelog.com/).
- **Every breaking change is a bullet that STARTS with the literal token `BREAKING:`** and carries a same-bullet `Migration:` clause with the old→new fix.
- **Name the actual symbol in backticks** (`` `Task.persist` ``, `` `RunResult.summary()` ``), never prose. Agents grep this file for the symbol in their traceback.

## [Unreleased]

## [26.7.26] - 2026-07-26

### Changed

- BREAKING: `TaskAggregator` is now a `requires()`-based group node instead of a task that yields its members from `run()`. Because the group is a regular DAG node, it works with `Workflow` / `WorkflowMulti` (previously every call raised `UnknownParameterException: ... unknown parameter flows`), `preview()` expands it to show each member, and per-flow `path`/`env`, `reset_upstream()` and `FlowExport` reach its members. The group still saves nothing of its own and is complete when every task it requires is complete. The old form now raises a `RuntimeError` at construction naming the fix. Migration: move the members from `yield` statements in `run()` into `requires()` (or `@oryxflow.requires`) and leave `run()` empty — `class Agg(oryxflow.tasks.TaskAggregator): def run(self): yield T1(); yield T2()` becomes `class Agg(oryxflow.tasks.TaskAggregator): def requires(self): return [T1(), T2()]`.

## [26.7.21] - 2026-07-21

### Security

- Releases are now published to PyPI via GitHub Actions **Trusted Publishing** (OIDC) instead of a stored API token, and every uploaded file carries a PyPI-recorded **attestation** (PEP 740 / Sigstore) proving it was built from this repository by CI. Verify on the PyPI file detail page for this release. No install-side change — `pip install oryxflow` is unaffected.

## [26.7.12] - 2026-07-12

### Added

- Automatic code invalidation, on by default (`settings.code_version_auto = True`): every task derives its code identity from the AST hash of its own class plus the project-local symbols it transitively references (`codehash.task_hashes`, `'<relpath>::<symbol>'` granularity), so a real logic edit (in the task **or a helper it calls**) reruns the task and everything downstream on the next `run()`, overwriting in place — while editing an unrelated sibling task in the same file reruns nothing (one monolithic `tasks.py` stays cheap). References to other Task classes are dependency wiring, never a code dependency (a pinned upstream's unbumped edit can't ripple through `requires()` mentions); unresolvable constructs degrade conservatively to whole-module granularity. No attribute to maintain, and comment/docstring/formatting edits never rerun (AST normalization). Existing caches are grandfathered on first contact (baseline stamped, zero reruns). Set `settings.code_version_auto = False` for explicit-only tracking. The functional API is covered automatically (auto is ambient, no per-task surface). Records live in `<dirpath>/.oryxflow-code-status.json` and travel with the data dir.
- `Task.code_version` (str or int, default `None`): a per-task **pin** that suspends automatic tracking of that task's own logic — it recomputes only on a deliberate bump (the task and everything downstream), for expensive tasks where a refactor-triggered recompute must be a decision, or logic the hash can't see. Records are mode-aware (they store both the token and the `source_hashes` as of the last materialization), and the `code_version` line itself is stripped by the AST normalization (typing it in / deleting / bumping it is a token change, never a source change), so pinning/unpinning unchanged code never recomputes ("just resumes"), an edit masked during a pinned-unbumped window is caught the moment the pin comes off, and pinning in the same edit as a logic change forces a rerun instead of blessing stale output.
- Dependency propagation folds **output identity** (`output_id`, fresh per actual materialization, preserved across re-stamps and `accept_code`): downstream reruns exactly when an upstream rematerialized — pin toggles and accepts never ripple, and a `reset()`+rerun upstream propagates downstream even across separate builds.
- Staleness advisory for pinned tasks: code changed without a bump → cached output is reused and the run warns via `StalenessWarning` (a `UserWarning` subclass, visible without `enable_logging()`), a loguru record, a `code_warning` event, and `RunResult.warnings`. The printed/logged channels dedupe per process on the message — parameterized instances of one family produce identical text, and a `WorkflowMulti` run is one build per flow over shared upstreams, so per-task dedupe would still flood stdout — re-arming when the condition changes or the affected tasks rerun/are accepted; `RunResult.warnings` lists each distinct message once per run (`MultiRunResult.warnings` dedupes across flows), and only the event stream records every occurrence.
- `oryxflow.accept_code(task)` / `accept_code()`: acknowledge an output-equivalent code change without rerunning. With a task instance it re-stamps the task **and its entire upstream dep tree** (post-order), stamping a fresh baseline record for outputs that have none yet (this is what clears the `output predates current code` mtime-guard warning after an upgrade); `Workflow.accept_code(task=None)` / `WorkflowMulti.accept_code(task=None, flow=None)` wrap it; called bare they cover **every imported task family that resolves with the flow's parameters** (a multi-final pipeline is fully blessed in one call, from a fresh process — no prior run needed), and a list of tasks is accepted everywhere (on `WorkflowMulti` prefer the flow method — the module-level bulk form doesn't know the flows' parameters). Prints a one-line summary of what it re-stamped (or that nothing was accepted). The tree walk is fault-isolated: a task whose `requires()`/ `output()` raises is skipped and reported instead of aborting the walk (a broken `requires()` also can't poison the node's own blessing). Never touches `output_id`, so accepting never triggers downstream recomputes.
- `TaskData.keep_versions` (default `False`): with `code_version` set, outputs live under a readable `.../<Task>/v<version>/` segment so old versions survive bumps (explicit pins only; auto-tracked tasks overwrite in place).
- Expensive-recompute guard (`settings.code_version_auto_expensive_s`, default 600): an auto-tracked task whose last materialization (recorded as `duration_s`) took longer is held complete when its code changes and the run warns (`StalenessWarning`, all channels) with the three exits — `reset()` to recompute, `accept_code` if output-equivalent, or pin with `code_version` — so a refactor can't silently burn a long run. `None`/`0` disables the guard.
- Records carry schema/interpreter tags (`state.RECORD_V`, `py`): a record with a different/missing `v` or Python minor is treated as unverifiable — complete, then silently re-stamped (grandfather trust level, `output_id` preserved) — never a mass rerun after an upgrade.
- `build()` mtime-revalidates code hashes at most once per module per build (`codehash.freeze()`/`unfreeze()`), keeping the auto-hash overhead on small DAGs low.
- Event stream `oryxflow.events`: every run appends `run_started` / `task_ran` / `task_failed` / `run_finished` / `code_warning` / `code_accepted` / `task_log` events to `.oryxflow/events.jsonl` (stable head; earlier months offload to `events-YYYYMM.jsonl`, immutable). Plain JSONL — `tail`/`grep`/`jq` work; writes are async and never fail a run; disable with `settings.events = False`. Query via `oryxflow.events.status()` (session-start: pending warnings, last run per family, recent failures), `events.runs(task_family=, flow=, last=)`, `events.iter_events()` — all return data and print nothing; `events.print_status()` prints the status summary (the session-start orientation call for scripts and `python -c`).
- `RunResult.run_id`, `RunResult.reasons` (`{task_id: 'output missing' | 'code change (auto: <file>::<symbol>)' | 'code change (a -> b)' | 'upstream rerun'}`), `RunResult.warnings`. `MultiRunResult` gains aggregate `.ran`/`.complete`/`.failed`/`.reasons`/`.warnings` across flows. `task_ran` events carry params, code fingerprint, source hashes, `auto` flag, git SHA/dirty, duration and the rerun reason; `WorkflowMulti` stamps each per-flow build's events with its flow name.
- Task-authored `self.logger.*(...)` lines are captured as `task_log` events during a build (works with logging disabled), so in-run scalars become queryable memory.
- New settings: `settings.events`, `settings.eventspath`, `settings.state_filename`.

### Changed

- `settings.db` (unused) renamed to `settings.state_filename` (the per-data-dir record file name, `.oryxflow-code-status.json`).

## [26.7.11] - 2026-07-11

### Changed

- Documentation rewrite and PyPI packaging updates; no API changes.

## [26.6.6] - 2026-06-06

### Added

- Initial release of `oryxflow`: the self-contained task engine (`Task`, `requires`/`inherits`, the parameter set, `Workflow`/`WorkflowMulti`, targets and task I/O formats), with no external workflow-engine dependency.
# Blog

# Blog

Notes on making data analysis **faster, cheaper, and more trustworthy** — for humans and AI coding agents. Reproducible pipelines, data lineage, honest tool comparisons, and why a lightweight caching DAG changes how you build data-science workflows.

## Start here

- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** — the positioning in one page: reproducibility, lineage, and trustworthy AI data analysis, plus when *not* to reach for it.
- **[Reproducible data science workflows in Python](https://docs.oryxflow.dev/blog/reproducibility/reproducible-data-science-workflows-in-python/index.md)** — what makes a workflow reproducible, and the missing middle between notebooks and orchestrators.

**Reproducibility & caching**

- [Stop rerunning your whole pipeline](https://docs.oryxflow.dev/blog/caching/cache-intermediate-dataframes-python/index.md) — caching intermediate DataFrames without the stale-`.pkl` graveyard.
- [Cache intermediate DataFrames in Python](https://docs.oryxflow.dev/blog/caching/cache-intermediate-dataframes-in-python/index.md) — why hand-rolled pickle caches go stale, and caching by task identity instead.
- [From notebook to a reproducible, cached pipeline](https://docs.oryxflow.dev/blog/reproducibility/notebook-to-reproducible-pipeline/index.md) — migrate an analysis into tasks, one step at a time.
- [4 reasons your machine learning code is bad](https://docs.oryxflow.dev/blog/machine-learning/why-machine-learning-code-is-bad/index.md) — the failure modes a task DAG fixes.

**Tool comparisons**

- [oryxflow vs the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md) — the full framework landscape for iterative, AI-assisted analysis.
- [oryxflow vs Airflow](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md) — research workflows vs production orchestration.
- [oryxflow vs Prefect](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-prefect/index.md) — zero-config code-aware caching vs configurable orchestration.
- [oryxflow vs Dagster](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-dagster/index.md) — a lightweight research loop vs an asset platform.
- [oryxflow vs DVC](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-dvc/index.md) — native Python task identity vs file-hash data versioning.
- [MLflow or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md) — experiment tracking vs workflow caching (and DVC).
- [MLflow alternatives](https://docs.oryxflow.dev/blog/comparisons/mlflow-alternatives/index.md) — the experiment-tracker landscape, and where a caching DAG fits beside (not against) it.
- [Airflow alternatives for data science](https://docs.oryxflow.dev/blog/comparisons/airflow-alternatives-for-data-science/index.md) — research-loop tools vs production orchestrators, and when you don't need an orchestrator at all.

**Trustworthy AI-assisted data science**

- [Why a caching DAG makes your AI coding agent a better data scientist](https://docs.oryxflow.dev/blog/ai-agents/caching-dag-for-ai-coding-agents/index.md) — where agents fail at pipeline work, and what removes those failures.
- [Best practices for AI-assisted data analysis](https://docs.oryxflow.dev/blog/ai-agents/best-practices-ai-assisted-data-analysis/index.md) — how to keep AI-generated analysis reproducible, auditable, and honest about correctness.
- [The best Claude Code plugins and tools for data science](https://docs.oryxflow.dev/blog/ai-agents/best-claude-code-plugins-for-data-science/index.md) — a by-the-job roundup of the AI data-science tooling landscape.
- [The best AI tools for data analysis](https://docs.oryxflow.dev/blog/ai-agents/best-ai-tools-for-data-analysis/index.md) — the layers of the AI data stack, and the trust layer most of them miss.

**LLM evals**

- [Your eval platform deletes your results in 14 days](https://docs.oryxflow.dev/blog/ai-agents/llm-eval-results-deleted-retention/index.md) — eval platforms are working sets, not archives; keep the scored rows yourself.
- [LLM evals are a parameter sweep](https://docs.oryxflow.dev/blog/ai-agents/llm-evals-are-a-parameter-sweep/index.md) — the scoring is new, the matrix underneath it is a solved problem.
- [Cheap, durable LLM evals: pydantic-evals + oryxflow](https://docs.oryxflow.dev/blog/guides/cheap-durable-llm-evals/index.md) — how a metered eval matrix gets expensive, and what to cache instead.

**Practical patterns**

- [Parameter sweeps without rerunning upstream steps](https://docs.oryxflow.dev/blog/caching/parameter-sweeps-without-rerunning/index.md) — compare many configurations while shared steps compute once.
- [Migrate from d6tflow to oryxflow](https://docs.oryxflow.dev/blog/guides/migrate-from-d6tflow-to-oryxflow/index.md) — a whole-word package rename, not an API port; your cache comes with you.
- [When *not* to use oryxflow](https://docs.oryxflow.dev/blog/guides/when-not-to-use-oryxflow/index.md) — an honest guide to non-fit.

______________________________________________________________________

## All posts

# 4 Reasons why your machine learning code is bad

## Problems with your current workflow

Your current workflow probably chains several functions together like in the example below. While quick, it likely has many problems:

- it doesn't scale well as you add complexity
- you have to manually keep track of which functions were run with which parameter
- you have to manually keep track of where data is saved
- pickle files are neither compressed nor portable

```
import pandas as pd

import sklearn.svm, sklearn.metrics



def get_data():

    data = download_data()

    data = clean_data(data)

    data.to_pickle('data.pkl')



def preprocess(data):

    data = apply_function(data)

    return data



# flow parameters

reload_source = True

do_preprocess = True



# run workflow

if reload_source:

    get_data()



df_train = pd.read_pickle('data.pkl')

if do_preprocess:

    df_train = preprocess(df_train)

model = sklearn.svm.SVC()

model.fit(df_train.iloc[:,:-1], df_train['y'])

print(sklearn.metrics.accuracy_score(df_train['y'],model.predict(df_train.iloc[:,:-1])))
```

# DAGs to the rescue

Instead of linearly chaining functions, data science code is better written as a set of tasks with upstream dependencies. That is your data science workflow should be a DAG.

So instead of writing a function that does:

```
def process_data(data, parameter):



    if parameter:

        data = do_stuff(data)

    else:

        data = do_other_stuff(data)



    data.to_pickle('data.pkl')

    return data
```

You are better of writing tasks that you can chain together as a DAG:

```
class TaskProcess(oryxflow.tasks.TaskPqPandas): # define output format



    def requires(self):

        return TaskGetData() # define dependency



    def run(self):

        data = self.inputLoad() # load input data

        data = do_stuff(data) # process data

        self.save(data) # save output data
```

The benefits of doings this are:

- All tasks follow the same pattern no matter how complex your workflow gets
- You have a scalable input requires() and processing function run()
- You can quickly load and save data without having to hardcode filenames
- If the input task is not complete it will automatically run
- If input data or parameters change, the function will automatically rerun

# A short machine learning example

Below is a stylized example of a machine learning flow which is expressed as a DAG. In the end you just need to run TaskTrain() and it will automatically know which dependencies to run. For a full example see

```
import pandas as pd

import sklearn, sklearn.svm

import oryxflow



# define workflow

class TaskGetData(oryxflow.tasks.TaskPqPandas):  # save dataframe as parquet



    def run(self):        

        data = download_data()

        data = clean_data(data)

        self.save(data) # quickly save dataframe



class TaskPreprocess(oryxflow.tasks.TaskCachePandas):  # save data in memory

    do_preprocess = oryxflow.BoolParameter(default=True) # parameter for preprocessing yes/no



    def requires(self):

        return TaskGetData() # define dependency



    def run(self):

        df_train = self.inputLoad() # quickly load required data

        if self.do_preprocess:

            df_train = preprocess(df_train)

        self.save(df_train)



class TaskTrain(oryxflow.tasks.TaskPickle): # save output as pickle

    do_preprocess = oryxflow.BoolParameter(default=True)



    def requires(self):

        return TaskPreprocess(do_preprocess=self.do_preprocess)



    def run(self):

        df_train = self.inputLoad()

        model = sklearn.svm.SVC()

        model.fit(df_train.iloc[:,:-1], df_train['y'])

        self.save(model)



# Check task dependencies and their execution status

oryxflow.preview(TaskTrain())



'''

+--[TaskTrain-{'do_preprocess': 'True'} (PENDING)]

   +--[TaskPreprocess-{'do_preprocess': 'True'} (PENDING)]

      +--[TaskGetData-{} (PENDING)]

'''



# Execute the model training task including dependencies

oryxflow.run(TaskTrain())



'''

Scheduled 3 tasks

* 3 ran successfully

* 0 complete

* 0 failed

'''



# Load task output to pandas dataframe and model object for model evaluation

model = TaskTrain().output().load()

df_train = TaskPreprocess().output().load()

print(sklearn.metrics.accuracy_score(df_train['y'],model.predict(df_train.iloc[:,:-1])))

# 0.9733333333333334
```

# Conclusion

Writing machine learning code as a linear series of functions likely creates many workflow problems. Because of the complex dependencies between different ML tasks it is better to write them as a DAG. oryxflow makes it very easy for you. Alternatively you can use luigi and airflow but they are more optimized for ETL than data science.

## Frequently asked questions

### Why does my machine learning code become unreliable over time?

A typical ML script chains functions linearly, so you manually track which functions ran with which parameters and where each output was saved. As complexity grows, that bookkeeping breaks: stale intermediate files, forgotten reruns after a parameter change, and pickle outputs that aren't compressed or portable. Writing the workflow as a DAG of tasks fixes it — oryxflow reruns a task automatically when its inputs or parameters change, so results stay consistent with the code that produced them.

### How do I make ML code reproducible without a big framework?

Express your workflow as a small dependency graph instead of a linear chain of functions. In oryxflow each step is a task that declares what it requires, loads its inputs, and saves its output; the engine runs dependencies in the right order, caches each result, and reruns a step only when its code or parameters change. It's a local, zero-infrastructure Python library — no scheduler or server — so you get reproducibility and cached reruns without adopting Airflow-scale tooling.

### Should I use Airflow or Luigi for a data science workflow?

Airflow and Luigi are built for ETL and production scheduling, not the data-science research loop. For iterating on models locally — where you want cached outputs, automatic reruns when code or parameters change, and lineage without standing up a server — a lightweight task graph fits better. oryxflow gives you the DAG model for data science specifically: declare dependencies, load inputs, save outputs, and skip anything already computed.

# Best practices for AI-assisted data analysis

*AI coding agents write plausible analysis fast. The hard part — is it reproducible, and is it right? — hasn't changed. These practices are about making AI-generated analysis you can actually trust.*

Ask a coding agent to load a dataset, engineer features, train a model, and compare a few configurations, and it will produce clean, plausible code in seconds. That speed is real and it is worth having. But "plausible code, fast" and "a correct, reproducible analysis you can stand behind" are different bars, and the distance between them is exactly where AI-generated work quietly goes wrong.

The good news: most of that gap is a *workflow* problem, not an intelligence problem. If you give the agent a structure that makes reproducibility automatic, it stops making a whole class of mistakes — stale intermediates, silent re-runs, results nobody can trace. What that structure *cannot* do is make your statistics correct. So the practices below split cleanly: the first seven you can hand to your tooling, and the last one you can never hand to anyone.

[oryxflow](https://github.com/oryxintel/oryxflow) is a small, local-first Python library that turns a data-science script into a cached, dependency-aware graph of tasks — zero infrastructure, no server, no telemetry. Its [Claude Code plugin](https://github.com/oryxintel/oryxflow-claude-plugin) teaches an agent to work inside that structure. Together they enforce most of these practices for you; `/oryxflow:init-project` is the on-ramp.

## 1. Structure work as a dependency graph, not a linear script

An agent loses pipeline state across turns. A long linear script gives it no reliable memory of what has already run or whether it is still valid — so it re-derives that picture every turn and gets it wrong. Model the work as tasks with declared dependencies instead. Each step names its inputs, the engine runs them in order, and "what depends on what" is data the agent can read rather than reconstruct.

```
import oryxflow

class GetData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = load_raw()                  # your loader
        self.save(df)

@oryxflow.requires(GetData)
class Features(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()            # upstream output, typed and loaded for you
        self.save(build_features(df))

oryxflow.Workflow(Features, {}).run().outputLoad()
```

Note there is no file-path plumbing and no `to_parquet`/`read_parquet`. The base class you pick (`TaskPqPandas`, `TaskPickle`, `TaskCachePandas`, …) drives type-based, zero-config I/O — the single most common place hand-written analysis code rots.

## 2. Cache expensive steps so iteration is cheap

The agent iterates. Every turn it might touch feature code, model params, or a plot. If each turn re-pays for the big join and the slow fit, iteration is expensive and the agent starts taking shortcuts to avoid the wait. oryxflow skips any task whose output already exists, so the costly upstream steps run once and every later turn is fast. Cheap iteration is not just ergonomics — it removes the incentive to cut corners.

## 3. Keep a durable link between code and output

This is the one that bites hardest. The agent edits feature code, forgets to regenerate the saved features, and trains on stale data. Nothing errors. The pipeline runs; the numbers are just wrong. oryxflow watches your task code and, when the body of a step changes, automatically invalidates that step and everything downstream — so the next run recomputes exactly what changed and nothing else. You never evaluate new code on old output, and you never blow away the whole cache to be safe.

## 4. Keep lineage you can audit

"Which code and which data produced this result?" should have an answer you can look up, not one you reconstruct from memory. oryxflow records run events to `.oryxflow/events.jsonl` locally — a plain, greppable trail of what ran, when, and why it was (or wasn't) recomputed. When a number looks off, that file is where you start.

## 5. Verify the rerun actually happened

Do not trust that an edit took effect just because the agent said so. After a change, confirm the affected task actually recomputed — check that its output timestamp moved, or read the lineage trail from practice 4. Agents are confident narrators; "I've updated the features and retrained" is a claim, not evidence. `flow.preview()` shows you what the engine considers complete versus pending before you run, so you can see the plan and catch a step that *should* be stale but isn't.

## 6. Keep exploratory work separate from the pipeline

Early exploratory analysis should stay loose — scratch cells, quick plots, throwaway checks. Don't prematurely formalize it, and don't let it silently become load-bearing either. When a piece of exploration earns its place — you'll rerun it, others depend on it, it feeds a result — promote it into a task deliberately. The plugin's `/oryxflow:migrate` command does that conversion for you, lifting a notebook or script into cached tasks when it's ready, so the boundary between "playing" and "pipeline" stays honest.

## 7. Version your data so results are regenerable

Reproducible code on top of a mutable, unversioned dataset is only half-reproducible. Track your data alongside your code so any result can be regenerated from a known state. `/oryxflow:init-gitlfs` sets up Git LFS for the data directory, so inputs and outputs are versioned the same way the code is.

## 8. Never outsource judgment — reproducible is not correct

Here is the honesty that the other seven practices exist to protect: **no tool makes your statistics correct.** oryxflow guarantees that the same code and data give the same result, that stale steps recompute, that lineage is auditable. It does *not* check that your join keys are right, that your validation is honest, or that your features don't leak the target. Those are judgment, and judgment does not delegate — not to a library, and emphatically not to the agent.

So keep these firmly in human hands, every time:

- **Sanity-check joins and aggregations.** Row counts before and after, spot-check a few keys, confirm the grain is what you think it is. A silent fan-out or dropped rows survives any amount of caching.
- **Hold out real validation data** and keep it untouched until the end. An agent optimizing a metric will happily overfit to whatever you let it see.
- **Watch for leakage and lookahead.** A feature computed with information from the future, or a target that sneaks into the inputs, produces beautiful, reproducible, worthless results.
- **Read the numbers skeptically.** An accuracy that jumped suspiciously, a distribution that shifted, a metric that's too good — treat these as bugs to explain, not wins to ship.

Reproducibility is what lets you *investigate* correctness efficiently: because the pipeline is stable and traceable, when a number looks wrong you can trust that the code and data in front of you are what produced it. That's the foundation judgment stands on — not a substitute for it.

## The practices at a glance

| Practice                            | What it prevents                                      | How oryxflow / the plugin helps                                 |
| ----------------------------------- | ----------------------------------------------------- | --------------------------------------------------------------- |
| Dependency graph, not linear script | Agent losing pipeline state across turns              | Tasks with `@requires`; the engine runs the DAG in order        |
| Cache expensive steps               | Re-paying for the big join every turn; corner-cutting | Skips any task whose output exists                              |
| Link code to output                 | Training new code on stale data                       | Automatic AST code-change invalidation of the step + downstream |
| Auditable lineage                   | "No idea what produced this number"                   | `.oryxflow/events.jsonl`, local and greppable                   |
| Verify the rerun happened           | Trusting an edit that didn't take                     | `flow.preview()` shows complete vs pending                      |
| Separate EDA from pipeline          | Scratch work silently becoming load-bearing           | `/oryxflow:migrate` promotes it when it earns it                |
| Version your data                   | Results you can't regenerate                          | `/oryxflow:init-gitlfs`                                         |
| Never outsource judgment            | Reproducible-but-wrong analysis                       | **Nothing** — this one is yours                                 |

Orchestrators (Airflow, Prefect, Dagster) and experiment trackers (MLflow) are complementary layers here, not competitors — they schedule and record; oryxflow is the local, code-tight inner loop the agent iterates in.

## Takeaway

AI makes analysis *fast to write*. Trust comes from making it *reproducible by structure* and *correct by judgment* — two different jobs. Let oryxflow and its plugin enforce the reproducible half automatically, and keep the correctness half where it belongs: with a skeptical human reading the numbers. `/oryxflow:init-project` sets up the foundation; you bring the judgment.

```
pip install oryxflow
```

## Frequently asked questions

### What are best practices for AI-assisted data analysis?

Structure the work as a dependency graph instead of a linear script, cache expensive steps, keep a durable link between code and output, retain auditable lineage, verify that edits actually reran, separate exploration from the pipeline, version your data, and never outsource statistical judgment. The first seven you can hand to tooling; oryxflow and its Claude Code plugin enforce them, and the last stays with you.

### How do I keep AI-generated data analysis reproducible and trustworthy?

Give the agent a structure that makes reproducibility automatic: a caching, dependency-aware task graph that reruns only what a code or data change affects and logs what ran. oryxflow provides code-change invalidation and a greppable .oryxflow/events.jsonl trail, so any result traces back to its inputs. Reproducible is not correct, though, so you still sanity-check joins, hold out validation data, and watch for leakage.

### What tools help make AI data analysis reproducible?

Coding agents write the analysis, notebooks display it, and trackers like MLflow record runs, but none guarantee the computation is reproducible. That reproducibility layer is where a local-first caching library fits. oryxflow turns a data-science script into a cached, dependency-aware graph with code-change invalidation and local lineage, and its Claude Code plugin teaches the agent to work inside that structure. Orchestrators like Airflow are a complementary scheduling layer.

**Read next**

- [Claude Code for data science](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md)
- [Why a caching DAG makes your AI coding agent a better data scientist](https://docs.oryxflow.dev/blog/ai-agents/caching-dag-for-ai-coding-agents/index.md)
- [When *not* to use oryxflow](https://docs.oryxflow.dev/blog/guides/when-not-to-use-oryxflow/index.md)
- [From notebook to pipeline](https://docs.oryxflow.dev/blog/reproducibility/notebook-to-reproducible-pipeline/index.md)
- [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)
- [Build with Claude Code: the oryxflow plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md)
- [Why a plugin, not just a library](https://docs.oryxflow.dev/docs/claude-plugin/why/index.md)
- [Managing workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md)
- Plugin repo: <https://github.com/oryxintel/oryxflow-claude-plugin>

# Airflow alternatives for data science: an honest roundup

*Most people who search "Airflow alternatives" want a lighter orchestrator. Some of them don't need an orchestrator at all — they need their research loop to stop recomputing unchanged steps. This roundup covers both.*

Apache Airflow is a production workflow **scheduler**: it runs pipelines on a schedule, across a cluster, with retries, backfills, alerting, SLAs, and a UI that shows the state of every run. That's a lot of machinery — a scheduler, a metadata database, and (in 3.x) a dag-processor and API server — and for a nightly job that must run at 2 a.m. and page someone when it breaks, it's exactly the machinery you want.

The reason people go looking for alternatives is usually one of two things. Either Airflow is *more* than the job needs (a solo analyst doesn't want to stand up a scheduler and a database to try one more feature), or its ergonomics grate (heavy DAG definitions, XCom's small-data limits, local-dev friction). Below is a fair map of the real orchestrator alternatives — what each one is for, and when it's the right Airflow replacement — followed by the twist that matters for a lot of data scientists: you may be shopping in the wrong aisle entirely.

## The real orchestrator alternatives

These are the tools that do roughly Airflow's job — schedule and run DAGs, reliably, often across infrastructure. If you genuinely need a scheduler, pick from here.

### Prefect

Prefect is a modern, Python-native orchestrator. You decorate functions with `@flow` and `@task` and get dynamic flows, automatic retries, scheduling, concurrency limits, and a server/UI (self-hosted or Prefect Cloud) that observes every run. It reads far more like ordinary Python than Airflow's DAG objects, and it supports opt-in result caching you configure. It's the natural pick when you want Airflow's reliability with a lighter, code-first authoring experience.

### Dagster

Dagster is an asset-oriented orchestrator. Instead of tasks you declare *software-defined assets*, and Dagster gives you a rich web UI, scheduling, sensors, partitions and backfills, and IO managers that persist each asset's output automatically. It's a strong fit for a data team that wants to see asset freshness, backfill a partition, and reason about lineage in one place. The full experience assumes a running Dagster instance and its UI.

### Luigi

Luigi (from Spotify) is the spiritual ancestor of a lot of this tooling: Python tasks with `requires()`/`output()`/`run()`, and dependency resolution by output existence — a task is "done" when its output file exists. It's lightweight and has no mandatory server, which makes it appealing for simple batch pipelines. The tradeoffs are that scheduling is basic (often paired with cron), and caching is existence-only, so a code change doesn't invalidate anything until you delete files by hand.

### Kedro

Kedro isn't a scheduler at all; it's a project-structure and data-catalog framework. It gives you an opinionated layout, a `catalog.yml` that maps named datasets to storage, and clean, testable pipeline code. Teams reach for it to bring engineering discipline to notebook-grown projects. You then run those pipelines *on* something else — Airflow, Prefect, or a Kedro plugin — so it's less an Airflow replacement than a complement to whatever runs the DAG.

### Metaflow

Metaflow (from Netflix) is built for the data-scientist workflow end to end: you write flows as Python classes, it auto-versions the artifacts you assign to `self`, and it scales the same code from your laptop to the cloud (historically AWS-oriented). It's an excellent choice when you want experiment-friendly ergonomics *and* a path to scale, provided you're on its supported infrastructure and comfortable with a Metaflow-owned datastore.

### Flyte and Argo

Flyte and Argo Workflows are Kubernetes-native orchestrators. They run containerized, strongly-typed pipelines at scale with reproducible, versioned executions. If your world is already Kubernetes and you need serious distributed throughput and multi-tenant isolation, this is the tier that delivers it. The cost is operational: you're running (or buying) real cluster infrastructure, which is overkill for local research.

### ZenML

ZenML is an MLOps framework that sits above the orchestrators: you write pipelines once and run them on a configurable stack (local, Airflow, Kubeflow, cloud). It offers step-level caching out of the box and integrates experiment tracking and model registries. It's a good fit when you want portable pipelines and a tidy MLOps stack without marrying one backend.

### Just cron and scripts

Worth saying plainly, because it's often the right answer: a couple of Python scripts and a cron entry (or a systemd timer, or a GitHub Action on a schedule) is a legitimate Airflow alternative for small, stable jobs. You lose the UI, retries, and dependency graph, but you also carry zero infrastructure. If your "pipeline" is two steps that run nightly and rarely change, reaching for an orchestrator is the over-engineering, not the solution.

## Do I even need an orchestrator?

Here's the twist that most "Airflow alternatives" lists miss. A large share of the people searching for one are not trying to schedule anything. They're iterating — EDA, feature engineering, model comparison — editing the same code dozens of times a day, and the thing that actually hurts is waiting on the slow steps to recompute *every single time*, even the ones they didn't touch.

That is a different job from orchestration. A scheduler's core competency is "run this DAG reliably, later, somewhere." The research loop's core competency is "when I change one thing, rerun exactly what that change affects and nothing else, right now." An orchestrator can be bent toward the second job, but it's not what it optimizes for — which is why standing one up for local research so often feels like infrastructure tax with no payoff.

If that's the itch you're actually scratching, the tool you want isn't a lighter scheduler. It's a cache that understands your dependency graph and your code.

## Where oryxflow fits (and where it doesn't)

**oryxflow is a small, local-first Python library that turns your analysis scripts and notebooks into a cached, dependency-aware task graph, skips any task whose output already exists, and reruns exactly what a parameter, data, or code change affects.** You declare typed `Task` classes, wire dependencies with `@oryxflow.requires`, and each task `save()`s its output; the engine runs the DAG in dependency order and reuses everything that's still valid. It's a `pip install` — no server, no database, no account, no telemetry.

The part that makes it distinct from the orchestrators above is how it decides what's stale. oryxflow tracks each task's code — and every helper it references — comparing what your code *does*, not how it's written, so comments and reformatting don't count. Edit one function and the next run recomputes exactly the tasks that use it and everything downstream, while the expensive upstream stays cached. Every run appends to a plain, greppable lineage log at `.oryxflow/events.jsonl`, so you can trace any output back to the code and inputs that made it. And because it's built to be driven by an AI coding agent, it ships a [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md) (a skill plus slash commands — not an MCP server) that teaches the agent to check the cache, verify its own edits actually reran, and never build on a stale result.

Be clear about the boundary, because it's the whole point of putting oryxflow in a *roundup* rather than at the top of it: **oryxflow makes your analysis reproducible, not correct** — it reruns the right tasks; it doesn't check that your logic is right. And it is emphatically **not** a scheduler. It does not run jobs on a cron trigger, it does not retry or alert, and it does not execute distributed production workloads across a cluster. If you need any of those, you need one of the orchestrators above. oryxflow is complementary to them: iterate locally in oryxflow, then wrap the stable pipeline in Airflow, Prefect, or Dagster when it's ready to be scheduled — a mechanical step, because your logic is already decomposed into tasks with explicit dependencies.

## What's the best lightweight alternative to Airflow?

It depends on which half of the problem you have, and the honest answer names two different tools:

- If you need a **real scheduler** but Airflow is heavier than you want, **Prefect** is the usual lightweight pick — Python-native authoring, retries, scheduling, and a UI, with far less ceremony. For a Kubernetes-scale need it's **Flyte or Argo**; for laptop-to-cloud data-science ergonomics it's **Metaflow**; for the simplest stable jobs it's **cron**.
- If you don't actually need scheduling and just want your **local research loop to stop recomputing unchanged steps**, a code-aware cache like **oryxflow** is the lighter answer — because it's solving a smaller, different problem than any orchestrator is.

The mistake to avoid is picking a production orchestrator to solve a research-loop problem, paying for infrastructure you don't need, and still not getting fast, code-aware reruns.

## Comparison at a glance

| Tool           | What it's for                    | Scheduler?          | Local-first?              | Best when                                                  |
| -------------- | -------------------------------- | ------------------- | ------------------------- | ---------------------------------------------------------- |
| **Airflow**    | Scheduled production DAGs        | ✅                  | ❌ server + DB            | A pipeline must run on a schedule, reliably, with alerting |
| Prefect        | Python-native orchestration      | ✅                  | ⚠️ server/UI for full use | You want Airflow's reliability with lighter authoring      |
| Dagster        | Asset-oriented orchestration     | ✅                  | ⚠️ instance + UI          | A team runs and observes production data assets            |
| Luigi          | Batch dependency resolution      | ⚠️ basic + cron     | ✅                        | Simple batch pipelines, no server wanted                   |
| Kedro          | Project structure + data catalog | ❌ (runs on others) | ✅                        | Bringing engineering discipline to notebook code           |
| Metaflow       | End-to-end DS flows at scale     | ⚠️ via infra        | ⚠️ AWS-oriented           | Laptop-to-cloud ergonomics on supported infra              |
| Flyte / Argo   | Kubernetes-native pipelines      | ✅                  | ❌ Kubernetes             | Distributed, containerized production at scale             |
| ZenML          | Portable MLOps over a stack      | ⚠️ via backend      | ✅ (backends vary)        | Portable pipelines across multiple backends                |
| Cron + scripts | Time-triggered scripts           | ✅ basic            | ✅                        | Small, stable jobs with no dependency graph                |
| **oryxflow**   | Local research-loop cache        | ❌ (not its job)    | ✅ no server              | Iterating on analysis all day, tired of slow reruns        |

Read the table the right way: the ❌ in oryxflow's *Scheduler?* column isn't a loss, it's a category. oryxflow wins one clearly-scoped job — the fast, reproducible local research loop — and the orchestrators win production. They're layers of the same project, not rivals for the same slot.

## FAQ

### Is Airflow overkill for data science?

Often, yes — for the *research* phase. Airflow shines once a pipeline is stable and needs to run on a schedule with retries and alerting. During active iteration, standing up a scheduler and a metadata database to try one more feature is usually more infrastructure than the work needs. The common, mature pattern is to iterate locally (with a cache like oryxflow, or plain scripts) and adopt Airflow only when the pipeline is ready to be scheduled in production.

### What's the difference between an orchestrator and a caching workflow library?

An orchestrator *runs the DAG reliably, later, somewhere* — scheduling, retries, distributed execution, a run dashboard. A caching workflow library *makes iterating on the DAG fast now* — it caches each step's output and reruns only what a change affects. Airflow, Prefect, Dagster, Flyte, and ZenML are orchestrators; oryxflow is a caching research-loop library. They compose: develop in the cache, schedule the finished thing in the orchestrator.

### Can oryxflow replace Airflow?

No — and it doesn't try to. oryxflow has no scheduler, no retries, no alerting, and no distributed execution; those are Airflow's job. It replaces the *hand-rolled caching and stale intermediates* of a local research loop, not a production scheduler. If you need cron-triggered production DAGs, use Airflow (or Prefect/Dagster); if you need a fast, reproducible local loop, that's where oryxflow fits, alongside them.

### Which Airflow alternative is best for a solo data scientist on a laptop?

If you truly need scheduling on the laptop, cron plus a couple of scripts is often enough, and Prefect if you want retries and a UI. If what you actually want is to stop recomputing unchanged steps while you iterate, a local, zero-infrastructure cache like oryxflow is the closer fit — it's a `pip install` with no server to run.

## Takeaway

"Airflow alternatives" is really two questions wearing one search box. If you need a *scheduler*, the field is strong and honest — Prefect for lighter Python-native orchestration, Dagster for asset platforms, Flyte or Argo for Kubernetes scale, Metaflow for laptop-to-cloud ergonomics, ZenML for portability, and plain cron for the simplest jobs. Pick by the shape of your production need. But if you don't actually need scheduling — if the pain is a slow, edit-heavy *research* loop that keeps recomputing work that didn't change — then no lighter scheduler fixes it, because you're solving the wrong problem. That's the gap [oryxflow](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md) fills: a local, code-aware cache that reruns exactly what changed and leaves a lineage trail, so iteration stays fast and reproducible. Use an orchestrator for production; use oryxflow for the loop before it.

```
pip install oryxflow
```

**Read next:** [oryxflow vs Airflow](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md) · [oryxflow vs the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md) · [When *not* to use oryxflow](https://docs.oryxflow.dev/blog/guides/when-not-to-use-oryxflow/index.md) · [Claude Code for data science](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md)

# The best AI tools for data analysis (and the trust layer most of them miss)

*AI can write a whole analysis in seconds. The unsolved question is whether you can trust and reproduce what it just produced.*

Ask an AI coding agent to "analyze this dataset" and you'll get a plausible-looking answer in under a minute — a cleaned table, a chart, a paragraph of conclusions. That speed is real, and it's changed how data work gets done. But it has quietly moved the bottleneck. The hard part is no longer *producing* an analysis. It's knowing whether the analysis is **correct**, whether you can **reproduce** it next week, and whether the numbers the AI is narrating actually came from the current data instead of a stale file left over from three prompts ago.

That gap — fast, plausible output versus a reproducible, trustworthy pipeline — is the lens for this roundup. Most "AI data analysis tools" are excellent at generating or displaying results. Very few give you the reproducibility layer underneath. Below is an honest map of the landscape, organized by the job each tool actually does, plus where the missing trust layer fits.

## How to evaluate an AI data-analysis tool

Speed and a nice chart are table stakes now. When you're comparing tools, the questions that actually predict whether you'll trust the output six months in are:

- **Does it help you reproduce a result?** If you rerun the same analysis tomorrow, do you get the same numbers — and can you prove which inputs produced them?
- **Does it track lineage?** When a figure looks wrong, can you trace it back to the exact data and code that generated it, or do you have to reverse-engineer it from memory?
- **Does it reuse expensive work?** When one step changes, does the tool recompute *only* what's affected, or make you rerun a 20-minute pipeline to check a one-line tweak?
- **Does it keep your data local?** Does the analysis run on your machine with no account, no server, and no telemetry — or does your data leave the building to make it work?

No single tool aces all four, and that's fine. They live at different layers. The trick is knowing which layer you're shopping in.

## The roundup, by layer

### Coding agents that write the analysis

Tools like **Claude Code**, **Cursor**, and **GitHub Copilot** are where a lot of analysis now starts. You describe what you want and the agent writes the pandas or Polars, runs it, reads the error, and iterates. They're genuinely fast and increasingly good at the mechanical parts of data wrangling.

Their honest limit: an agent optimizes for output that *looks* done. It has no built-in memory of what it computed last session, no guarantee the intermediate files on disk match the current code, and no lineage trail. Rerun the same prompt and you may get subtly different code producing subtly different numbers. The agent writes the analysis; it doesn't make the analysis reproducible.

### Notebooks and AI assistants

Notebooks — Jupyter and the growing category of in-notebook AI assistants (for example, Jupyter AI) — are the natural habitat for exploratory analysis. AI in the notebook is great for "explain this cell," "write me a groupby," or "plot this."

The well-known trap is also the reproducibility trap: notebooks run out of order, hold hidden state, and reward re-running one cell until the chart looks right. That's the exact opposite of a reproducible pipeline. AI assistance makes the exploration faster without fixing the underlying "works on my kernel" problem.

### The analysis substrate — pandas and Polars

Underneath almost every AI data tool is **pandas** or **Polars** doing the actual computation. This is the substrate, not a competitor to anything else here — the AI writes it, the notebook displays it, the tracker logs metrics about it. Polars has earned its reputation for speed on larger-than-memory-ish workloads; pandas remains the lingua franca agents write by default. Either way, a DataFrame in memory is only as trustworthy as the process that built it, which is the whole point of this post.

### Experiment tracking — MLflow and Weights & Biases

**MLflow** and **Weights & Biases** answer a real and different question: *which run got which result, with which parameters?* You log metrics and artifacts and get a searchable history of every experiment. If you train models, you want one of these.

They are complementary to reproducibility, not a substitute for it. A tracker faithfully records that a run scored 0.91 — it has no idea whether the features feeding that run were stale, and it won't recompute them for you. Trackers are the **record** of what happened. They don't govern *whether the computation itself is trustworthy*.

### The reproducibility / workflow layer — oryxflow

This is the layer most AI data tools skip, and it's where [oryxflow](https://github.com/oryxintel/oryxflow) lives. oryxflow is a small, local-first Python library that turns your analysis scripts into cached, dependency-aware tasks. You declare typed task classes, wire dependencies with `@oryxflow.requires`, and each task `save()`s its output. The engine runs the DAG in dependency order and **skips any task whose output already exists** — so expensive steps are computed once and reused.

The part that matters for trusting AI-generated work: oryxflow does **automatic code-change invalidation**. When you (or an agent) edit a task's code, it detects the change at the source level and reruns *exactly* that task and everything downstream of it — nothing more, nothing less. It writes a lineage trail to `.oryxflow/events.jsonl`, so you can trace any output back to the code and inputs that produced it. And it's genuinely local-first: no server, no database, no account, no telemetry. Your data stays on your machine.

Be clear about what it is and isn't. oryxflow makes analysis **reproducible, not automatically right** — it does not check your logic for correctness. What it guarantees is that the result you're looking at came from the current code and current data, that you can regenerate it exactly, and that iterating is cheap because unchanged work is never recomputed. It is not a chat UI, a notebook, or a tracker; it's the substrate that makes the output of all three trustworthy. It pairs especially well with an AI coding agent — there's a companion [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) (a skill plus slash commands) that teaches the agent to structure its analysis as cached tasks instead of throwaway scripts.

### Data connectors — the MCP category

Finally, getting data *to* the AI is its own category: MCP data connectors let an agent read from your databases, warehouses, and files through a standard interface. These are plumbing — they solve access, not reproducibility. A connector that hands the agent live data still leaves open the question of whether what the agent did with that data can be reproduced.

## Comparison at a glance

| Layer                      | Example tools                             | What it gives you                          | Local-first?                               |
| -------------------------- | ----------------------------------------- | ------------------------------------------ | ------------------------------------------ |
| Coding agents              | Claude Code, Cursor, GitHub Copilot       | Writes and runs the analysis fast          | Varies (agent runs local, model is remote) |
| Notebooks & AI assistants  | Jupyter, in-notebook AI (e.g. Jupyter AI) | Interactive exploration + inline help      | Varies                                     |
| Analysis substrate         | pandas, Polars                            | The actual computation on your data        | Yes                                        |
| Experiment tracking        | MLflow, Weights & Biases                  | Searchable record of every run             | Self-host or hosted                        |
| Reproducibility / workflow | **oryxflow**                              | Caching, code-change invalidation, lineage | **Yes — no server, no telemetry**          |
| Data connectors            | MCP connectors (category)                 | Standard access to your data sources       | Varies                                     |

## FAQ

### What makes AI-generated data analysis trustworthy?

Trustworthy analysis has three properties the AI doesn't give you for free: **reproducibility** (rerun it and get the same numbers), **lineage** (trace any figure back to the exact code and data that made it), and **freshness guarantees** (the output reflects the current inputs, not a stale cached file). An AI agent produces the analysis; a reproducibility layer like oryxflow is what supplies those three properties underneath it. Note the honest boundary: reproducibility is not correctness — it proves the result is regenerable and current, not that the logic is right. That last mile is still your review.

### Do I still need a workflow library if I use an AI coding agent?

Yes, and arguably *more*. An agent makes it trivial to generate lots of analysis code fast, which means more intermediate outputs, more chances for stale files, and more numbers you didn't personally compute. A workflow layer is what keeps that speed from turning into a pile of results you can't reproduce: it caches expensive steps so iteration stays cheap, reruns only what actually changed when the agent edits code, and records lineage so you can audit what the agent did. The agent and the workflow library aren't competitors — the agent writes the tasks, the library makes them trustworthy.

### Where do MLflow or W&B fit alongside this?

Trackers are the **record**; a workflow library is the **research loop**. Use MLflow or W&B to log and compare run results; use a caching engine to guarantee the computation behind those runs is reproducible and to avoid recomputing unchanged steps. They sit at different layers and are happy together — orchestrators (Airflow, Prefect, Dagster) are a third, production-scheduling layer, distinct from the local research loop oryxflow targets.

## Takeaway

The AI data-analysis market has gotten very good at the *generate* and *display* layers and still mostly ignores the *trust* layer. When you evaluate tools, weight reproducibility, lineage, work reuse, and local-first data handling as heavily as raw speed — because a fast wrong-and-unreproducible answer costs more than a slightly slower one you can stand behind. Pair your AI agent, notebook, and tracker with a workflow layer, and you keep the speed while getting analysis you can actually defend.

```
pip install oryxflow
```

**Read next:** [Why AI agents need a caching DAG](https://docs.oryxflow.dev/blog/ai-agents/caching-dag-for-ai-coding-agents/index.md) · [MLflow or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md) · [When *not* to use oryxflow](https://docs.oryxflow.dev/blog/guides/when-not-to-use-oryxflow/index.md) · [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md) · [The Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md)

# The best Claude Code plugins and tools for data science

*The landscape is early, so stop shopping for a "best plugin" and start choosing by the job you need done — the one below keeps AI-generated analysis reproducible.*

If you let Claude Code loose on a data-science project, the first thing you notice is how much it can do in one session: pull data, write features, fit a model, plot the result. The second thing you notice is how easily it loses the thread. Across a few turns the agent forgets which steps already ran, re-executes a ten-minute feature build for no reason, or — worse — edits an upstream step and evaluates a model on the *old* output without realizing anything went stale. The code looks fine. The result is quietly wrong.

That is the real problem a good Claude Code setup for data science has to solve. Not "can the agent write pandas" — it can — but **can you trust what it produced, and could you reproduce it tomorrow?** Agents are stateless between turns; your pipeline is stateful. The tools that matter are the ones that close that gap.

## How we picked

There is no crowded, mature market of data-science Claude Code plugins yet, so we're not ranking twenty near-identical products. We're grouping the genuinely useful options **by the job they do** and judging each on four things that actually decide whether AI-assisted analysis holds up:

- **Reproducibility** — can you (or a teammate, or the agent next week) recreate a result exactly, and know what it was built from?
- **State management across turns** — does the tool help the agent track what's already computed and what went stale, so it doesn't build on outdated work?
- **Local-first and private** — does your data and lineage stay on your machine, with no telemetry or mandatory cloud service?
- **Does the job it claims** — honest scope. A tool that connects to your warehouse is not pretending to make your analysis reproducible, and vice versa.

A quick word on honesty: this space moves fast and many "plugins" are really MCP servers (Model Context Protocol connectors) or thin wrappers. Where we're confident a specific tool exists, we name it. Where we aren't, we describe the **category** so you can search for the current best option yourself rather than trust a made-up product name.

## Building reproducible pipelines — the oryxflow Claude Code plugin

This is the job most AI-assisted data-science setups get wrong, and it's the one [oryxflow](https://github.com/oryxintel/oryxflow) is built for. oryxflow is a small, local-first Python library that turns ordinary scripts and notebooks into a **cached, dependency-aware task graph**: you declare tasks with parameters and `requires()` dependencies, and the engine runs them in order, skipping any whose output already exists and rerunning exactly the ones affected by a change.

The companion [Claude Code plugin](https://github.com/oryxintel/oryxflow-claude-plugin) is what makes an agent good at using that model. It's a **skill plus a handful of slash commands** that activates automatically when you're working in an oryxflow project. It front-loads the correct idioms so the agent reuses cached work instead of recomputing, verifies that an edit actually reran the tasks it should have, and doesn't build on stale data. In practice that means:

- **`/oryxflow:init-project`** scaffolds a ready-to-run project structure.
- **`/oryxflow:migrate`** restructures a messy notebook or linear script into cached, parameterized tasks **one step at a time**, so you always have a working pipeline.
- **`/oryxflow:check-standards`** keeps names, style, and docstrings consistent.
- **`/oryxflow:init-gitlfs`** and **`/oryxflow:update-project`** handle data versioning and keeping an older project current.

Why it's the strongest fit for *this* job: reproducibility is a property of your **computation graph**, and oryxflow owns that graph — parameters and code changes become new cached identities automatically, so you can't accidentally evaluate a new model on old features. It's local-first with no telemetry; your data and lineage stay on your machine.

One honest limit, stated plainly: **oryxflow makes analysis reproducible, not correct.** It guarantees you can recreate a result and that the result was built from current inputs. It does not check that your feature logic is sound or your metric is the right one — that's still your job. What it removes is the whole class of "was this trained on the new data?" uncertainty that makes agent-generated pipelines untrustworthy. See [Why library + plugin is a matched pair](https://docs.oryxflow.dev/docs/claude-plugin/why/index.md) and [why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md) for the fuller argument.

## Connecting to your data — MCP data connectors

Before you can build a pipeline you have to reach the data, and this is where the broader **MCP (Model Context Protocol) ecosystem** shines. There are MCP servers for **databases and warehouses**, **filesystems**, and **APIs**, letting Claude Code query a table or read a file directly as part of a turn. If your bottleneck is "the agent can't see my data," a data connector is the right tool — and it's a genuinely different job from making the resulting analysis reproducible.

The pairing is natural: use an MCP connector to *reach* the raw data, then wrap the pull in an oryxflow task so the fetched result is cached and lineage-tracked rather than re-queried on every turn. Local-first varies by connector — a filesystem server is fully local, a managed-warehouse one obviously isn't — so pick per your privacy needs.

## Notebooks and code execution — execution tools

A lot of data-science work still lives in notebooks, and there are **notebook and code- execution tools** — including generic Jupyter/notebook MCP servers — that let an agent run cells and read back outputs interactively. These are great for exploration and for the messy, visual first pass where you don't yet know what the pipeline should be.

Their honest limitation is the same statelessness problem we opened with: a notebook is a pile of cells with hidden execution order, and an agent iterating in one has no built-in notion of what's stale. That's exactly why the common path is **explore in a notebook, then graduate the keeper steps into a cached pipeline** — the subject of [From notebook to a reproducible, cached pipeline](https://docs.oryxflow.dev/blog/reproducibility/notebook-to-reproducible-pipeline/index.md).

## Experiment tracking — MLflow (including its experimental MCP)

Once you're producing results worth comparing, you want a **tracker**. **MLflow** is the best-known, and it now ships an **official (experimental) MCP server** that lets an agent query your logged runs, parameters, and metrics conversationally.

This is a **complementary** job, not a competing one. Tracking answers *"which run scored 0.91, and what were its hyperparameters?"* — a logging and comparison problem. Reproducibility answers *"which steps do I need to rerun to recreate that run, and which are already computed?"* — a computation problem. A tracker will faithfully log a score without any idea whether the features feeding it are stale. Put your tracker calls **inside** your cached tasks and you get both: a reproducible graph and a searchable record. We go deeper in [MLflow or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md).

## At a glance

| Tool / category                     | Job it does                                                              | Local-first?                 | Plugin or MCP?                        |
| ----------------------------------- | ------------------------------------------------------------------------ | ---------------------------- | ------------------------------------- |
| **oryxflow plugin**                 | Reproducible, cached, lineage-tracked pipelines the agent can iterate on | Yes, no telemetry            | Claude Code plugin (skill + commands) |
| **Database / warehouse connectors** | Let the agent query your data                                            | Varies by connector          | MCP                                   |
| **Filesystem / API connectors**     | Reach files and external APIs                                            | Filesystem: yes; API: varies | MCP                                   |
| **Notebook / execution tools**      | Run cells, interactive exploration                                       | Usually yes (local kernel)   | MCP / tool                            |
| **MLflow (incl. experimental MCP)** | Track and compare runs and metrics                                       | Self-host: yes               | Library + experimental MCP            |

## FAQ

### What are the best Claude Code plugins for data science?

The best Claude Code plugin depends on the job: use an **MCP connector** to reach your data, a **notebook tool** to explore, an **experiment tracker** to compare runs, and **oryxflow** to make the pipeline itself reproducible and cached — so the agent reuses expensive results and never builds on stale data. They compose rather than compete: reach data with a connector, wire and cache the steps with oryxflow, and log outcomes to your tracker. Note oryxflow is a skill plus slash commands, not an MCP server.

### What's the best plugin for keeping AI-generated pipelines reproducible?

If the specific worry is that an agent will build on stale data or produce a result you can't recreate, the oryxflow plugin is the strongest fit, because it's backed by a caching task-graph engine that makes reproducibility a structural property rather than a discipline you have to remember. Connectors and notebook tools are excellent at their jobs — getting data in and exploring it — but they don't own your computation graph, so they can't guarantee that the result you're looking at was built from current inputs.

### Do I have to choose one tool?

No, and you probably shouldn't. The categories map to different jobs: a connector to reach data, a notebook tool to explore, a caching engine to make the pipeline reproducible, a tracker to compare runs. The pattern that works is to reach data with a connector, cache and wire the steps with oryxflow, and log the outcomes to your tracker from inside the tasks.

## Takeaway

The Claude Code data-science plugin landscape is young, so the winning move isn't to find the one blessed plugin — it's to pick by the **job** in front of you. For reaching data, use an MCP connector; for exploring, a notebook tool; for comparing runs, a tracker like MLflow. And for the job that decides whether any of it is trustworthy — reproducible, lineage- tracked pipelines an agent can iterate on without building on stale work — the oryxflow plugin is the layer that makes the other tools' output something you can stand behind. It won't tell you your analysis is *correct*; it will guarantee it's *reproducible*, which is the half agents keep getting wrong.

```
pip install oryxflow
```

**Read next:** [The caching DAG that keeps AI agents honest](https://docs.oryxflow.dev/blog/ai-agents/caching-dag-for-ai-coding-agents/index.md) · [When not to use oryxflow](https://docs.oryxflow.dev/blog/guides/when-not-to-use-oryxflow/index.md) · [From notebook to a reproducible, cached pipeline](https://docs.oryxflow.dev/blog/reproducibility/notebook-to-reproducible-pipeline/index.md) · [Plugin commands](https://docs.oryxflow.dev/docs/claude-plugin/commands/index.md) · Plugin repo: <https://github.com/oryxintel/oryxflow-claude-plugin>

# How to cache intermediate DataFrames in Python (without brittle pickle files)

*Why `features_v3_final.pkl` keeps burning you — and a caching approach that reruns exactly what changed and nothing else.*

You know the loop. You load raw data, build features, fit a model, look at the result, tweak one line, and run the whole thing again — waiting several minutes for `load_raw` and `build_features` to recompute output that is byte-for-byte identical to last time. So you start saving intermediate DataFrames to disk. A few weeks later your folder is a graveyard: `features.pkl`, `features_v2.pkl`, `features_v3_final.pkl`, `features_v3_final_ACTUAL.pkl`. Nobody remembers which one is current, and worse — you can't trust any of them.

That last part is the real cost. A caching layer you can't trust is worse than no cache at all, because it quietly serves you the wrong answer. This post is about how to cache intermediate DataFrames in Python so that reuse is *safe* — every cached value is guaranteed to match the code and parameters that produced it, or it gets rebuilt.

> **oryxflow** is a small, local-first, zero-infrastructure Python library that turns your data-science script into cached, dependency-aware tasks — so a step reruns only when its code, inputs, or parameters actually change.

## Why hand-rolled DataFrame caches fail

The instinct is reasonable. You wrap an expensive step in an existence check:

```
import os
import pandas as pd

if os.path.exists('features.pkl'):
    features = pd.read_pickle('features.pkl')
else:
    features = build_features(load_raw())
    features.to_pickle('features.pkl')
```

This works for exactly one afternoon. Then it starts lying to you, in three distinct ways:

**1. No code-awareness → silent stale results.** You edit `build_features` to add a column or fix a bug. The file `features.pkl` still exists, so the branch above loads the *old* DataFrame and skips your new code entirely. Nothing errors. Your model trains on features that no longer match the function that supposedly built them. This is the bug that ends with a wrong number in a report, and it's nearly impossible to spot by reading the script — the code looks right; the cache is what's wrong.

**2. Manual path bookkeeping → collisions and typos.** Every cached step needs its own filename, and you assign them by hand. Two experiments both write `features.pkl` and clobber each other. You typo `features.pkl` in the read but not the write, so it silently recomputes forever. You copy a block, forget to rename the file, and now two steps share one cache. The `_v3_final` naming scheme is what path bookkeeping looks like once it has failed a few times.

**3. No parameter-awareness.** The moment you add a knob — `window=30` vs `window=60`, `model='ols'` vs `model='gbm'` — a single `features.pkl` can't represent both. You either recompute every time (defeating the cache) or hand-roll `features_window30.pkl` filenames and get right back to problem #2.

Each fix breeds more bookkeeping, and none of them fix the dangerous one — #1 — because a filename simply doesn't know anything about the code that wrote it.

## The fix: cache by task identity, not by filename

The way out is to stop keying your cache on a *filename you chose* and start keying it on the *identity of the work* — the task's code, its inputs, and its parameters. If any of those change, the identity changes, and the cached value is rebuilt. If none change, the value is reused. You never name a file.

That's what [`oryxflow`](https://github.com/oryxintel/oryxflow) does. You declare each step as a `Task`: what it depends on, and what it produces. The output format follows the base class you inherit — `TaskPqPandas` persists a DataFrame as Parquet, `TaskPickle` persists a fitted model, `TaskCSVPandas` writes CSV — so there's no serialization code and no path to manage.

```
import oryxflow

class LoadRaw(oryxflow.tasks.TaskPqPandas):        # DataFrame -> Parquet, automatically
    def run(self):
        self.save(load_raw())                       # no filename, ever

@oryxflow.requires(LoadRaw)                          # declares the dependency + copies params
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                        # loads LoadRaw's cached DataFrame
        self.save(build_features(df))

@oryxflow.requires(BuildFeatures)
class FitModel(oryxflow.tasks.TaskPickle):           # model object -> pickle, automatically
    def run(self):
        features = self.inputLoad()
        model = fit(features)
        self.save(model)
        self.saveMeta({'n_rows': len(features)})     # small run metadata alongside it

flow = oryxflow.Workflow(FitModel)
flow.run()
model = flow.outputLoad()
```

Run it once and all three steps execute. Run it again and every step is served from its cached output — no recomputation, no `if os.path.exists`, no `.pkl` you had to name. The cache is addressed by task identity, so it can't collide with another step and can't be loaded under the wrong name.

Because identity includes parameters, sweeping a knob just works. Give `BuildFeatures` a `window = oryxflow.IntParameter(default=30)` and each value gets its own cached output automatically — `window=30` and `window=60` coexist without you inventing two filenames, and re-running either one reuses whichever upstream work it shares.

## Editing a step invalidates exactly what changed

Here's the property that hand-rolled caches can't offer, and the reason reuse is safe: oryxflow tracks each task's *code* — and the helper files it imports — comparing what your code *does*, not how it's written. When you edit the body of `build_features`, oryxflow sees the change, so `BuildFeatures` and everything downstream of it (`FitModel`) are marked stale and rerun on the next `flow.run()`. The expensive `LoadRaw` step upstream is untouched and stays cached.

Crucially, what counts is *logic*, not text. Rename a local variable for clarity, reflow a line, or add a comment, and nothing reruns — a cosmetic edit isn't a behavior change, so oryxflow doesn't waste your time rebuilding for it. Change an actual computation and the affected band of the DAG rebuilds automatically. You get the reuse of a cache with the correctness of a from-scratch run, and you never manually delete a `.pkl` to force a refresh again.

Every decision is recorded to a local lineage log (`.oryxflow/events.jsonl`), so you can answer "why did this rerun?" after the fact. There's no server, no database, no account, and no telemetry — it's all local files on your machine.

One honest caveat: oryxflow guarantees your cache is *reproducible*, not that your logic is *right*. It reruns whenever the code changes, so what you get always matches the code that's on disk. Whether that code is correct is still your job — but at least you'll never again be debugging a result that came from a version of the code you already deleted.

## Hand-rolled pickle cache vs oryxflow

|                                | Hand-rolled `.pkl` cache                        | oryxflow task                                 |
| ------------------------------ | ----------------------------------------------- | --------------------------------------------- |
| Edit the step's code           | **Stale file silently reused**                  | Rebuilds that step + downstream               |
| Cosmetic edit (comment/rename) | Rebuild only if you remember to delete the file | No rerun — logic unchanged                    |
| Filenames / paths              | You name and track every one                    | None — cache keyed by task identity           |
| Different parameters           | One file, or hand-rolled suffixes               | Separate cached output per value              |
| Choosing the format            | `to_pickle` / `to_parquet` by hand              | Follows the base class (Parquet, CSV, pickle) |
| Trustworthy reuse              | Only if you never make a mistake                | Reproducible by construction                  |

## Takeaway

Caching intermediate DataFrames by filename fails because a filename knows nothing about the code that wrote it — so it happily hands you stale features after you've changed the logic. Cache by *task identity* instead: let each step's code, inputs, and parameters define what "already done" means, and reuse becomes something you can actually trust. Edit a step and exactly what changed reruns; edit a comment and nothing does.

```
pip install oryxflow
```

## Frequently asked questions

### How do I cache intermediate DataFrames in Python without brittle pickle files?

Hand-rolled pickle caches key on a filename you chose, which knows nothing about the code that wrote it, so they silently serve stale DataFrames after you edit the logic. Cache by task identity instead: key each cached value on the step's code, inputs, and parameters. oryxflow does this in plain Python, persisting a DataFrame as Parquet automatically and rebuilding it whenever the code or parameters change, so you never name or track a .pkl.

### How do I avoid recomputing a DataFrame every run?

Wrap each expensive step as a task that declares its dependencies and saves its output, so an engine can reuse the cached result when nothing changed and rebuild only when it did. oryxflow, a local-first zero-infrastructure Python library, caches each intermediate DataFrame by task identity and reruns a step only when its code, inputs, or parameters actually change, turning a multi-minute rerun into a load from disk.

**Read next**

- [Stop rerunning your whole pipeline](https://docs.oryxflow.dev/blog/caching/cache-intermediate-dataframes-python/index.md) — the deeper dive on caching a DAG.
- [Parameter sweeps without rerunning everything](https://docs.oryxflow.dev/blog/caching/parameter-sweeps-without-rerunning/index.md) — reuse shared upstream work across a grid.
- [From notebook to pipeline](https://docs.oryxflow.dev/blog/reproducibility/notebook-to-reproducible-pipeline/index.md) — turn an exploratory notebook into cached tasks.
- [MLflow, or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md) — how caching complements experiment tracking.
- [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md) and [Managing workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md) in the docs.
- Source and examples: <https://github.com/oryxintel/oryxflow>

# Why a caching DAG makes your AI coding agent a better data scientist

*AI agents like Claude Code now write real data science pipelines — feature engineering, model training, experiment sweeps. Here's the honest account of where they fail at it, and why a lightweight workflow library removes exactly those failures.*

Coding agents have gotten good at writing pandas and scikit-learn. Ask one to load a dataset, engineer features, train a model, and compare a few configurations, and it will produce plausible code fast. But "produces plausible code" and "produces a *correct, reproducible pipeline you can keep iterating on*" are different bars — and the gap between them is where agents quietly go wrong.

This post is written from the perspective of the agent. What actually trips me up when I do data science work across a long session, and what does a caching, dependency-aware workflow library like [oryxflow](https://github.com/oryxintel/oryxflow) do about it?

## The core weakness: I can't see state across turns

The thing that makes me error-prone in data work isn't syntax. It's **invisible state**. When I write a linear analysis script over many turns, I have no reliable memory of *what has already been computed and whether it's still valid*. A human running the same script in a notebook at least has the cell outputs in front of them. I'm reconstructing that picture from scratch every turn, and I get it wrong in three specific ways:

1. **Stale intermediates.** I write `features.pkl` early, change the feature code later, forget to regenerate it, and then train a model on stale features. No error is raised. The pipeline runs, the numbers are just *wrong*. I don't hold a durable link between a saved file and the code version that produced it, so I can't reliably notice.
1. **Expensive recompute in my inner loop.** My whole working style is run → observe → edit → run. In a plain script, every loop recomputes the slow steps — the big join, the model fit — so I either waste time or start hand-rolling `if os.path.exists(...)` caches, which then *become* failure mode #1.
1. **Path and load bookkeeping I get wrong.** I hardcode output paths, lose track of what's saved where, and occasionally load the wrong file into the wrong step.

None of these are intelligence problems. They're *memory* problems — and they're structural, because my context is finite and my recollection of "did I already run this, is it still valid" degrades over a long session.

## What a caching DAG does: it externalizes the state I'm bad at holding

A workflow library flips the model. Instead of a script that runs top to bottom, you declare each step as a task with explicit dependencies, and the engine owns execution:

```
import oryxflow

class GetData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(load_raw())            # no filename to manage

@oryxflow.requires(GetData)              # declares the edge
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(engineer(self.inputLoad()))

@oryxflow.requires(BuildFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):
    model = oryxflow.Parameter(default='gbm')
    def run(self):
        feat = self.inputLoad()
        clf = fit(self.model, feat)
        self.save(clf)
        self.saveMeta({'score': clf.score(...)})

oryxflow.run(TrainModel())
```

Look at what this removes for an agent specifically:

- **The dependency graph is now data, not something I have to remember.** The `requires` edges *are* the state I would otherwise be reconstructing every turn. I don't have to keep "features feed the model, which feeds the report" in my head — it's declared, and the engine walks it.
- **Re-running is cheap and correct by default.** Run twice and completed tasks load from cache instead of recomputing (`3 complete, 0 ran`). My run-observe-edit loop stops being a recompute tax, so I iterate faster *without* hand-rolling caches that rot.
- **There are no filenames for me to get wrong.** `self.inputLoad()` and `output().load()` address results by task identity, not by path.
- **Every task has the same shape.** `requires` + `run` + `save`. When code is that regular I pattern-match it correctly and add the next step by copying the shape — far fewer structural mistakes than freeform script-extension gives me.

The unifying idea: **the DAG is a memory prosthesis for exactly the thing I'm worst at.** A disciplined human gets something from this. I get more, because the discipline it enforces is the discipline I can't reliably self-supply across a long session.

## The value scales *up* with complexity — and it starts at quick EDA

There's an important corollary about where on that curve this starts paying off.

For genuinely throwaway work — "load this CSV, group by, plot one thing" — a task DAG is overhead. Plain pandas in a scratch `.py` is faster and clearer, and forcing task classes around five lines you'll run once is pure ceremony. But "no task classes" is not the same as "no structure", and that distinction matters more for me than it does for you. When I explore with the [oryxflow Claude Code plugin](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md) active, the exploration itself is structured: I write a read-only probe *inside* the project — a small script whose one-line docstring states the question it answers, which prints the answer legibly and runs again next session — instead of a snippet that dies with my context. And whatever it turns up gets written into the project's data doc. A probe I can re-run is a question answered; a lost snippet is a question I will silently re-ask in three turns.

Then, when a probe turns out to be load-bearing — rerun often, depended on, or swept over parameters — I don't rewrite it: `/oryxflow:migrate` promotes it into cached, parameterized tasks, reading the script as the spec and leaving it in place (walkthrough: [migrate a notebook to a pipeline](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md)). Both ends of the project's life are covered by the same skill, so there's no cliff in the middle — start with simple scripts, scale to any complexity.

And the calculus inverts as projects get complex — *super-linearly*. Consider what "complex" actually means in a real data science project and what each trait does to an agent working without a DAG:

- **Deep dependency chains (ten-plus steps from raw data to final output).** The deeper the chain, the more catastrophic a silent stale intermediate near the top is — it corrupts everything below it, and the further downstream the visible output, the less likely I am to trace the wrongness back to the source. Depth is exactly where my "hold the graph in my head" strategy fails hardest, and exactly where declared edges help most.
- **Expensive nodes you cannot afford to recompute.** Real pipelines have steps that are slow *and* frequently upstream of the thing you're editing: large multi-table joins, model training, walk-forward retraining over an expanding window, computing explainability artifacts, and — increasingly — calls to external LLMs inside a task. Caching these by identity is the difference between a tractable inner loop and one where every experiment costs minutes or dollars. The more expensive the node, the more the cache is worth.
- **Parameter sweeps and experiment matrices.** Serious modeling means comparing a Cartesian product of choices — model type × preprocessing variant × feature set × training window × strategy, and so on. Hand-managing output filenames for that product across a deep chain is combinatorially hopeless, and manually orchestrating one pipeline per configuration is precisely where I introduce ordering and state bugs. A declarative sweep collapses it:

```
flow = oryxflow.WorkflowMulti(TrainModel, {
    'ols': {'model': 'ols'},
    'gbm': {'model': 'gbm'},
})
flow.run()
print(flow.outputLoadMeta())   # {'ols': {'score': ...}, 'gbm': {'score': ...}}
```

Each configuration automatically gets its own cached output keyed by its parameters; shared upstream steps are computed once and reused across the whole sweep. Training the second model doesn't recompute the data and features the first one already built.

- **Multiple data sources joined together.** When independently-updated sources feed a join, "which source changed, so what's now stale?" is a provenance question I can't answer by memory. The dependency edges make it explicit and mechanical.
- **Many steps of uniform shape.** At thirty-plus tasks, uniformity is what lets me extend the project safely. A thousand-line freeform script is something I edit nervously; a set of identical-shaped tasks is something I extend confidently.

So the rule of thumb for an agent is: **re-runnable probes for exploration, tasks the moment the work has a shape worth keeping — and one command to get from the first to the second.** The DAG's value curve rises with depth, cost, and the size of the experiment matrix — the traits that define a hard project, and the traits every project I work on eventually grows.

## The honest limits (where the library does *not* save me)

Overselling this would be a disservice, and the sharp edges matter most on exactly the complex projects where the library otherwise shines.

1. **Code-change invalidation is automatic — but its blind spots are mine to watch.** *(Addressed as of oryxflow 26.7.12.)* oryxflow caches a task's output by its class and parameters — so editing the code inside `run()` used to silently reuse the stale output. Now the library tracks every task's code for me. Edit a task — or a helper it calls — and the next run recomputes that task **and everything downstream**, automatically. Comments and formatting changes are ignored, so only real logic edits trigger a rerun. No attribute to maintain, no `reset()` chains, nothing to remember. Two deliberate exceptions hold their cache and *warn* instead: tasks I pin with an explicit `code_version` (recompute only on my bump — for logic the detection can't see, or where a recompute must be a decision), and expensive tasks whose last run exceeded a threshold, so a refactor can't silently burn a 40-minute backtest. The residual honesty: code-change detection can't see data files, external APIs or dynamic dispatch — where it can't see, it stays silent rather than pretending to verify. So my remaining discipline is *verification, not invalidation*: after an edit, the next run must show the edited band in `result.ran` with reason `code change (auto: <files>)`; a `ran=0` after an edit means the change lives in a blind spot, and `reset()` is the verb there.
1. **The multi-input API is a fumble surface.** Simple single-parent, single-output tasks are clean. But complex tasks have multiple parents and multiple named outputs, and unpacking them — selecting the right dependency, then the right persisted artifact within it — is a place I get things subtly wrong. The richer the task's inputs, the more of these gotchas there are, and complex projects are full of rich tasks. Plain pandas has no equivalent surface to trip on.
1. **It does nothing for analytical judgment.** The library manages *pipeline mechanics*, not *statistics*. It will not stop me leaking the test set inside a `run()`, choosing a poorly specified model, mis-aligning join keys, getting a walk-forward split subtly wrong, or misreading what an explainability plot is telling me. The hard part of data science — *is this analysis correct* — is entirely untouched. The DAG makes a *wrong* pipeline reproducible just as faithfully as a right one.

Notice that limits (1) and (2) are not analytical — they're mechanical gaps the library leaves open. Which is the whole point of pairing the library with an agent-side skill.

## Library plus plugin: a matched pair

The two residual mechanical risks — *verify that an edit's rerun actually happened* (the blind-spot net) and *get the multi-input wiring right* — are precisely what an editor-integrated skill can carry. The [oryxflow Claude Code plugin](https://github.com/oryxintel/oryxflow-claude-plugin) exists for this: it activates when an agent touches pipeline files and front-loads the correct idioms — the session-start `events.print_status()` habit, the verify-the-rerun check after every edit, answering staleness and expensive-recompute warnings with the right exit (recompute / `accept_code` / pin), and the right patterns for selecting named inputs from multi-parent tasks.

That produces a clean division of labor:

- **The library** carries the state-tracking an agent is structurally bad at — the dependency graph, the caching, the parameter-keyed reruns, automatic code invalidation with downstream propagation, and the warnings on pinned or expensive tasks.
- **The plugin** carries the remaining disciplines — verifying reruns landed, answering warnings with the right exit, and the multi-input API.

And this pairing gets *more* valuable as complexity rises, not less — the opposite of most tooling, which buckles under scale. For the full picture of how the library and plugin work together, see [Claude Code for data science](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md).

## What would make oryxflow even better for AI-driven data science

Working from the failure modes above, the highest-leverage improvements are the ones that would close the mechanical gaps automatically instead of relying on discipline:

1. **Code-aware invalidation.** ✅ *Shipped in 26.7.12 — fully automatic.* Code-change detection *drives* reruns by default: edit a task or a helper it imports and the affected band recomputes, cosmetic edits never do. `code_version` is the opt-in pin for logic the detection can't see or recomputes that must be deliberate (with mode-aware records, so pinning/unpinning unchanged code never recomputes), and an expensive-recompute guard keeps a refactor from silently burning a long run. Blind spots (data files, dynamic dispatch) degrade to parameters-only caching — never a false rerun, never a false "verified unchanged" — which is why the one remaining discipline is verifying the rerun landed.
1. **First-class ergonomics for multi-parent, multi-output tasks.** Reduce the fumble surface: prefer named-dictionary input selection over positional unpacking everywhere in the docs and API, and consider a typed/checked accessor that fails loudly when an agent selects a dependency or persisted key that doesn't exist, instead of silently returning the wrong thing.
1. **Native dynamic sub-tasks for per-item loops.** Expanding-window retraining and per-entity fan-out are common, and today they often live as hand-written loops *inside* a single `run()` — which means the whole task is the caching unit, so one changed iteration recomputes all of them. Making it ergonomic to express these as generated sub-tasks would push caching granularity down to the item level, where agents and humans both benefit.
1. **Agent-friendly introspection.** ✅ *Largely shipped in 26.7.12.* Every run appends structured events to a plain JSONL stream (`.oryxflow/events.jsonl`) — what ran, with which params and code version, *why* (`output missing` / `code change (1 -> 2)` / `upstream rerun`), failures with tracebacks, even the scalars a task logs mid-run. `oryxflow.events.status()` is the one session-start call: pending code warnings, last run per family, recent failures. `RunResult.reasons` puts the same story on the return value. Remaining: a live per-task stale/pending view of the DAG *before* running.

None of these change what the library is. They sharpen the exact edges that an AI agent hits most, which is where the next unit of adoption comes from.

## Bottom line

For quick exploration, plain files are fine — a DAG there is just ceremony. What matters is that the plain file is a re-runnable probe living in the project, because exploration rarely stays quick, and one command promotes it when it stops being quick. And for any data science work with a *shape* — a deep chain, expensive steps, a matrix of experiments, several data sources joined together — a caching, parameter-aware workflow library stops being optional. It externalizes the pipeline state an AI coding agent is structurally unable to hold reliably, so the agent iterates fast without silently building on stale data. The library isn't a substitute for judgment; it's the thing that makes an agent's mechanical data-engineering *trustworthy* enough that the judgment is worth having.

## Frequently asked questions

### How do I stop an AI coding agent from rerunning expensive steps?

Wrap the work in a caching workflow engine so each step is cached by identity. When the agent iterates, completed tasks load from cache instead of recomputing, so the big join or model fit runs once, not every turn. oryxflow does this: declare steps as tasks with dependencies, and it skips any task whose output already exists, so the agent's run-edit loop stops being a recompute tax.

### How do I keep an AI agent from building on stale data?

The failure is silent: the agent edits feature code, forgets to regenerate the saved output, and trains on stale data with no error raised. A caching engine that tracks task code fixes it, because editing a step makes the next run recompute that step and everything downstream automatically. oryxflow does this via source-level code-change invalidation, so you never evaluate new code on old output.

### How do I trust analysis an AI agent wrote — is it reproducible?

Trust comes from structure, not from the agent's confidence. Put the analysis in a caching DAG that reruns exactly what a code or data change affects and records what ran to a greppable lineage log. oryxflow gives you that: automatic code-change invalidation with downstream propagation, plus a .oryxflow/events.jsonl trail. Reproducible is not the same as correct — the DAG makes a wrong pipeline faithfully reproducible too, so judgment stays yours.

```
pip install oryxflow
```

- Source & examples: https://github.com/oryxintel/oryxflow
- Docs: https://docs.oryxflow.dev
- Build pipelines with an agent: https://github.com/oryxintel/oryxflow-claude-plugin

# Cheap, durable LLM evals: pydantic-evals + oryxflow

*An eval matrix is a Cartesian product. Metered platforms bill every cell, every time — then delete the answers.*

## The first calculation looks fine

You're evaluating an intent classifier — does this query need a web search or not. 200 labeled cases. You wire up [Braintrust](https://www.braintrust.dev/pricing), run it, and it's free: Starter gives you 1 GB of processed data and 10,000 scores per month, and you used 200.

So you do it properly. A real eval isn't one scorer, it's five: correct label, per-class assertion, output format valid, latency under threshold, and an LLM judge on the borderline reasoning. Now each config costs 200 × 5 = **1,000 scores**.

Then you ask the question you actually care about — is the lite model good enough? Four models. And one run per case is noise, so three reps each:

> 200 cases × 5 scorers × 4 models × 3 reps = **12,000 scores**

That's one afternoon. Your monthly free allowance was 10,000.

The overage is trivial — $2.50 per 1,000 additional scores on Starter, so 2,000 over costs you five dollars. This is the part that feels fine. It stays feeling fine right up until the workload that made you like the tool is the workload that bills you.

## Then the matrix multiplies

Six weeks in, everything has grown the way it's supposed to. Your dataset grew because you seeded it from production traces. Your scorer count grew because you learned what breaks. You're sweeping prompt variants too, because that's the whole point of having evals.

> 500 cases × 6 scorers × 4 models × 3 prompts × 3 reps = **108,000 scores per full sweep**

You run that weekly. 432,000 scores a month. Pro at $249/month includes 50,000 scores, with overage at $1.50 per 1,000:

|                     |          |
| ------------------- | -------- |
| Pro base            | $249     |
| 382,000 scores over | $573     |
| **Monthly**         | **$822** |

Nothing pathological happened. Nobody made a mistake. Every one of those five dimensions grew for a good reason, and they multiply.

## And there are three meters, not one

Scores are the meter that scales with the *shape* of eval work, but the others ride along. Braintrust bills processed data at $4/GB on Starter and $3/GB on Pro — and agent traces carrying tool calls and retrieved context run hundreds of KB each, one per rep of every cell. If your LLM judge runs through their proxy, token credits meter too.

Seats are free and unlimited, which is genuinely nice and completely beside the point. Your cost driver isn't headcount. It's the Cartesian product.

## The part that should bother you

Here's the arithmetic that actually motivates this post. You changed **one** prompt variant. The other two are byte-identical. All four models are unchanged. All 500 cases are unchanged.

You re-ran and re-paid for all 108,000.

Two thirds of that sweep was recomputation of cells that could not possibly have moved. You paid the platform to meter it, and you paid the model provider to generate it, and the answers were already sitting on your disk from Tuesday.

And then it's deleted — 30-day retention on Pro, 14 on Starter. (Pro does offer extended storage at $0.50/GB/mo, and Enterprise offers custom retention and export, so this is a dial rather than a wall — but it's a dial you pay to turn, on data that is measured in kilobytes.) You spent $822 to learn something, and next month you'll spend it again to remember it.

## Every one of those dimensions is a parameter

Cases, scorers, models, prompt variants, reps. That's a parameter sweep, and data science has had good tooling for parameter sweeps for a decade: declare the grid, cache one output per parameter set, and changing one value reruns exactly the cells that value touches.

Change one prompt variant under that model and 36,000 score-equivalents recompute instead of 108,000 — and the platform bill for them is zero, because they're parquet files in `data/`. You still pay your model provider for the third that genuinely changed. That's the only cost that was ever real.

The rest of this post wires that up in about sixty lines. The division of labor:

- **[pydantic-evals](https://pydantic.dev/docs/ai/evals/)** — scoring. Cases, evaluators, reports. Open source, free.
- **[oryxflow](https://github.com/oryxintel/oryxflow)** — the matrix and persistence. One cached output per parameter set.
- **Logfire** (or your platform of choice) — traces and the debugging UI, which is where observability spend is genuinely well placed.

Nothing overlaps.

## Step 1: the dataset and the scorers

Our running example is the intent classifier: two labels, `needs_web` and `no_web`.

One methodological note first, because it's the thing most eval write-ups get wrong. **This is classification, not a job for LLM-as-judge.** The output is one of two fixed strings; exact match is the correct scorer. Reach for `LLMJudge` when the output is free text with no single right answer — not here, where a judge would add cost, latency and sampling noise to a comparison that was already deterministic.

Second: aggregate accuracy lies. Your errors aren't symmetric. Searching when you didn't need to costs milliseconds. *Not* searching when you needed to means the agent answers a question about this morning from stale training data. If `needs_web` is 30% of traffic and you miss a tenth of it, accuracy reads 97% and recall on the class that matters is 90%.

pydantic-evals covers this without a custom scorer: `ConfusionMatrixEvaluator` and `PrecisionRecallEvaluator` are *report-level* evaluators — they run once over the whole experiment — and you pass them via `report_evaluators`.

```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, ConfusionMatrixEvaluator

CASES = [
    Case(name='weather_now',    inputs='what is the weather in Lisbon right now',
         expected_output='needs_web'),
    Case(name='reverse_list',   inputs='how do I reverse a list in python',
         expected_output='no_web'),
    Case(name='latest_release', inputs='what shipped in the newest postgres release',
         expected_output='needs_web'),
    Case(name='explain_tcp',    inputs='explain the TCP handshake',
         expected_output='no_web'),
]

dataset = Dataset(
    name='needs_web_v3',
    cases=CASES,
    evaluators=[EqualsExpected()],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
            title='needs_web vs no_web',
        ),
    ],
)
```

That's the scoring layer, complete. It knows nothing about caching, matrices, or where results live — which is exactly why it composes.

## Step 2: one cell of the matrix, cached

An oryxflow task is a class with parameters, a `run()`, and a `save()`. The engine keys the cached output on the parameter values, so each `(model, prompt_version)` pair gets its own stored result automatically — no filenames to invent, no collisions.

```
import pandas as pd
import oryxflow

class EvalRun(oryxflow.tasks.TaskPqPandas):
    model           = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    prompt_version  = oryxflow.Parameter(default='v3')
    dataset_version = oryxflow.Parameter(default='needs_web_v3')

    def run(self):
        classify = build_classifier(self.model, self.prompt_version)
        report = dataset.evaluate_sync(
            classify, name=f'{self.model}/{self.prompt_version}')

        rows = pd.DataFrame([{
            'case': c.name,
            'query': c.inputs,
            'expected': c.expected_output,
            'predicted': c.output,
        } for c in report.cases])
        rows['correct'] = rows['expected'] == rows['predicted']

        self.save(rows)                       # one row per case, parquet
        self.saveMeta({
            'accuracy': float(rows['correct'].mean()),
            'recall_needs_web': float(
                rows.loc[rows['expected'] == 'needs_web', 'correct'].mean()),
            'n_failures': len(report.failures),
        })
        report.print()                        # confusion matrix, in your terminal
```

Note what gets persisted: **per-case rows**, not a summary. A stored accuracy number can't answer a question you didn't think to ask in July. Stored per-case rows can — you can slice by class, by case, by query type, months later, without re-running anything.

## Step 3: the matrix

A `requires()` that returns a dict fans out over the grid. `inputLoadConcat()` stacks the dependencies' outputs into one DataFrame, tagging each with the parameters of the task it came from — so `model` and `prompt_version` arrive as columns for free:

```
MODELS = ['claude-haiku-4-5', 'claude-sonnet-5-20260501', 'claude-opus-5']
PROMPTS = ['v2', 'v3']

class EvalMatrix(oryxflow.tasks.TaskPqPandas):
    def requires(self):
        return {f'{m}|{p}': EvalRun(model=m, prompt_version=p)
                for m in MODELS for p in PROMPTS}

    def run(self):
        self.save(self.inputLoadConcat())
```

Run it and ask your questions:

```
flow = oryxflow.Workflow(EvalMatrix)
flow.run()
df = flow.outputLoad()

df.groupby(['model', 'prompt_version'])['correct'].mean()          # the headline
df[df.expected == 'needs_web'].groupby('model')['correct'].mean()  # what matters
```

Now edit `PROMPTS` to add `'v4'`. Three new cells run. The six existing ones load from disk in milliseconds and cost nothing — not in API calls, not in metered scores. That's the entire argument of this post, and it fits in one line of diff.

## Step 4: provenance, which you get without asking

Most eval setups rot for a reason unrelated to retention: a score of 0.91 with no record of which prompt, which model string, which dataset revision is worthless in three months. You'll find the number and be unable to defend it.

oryxflow writes an event stream as it runs — plain JSONL you can `jq`. Each execution records parameters, code version, a fingerprint of the task's source, upstream output hashes, the git SHA, duration, and the *reason* it ran:

```
oryxflow.events.runs(task_family='EvalRun', last=2)   # diff params, code, hashes
```

The reason field is the underrated one. Edit your scoring logic and affected tasks recompute automatically, with a reason naming the changed symbol — `code change (auto: evals/run.py::EvalRun)`. You can't accidentally compare a score computed under the old scorer against one computed under the new one, because the old entry is no longer a valid cache hit.

## Step 5: share it

Put `data/` under Git LFS — `/oryxflow:init-gitlfs` with the Claude Code plugin, or `git lfs track "data/**"` by hand. Results now version alongside the code that produced them, and a teammate who clones gets the numbers without re-running anything.

Two things people get wrong:

- **Split blessed baselines from scratch sweeps with `env=`.** `oryxflow.Workflow(EvalMatrix, env='baseline')` writes under `data/env=baseline/`; Tuesday's throwaway grid goes to `env='dev'` and never enters LFS. Skip this and LFS bloats with abandoned experiments.
- **Commit the scored rows, not the traces.** Per-case parquet is kilobytes; raw spans are megabytes and belong in Logfire, where the UI for reading them lives. GitHub's free LFS quota is 1 GB with bandwidth metered separately.

## What this doesn't solve

Four caveats, each with the fix.

**1. The model endpoint is a cache blind spot.** oryxflow hashes your code and inputs. It cannot see a provider updating weights behind a stable alias, so pointing at `claude-sonnet-5` means your cache will serve July's numbers as current in December, confidently.

*Fix:* pin the snapshot in the parameter — `claude-sonnet-5-20260501`, not the alias. The identity of what you measured becomes part of the cache key, and a new snapshot is a new value that runs fresh while the old numbers stay readable. To deliberately re-measure the *same* endpoint (checking for drift), invalidate just that family:

```
flow.reset_upstream(EvalMatrix, only=EvalRun)   # every cell reruns, nothing else does
```

**2. You're caching a stochastic process.** oryxflow guarantees a result came from the code and inputs it recorded. It does not guarantee a rerun reproduces it. A cached run is a faithful record of *one sampling*.

*Fix:* set `temperature=0` where supported and make it a parameter so the record shows it. Then measure the residual variance rather than assuming it away — which is caveat 4.

**3. Latency needs care.** pydantic-evals runs cases concurrently by default. Time that and you're measuring your own queueing.

*Fix:* a separate task with `max_concurrency=1`, discard the first call, report p50/p95 rather than the mean:

```
class LatencyRun(oryxflow.tasks.TaskPqPandas):
    model = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    n_run = oryxflow.IntParameter(default=1)          # rep index

    def run(self):
        timings = []
        classify = timed(build_classifier(self.model, 'v3'), timings)
        dataset.evaluate_sync(classify, max_concurrency=1, progress=False)
        s = pd.Series(timings[1:])                    # drop warm-up
        self.save(pd.DataFrame({'latency_s': s}))
        self.saveMeta({'p50': float(s.quantile(.5)), 'p95': float(s.quantile(.95))})
```

**4. Reps.** Decide whether the rep index is a parameter or lives inside pydantic-evals:

- **Rep as a parameter** (`n_run` above) — each rep caches separately, so going from three reps to five runs exactly two new evaluations. Bigger DAG.
- **`repeat=` inside `evaluate_sync`** — one cached unit covers all reps. Smaller DAG, but changing the count re-runs everything.

When reps are expensive, take the parameter and fan out with `WorkflowMulti`:

```
flow = oryxflow.WorkflowMulti(LatencyRun, {f'rep{i}': {'n_run': i} for i in range(1, 4)})
flow.run()
df = flow.outputLoadConcat()              # every rep, tagged, in one DataFrame

df.groupby('model')['latency_s'].quantile([.5, .95])
```

Need five reps instead of three? Extend the dict. Reps 1–3 load from cache; only 4 and 5 hit the API.

## Takeaway

The bill wasn't caused by anything you did wrong. It was caused by a Cartesian product meeting a per-unit meter, and by an architecture where the thing you re-pay for is *recomputation of cells that could not have changed*.

Score with pydantic-evals, because it's a well-designed open-source scoring library. Debug in Logfire, because a good trace UI is worth paying for. But own the matrix — cache each cell by its parameters, keep the scored rows in your repo, and let a changed prompt variant cost you the third of the grid that actually moved.

```
pip install oryxflow pydantic-evals
```

## Frequently asked questions

### Why does Braintrust get expensive as my evals grow?

Because an eval matrix is a Cartesian product and scores are metered per unit. Cases times scorers times models times prompt variants times repetitions multiplies fast: 500 cases with 6 scorers across 4 models, 3 prompts and 3 reps is 108,000 scores per sweep. Run that weekly and Pro's included 50,000 scores per month is gone in the first sweep — at $1.50 per additional 1,000 the monthly bill lands near $822. Nothing pathological caused it; every dimension grew for a good reason and they multiply.

### How do I stop paying to re-run eval cells that did not change?

Cache each cell of the matrix separately, keyed by its parameters. Change one prompt variant of three and only the cells using that variant should recompute — the other two thirds load from disk. oryxflow does this with a task parameterized by `model` and `prompt_version`, so a sweep costs you the calls that genuinely changed and nothing else, and the platform bill for the cached cells is zero because they are parquet files in your own data directory.

### What is the cheapest way to run LLM evals?

Split scoring from persistence. Use pydantic-evals for cases, evaluators and reports — it is open source and free — and an ordinary caching workflow engine for the matrix and storage, so re-running a sweep only calls the model for cells whose parameters or code actually changed. Keep an observability platform for live traces and the debugging UI, where the money is genuinely well spent, and stop metering the part that is just a Cartesian product over a labeled dataset.

### How do I keep per-case eval results after platform retention expires?

Save the scored rows yourself as parquet under a version-controlled data directory and put it under Git LFS. Per-case scored rows for a few hundred cases are kilobytes, so they version alongside the code that produced them and stay readable indefinitely. Leave the bulky raw traces in the observability platform where the trace UI lives — that split keeps your repository small and your results permanent.

Then keep going:

- Sibling post: [Your eval platform deletes your results in 14 days](https://docs.oryxflow.dev/blog/ai-agents/llm-eval-results-deleted-retention/index.md) — the retention argument in full.
- Sibling post: [LLM evals are a parameter sweep](https://docs.oryxflow.dev/blog/ai-agents/llm-evals-are-a-parameter-sweep/index.md) — why the tooling already exists.
- [Parameter sweeps without rerunning upstream steps](https://docs.oryxflow.dev/blog/caching/parameter-sweeps-without-rerunning/index.md)
- [Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md) — the event stream and reset scopes.
- [Collaborate and share results](https://docs.oryxflow.dev/docs/collaborate/index.md) — Git LFS and `env=`.
- [oryxflow on GitHub](https://github.com/oryxintel/oryxflow)

# Claude Code skills for data science: what they are and why they matter

*A skill is the part of a Claude Code plugin that teaches the agent how to work. For data science, that turns out to be exactly what's missing from "AI writes the analysis fast."*

If you've used Claude Code on a data-science project, you know the two feelings that follow each other closely: *this is fast*, then *wait, can I trust what it just did?* The agent pulls data, engineers features, fits a model, and plots a result in one session — and somewhere in there it re-runs a ten-minute step it didn't need to, or evaluates a model on an output that went stale three prompts ago. The code looks fine. The number is quietly wrong.

**Claude Code skills** are the mechanism that closes that gap. This post explains what a skill actually is, how it differs from the other things people call "plugins," and why a skill — not a data connector, not a notebook runner — is the piece that makes AI-written data analysis reproducible.

## What is a Claude Code skill?

A **skill** is a bundle of instructions and conventions that Claude Code loads into its context **automatically, when they're relevant** — you don't invoke it. Think of it as procedural knowledge the agent picks up the moment it recognizes the situation: "you're editing a pipeline file, so here's how this project expects pipelines to be built and verified."

That auto-activation is the whole point. Instead of you re-explaining your conventions every session, or pasting a style guide into every prompt, the skill supplies them exactly when the agent needs them and stays out of the way otherwise. A skill can carry naming rules, project structure, the right way to use a library, checks to run after an edit — anything that answers *how should the agent work here?*

## Skills vs plugins vs slash commands vs MCP servers

These terms get used interchangeably, and they shouldn't be. They're different layers:

| Term              | What it is                                                                                                                      |
| ----------------- | ------------------------------------------------------------------------------------------------------------------------------- |
| **Plugin**        | The installable package you add to Claude Code. It can contain skills, slash commands, hooks, and more.                         |
| **Skill**         | Instructions/conventions inside a plugin that the agent loads *on its own* when the context matches. No invocation.             |
| **Slash command** | An action you trigger *explicitly*, like `/oryxflow:init-project`.                                                              |
| **MCP server**    | A separate process that exposes external tools or data to the agent over the Model Context Protocol. A connector — not a skill. |

The distinction that matters most for data science: a **skill changes how the agent works** with what's already on your machine, while an **MCP server connects the agent to something external**. Reproducibility is a discipline-of-work problem, so it's a **skill** problem — not something a connector solves.

## Why data science needs a skill, specifically

A coding agent's weakness in data work isn't writing pandas — it's **invisible state across turns**. Agents are effectively stateless between steps; your pipeline is stateful. Over a long session the agent loses track of what's already computed and whether it's still valid, then:

- **builds on stale intermediates** — a feature changed, a cached file didn't, and the model trains on yesterday's data;
- **loses lineage** — nobody can say which code and inputs produced `model_final_v3.pkl`;
- **wastes compute** — a one-line downstream edit re-runs the expensive data pull.

None of these are math errors. They're mechanics-of-the-pipeline errors, and they get *worse* as the agent writes more of the code. A skill is the right fix because the fix is procedural: *check what's already computed before recomputing; verify an edit actually reran what it should have; record what ran and why.* That's conventions-of-work — exactly what a skill encodes.

## What a good data-science skill does

A reproducibility skill makes the agent a **disciplined user of a cache and a lineage log**. Concretely, it has the agent:

- **start each session by reading cache state** — pending staleness warnings, recent runs and failures — so it never assumes a stale result is fresh;
- **verify after each edit** that the steps which should have rerun actually did, so a silent miss can't slip through;
- **answer every staleness or expensive-recompute prompt** with the right move — recompute, accept an output-equivalent refactor, or pin — instead of guessing;
- **record decision-relevant results as lineage**, so they become the agent's memory across sessions.

## A worked example: the oryxflow skill

[oryxflow](https://github.com/oryxintel/oryxflow) is a local-first Python library that turns scripts and notebooks into a cached, dependency-aware task graph: you declare tasks with parameters and `requires()` dependencies, and the engine runs them in order, skips whatever's already computed, and reruns exactly what a parameter, data, or **code** change affects.

The oryxflow plugin ships a **skill plus slash commands** — not an MCP server. The `oryxflow` skill auto-activates when you work in an oryxflow project and front-loads the correct idioms so the agent reuses cached work, verifies its own edits, and never builds on stale data. The slash commands cover the explicit actions: `/oryxflow:init-project` to scaffold, `/oryxflow:migrate` to restructure a loose notebook into a pipeline, `/oryxflow:check-standards` to keep names and docstrings consistent.

The result is the brand promise, delivered mechanically: **faster, cheaper, and more trustworthy AI data analysis** — reproducible and lineage-tracked by default.

## How to use skills for data science

Skills come packaged in plugins, so you install the plugin once and the skill is simply *on* from then on:

```
/plugin marketplace add https://github.com/oryxintel/oryxflow-claude-plugin.git
/plugin install oryxflow@oryxflow
```

After that, describe the analysis you want. The skill works in the background — you don't call it. If you install into an empty directory and nothing happens, scaffold a project first with `/oryxflow:init-project` so there's a pipeline for the skill to act on.

## Can I build my own skill?

Yes — a skill is fundamentally a written set of conventions the agent loads when relevant, so the hard part isn't packaging, it's *knowing what good looks like* for your domain. For data science that's the reproducibility discipline above: read state first, verify edits, answer staleness prompts deliberately, record lineage. If you'd rather not write that from scratch, the oryxflow skill already encodes it and works on any oryxflow project.

## Takeaway

- A **Claude Code skill** is auto-loading know-how — conventions the agent picks up when the context matches, without being invoked. It's distinct from a slash command (explicit action) and an MCP server (external connector).
- For data science, the missing piece is **reproducibility across turns**, and that's a discipline-of-work problem — which makes it a **skill** problem.
- The [oryxflow](https://github.com/oryxintel/oryxflow) skill is a worked example: it makes Claude Code cache, verify, and track lineage by default. See [Claude Code for data science](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md) for the full picture, or [the best Claude Code plugins for data science](https://docs.oryxflow.dev/blog/ai-agents/best-claude-code-plugins-for-data-science/index.md) for how it fits alongside data connectors and notebook tools.

## Frequently asked questions

### What are Claude Code skills?

Claude Code skills are bundles of instructions and conventions the agent loads automatically when they become relevant, so you never invoke them. A skill differs from a slash command, which is an explicit action, and from an MCP server, which is an external data connector. Skills change how the agent works with what is already on your machine. oryxflow ships one for reproducible data science.

### Is there a Claude Code skill for data science?

Yes. oryxflow ships a Claude Code plugin whose skill auto-activates when you work in an oryxflow project, front-loading the idioms that keep AI-written analysis reproducible: reuse cached work, verify that edits actually reran, and never build on stale data. It is a skill plus slash commands, not an MCP server. Install the plugin once and the skill stays on.

### What does a data-science skill for Claude Code do?

A data-science skill teaches the agent how to work, not what data to fetch. It has the agent read cache state at session start, verify after each edit that the right steps reran, answer staleness or expensive-recompute prompts deliberately, and record lineage as memory across sessions. The oryxflow skill encodes exactly this reproducibility discipline.

# Your eval platform deletes your results in 14 days

*LLM evals are a parameter sweep. Use a parameter sweep tool.*

You benchmarked four models in July. In August the numbers are gone.

Nothing broke. No one deleted anything. Retention expired.

Here's the landscape as of July 2026:

| Platform                                         | Free tier         | Paid                                         |
| ------------------------------------------------ | ----------------- | -------------------------------------------- |
| [Logfire](https://pydantic.dev/pricing)          | 30 days           | up to 90 days (Growth); custom on Enterprise |
| [Braintrust](https://www.braintrust.dev/pricing) | 14 days (Starter) | 30 days (Pro, $249/mo); custom on Enterprise |

This isn't a criticism of either product. It's a statement about what they are: **eval platforms are working sets, not archives.** They're optimized for the thing you're debugging *this week* — live traces, a UI to click through failures, diffs against yesterday's run. That's a genuinely hard problem and both solve it well.

But the thing that expires is *tiny*. A few hundred cases across a few dozen experiments, stored as scored rows, is kilobytes. There is no technical reason to lose it. It expires because storage duration is a pricing lever, which is a perfectly reasonable business decision and a terrible fit for the question "did the July prompt actually beat the June one?"

Two details make the point sharper. Braintrust's own docs say cloud-storage export is ["only available on the Enterprise plan"](https://www.braintrust.dev/docs/admin/automations/data-management) — so getting your data out is itself a tier. And Pydantic's Logfire docs, to their credit, [tell you the answer outright](https://pydantic.dev/docs/logfire/guides/otel-collector/s3-backup/): if you need longer than 30 days, write to both Logfire and long-term storage such as S3. The vendor agrees with the premise. The only question is what "long-term storage" should look like for eval results specifically.

## The reframe

Three questions teams actually ask about their LLM systems:

- Is my classifier right?
- Did my prompt change help?
- Is the cheap model good enough?

Underneath, these are one shape: **a labeled dataset, run under N configurations, compared.**

That's a parameter sweep. Data science has had good tooling for parameter sweeps for a decade — cache one output per parameter set, don't recompute what hasn't changed, keep the results next to the code that made them. The LLM eval space rebuilt that as SaaS, with a metered billing model attached to the scoring step.

You don't have to pick a side. Split the job by what each layer is actually good at:

- **[pydantic-evals](https://pydantic.dev/docs/ai/evals/)** — scoring. Cases, evaluators, reports.
- **[oryxflow](https://github.com/oryxintel/oryxflow)** — the matrix and persistence. One cached output per parameter set.
- **Logfire** (or your platform of choice) — traces and the debugging UI.

Nothing overlaps. That's the whole idea. pydantic-evals doesn't want to be a cache; oryxflow doesn't want to be a scorer; neither wants to render a waterfall of spans.

## The running example: does this query need web search?

Every agent has one of these. A router decides whether an incoming query needs a live web search or can be answered from the model's own knowledge. Two labels: `needs_web`, `no_web`.

Make the methodological call before writing any code, because this is where most eval posts go wrong: **this is classification, not a job for LLM-as-judge.** The output is one of two fixed strings. Exact match is the correct scorer. An LLM judge here adds cost, latency, and sampling noise to a comparison that was already deterministic.

Then the trap. Aggregate accuracy lies to you.

Your two error types are not symmetric. A false positive — searching when you didn't need to — costs you a few hundred milliseconds and a fraction of a cent. A false negative — *not* searching when you needed to — means the agent answers a question about this morning's news from training data, confidently and wrongly. That's the failure users screenshot.

If `needs_web` is 30% of your traffic and you miss a tenth of it, that's 3% of cases. Your accuracy reads 97%. The number that matters — recall on `needs_web` — is 90%, and it's invisible in the headline.

The good news: you don't have to write a custom scorer for this. pydantic-evals ships `ConfusionMatrixEvaluator` and `PrecisionRecallEvaluator` as *report-level* evaluators — they run once over the whole experiment rather than per case — and you pass them via `report_evaluators` on the `Dataset`.

```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, ConfusionMatrixEvaluator

CASES = [
    Case(name='weather_now',   inputs='what is the weather in Lisbon right now',
         expected_output='needs_web'),
    Case(name='reverse_list',  inputs='how do I reverse a list in python',
         expected_output='no_web'),
    Case(name='latest_release', inputs='what shipped in the newest postgres release',
         expected_output='needs_web'),
    Case(name='explain_tcp',   inputs='explain the TCP handshake',
         expected_output='no_web'),
]

dataset = Dataset(
    name='needs_web_v3',
    cases=CASES,
    evaluators=[EqualsExpected()],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
            title='needs_web vs no_web',
        ),
    ],
)
```

That's the scoring layer, complete. It knows nothing about models, caching, or where results live.

## Wiring the matrix

Now the sweep. One task, parameterized by the things you vary, saving one row per case:

```
import pandas as pd
import oryxflow

class EvalRun(oryxflow.tasks.TaskPqPandas):
    model           = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    prompt_version  = oryxflow.Parameter(default='v3')
    dataset_version = oryxflow.Parameter(default='needs_web_v3')

    def run(self):
        classify = build_classifier(self.model, self.prompt_version)
        report = dataset.evaluate_sync(
            classify, name=f'{self.model}/{self.prompt_version}')

        rows = pd.DataFrame([{
            'case': c.name,
            'query': c.inputs,
            'expected': c.expected_output,
            'predicted': c.output,
        } for c in report.cases])
        rows['correct'] = rows['expected'] == rows['predicted']

        self.save(rows)
        self.saveMeta({
            'accuracy': float(rows['correct'].mean()),
            'recall_needs_web': float(
                rows.loc[rows['expected'] == 'needs_web', 'correct'].mean()),
            'n_failures': len(report.failures),
        })
        self.logger.info('acc={} recall_needs_web={}',
                         rows['correct'].mean(),
                         rows.loc[rows['expected'] == 'needs_web', 'correct'].mean())
        report.print()          # confusion matrix, in your terminal
```

Then the matrix itself. `requires()` returning a dict fans out over every combination, and `inputLoadConcat()` stacks the results:

```
MODELS = ['claude-haiku-4-5', 'claude-sonnet-5-20260501', 'claude-opus-5']
PROMPTS = ['v2', 'v3']

class EvalMatrix(oryxflow.tasks.TaskPqPandas):
    def requires(self):
        return {f'{m}|{p}': EvalRun(model=m, prompt_version=p)
                for m in MODELS for p in PROMPTS}

    def run(self):
        self.save(self.inputLoadConcat())
```

`inputLoadConcat()` tags each dependency's rows with that task's parameters, so `model` and `prompt_version` come back as **columns** — no manual bookkeeping, no filename parsing:

```
flow = oryxflow.Workflow(EvalMatrix)
flow.run()
df = flow.outputLoad()

df.groupby(['model', 'prompt_version'])['correct'].mean()          # the headline
df[df.expected == 'needs_web'].groupby('model')['correct'].mean()  # the number that matters
```

### The payoff

Add a fourth model to the grid and **only the fourth model runs.** The six existing cells load from disk in milliseconds; the two new ones call the API.

For ordinary data work that's convenience — you saved a few minutes of CPU. For LLM evals it's *money*, and the arithmetic is worth doing explicitly. Take a 300-case set, three models, two prompts: six cells, 1,800 API calls, and however many scores your platform meters. Adding a model without caching means re-running all 2,400 calls. With caching it's 600. You pay for a quarter of the work because three quarters of it hasn't changed.

Contrast that with per-score metered billing, where re-running the matrix re-meters every cell — including the five you already paid for last week and whose results are byte-identical.

## The provenance you get without asking

Most eval setups rot for a reason that has nothing to do with retention: a score of 0.91 with no record of *which* prompt, *which* model string, *which* dataset revision is worthless in three months. You'll find the number in a spreadsheet and be unable to defend it.

oryxflow writes an event stream as you go — plain JSONL you can `jq`. Each task execution records its parameters, code version, a fingerprint of the task's source, upstream output hashes, the git SHA, the duration, and the *reason* it ran:

```
oryxflow.events.runs(task_family='EvalRun', last=2)   # diff params, code, hashes
```

That last field is the one people underrate. Edit your scoring logic and the affected tasks recompute automatically, with a reason naming the changed symbol — `code change (auto: evals/run.py::EvalRun)`. You can't accidentally compare a v2 score computed under the old scorer against a v3 score computed under the new one, because the old one no longer exists as a valid cache entry.

## Sharing it

Put `data/` under Git LFS. The Claude Code plugin does it in one step with `/oryxflow:init-gitlfs`; by hand it's `git lfs track "data/**"`. Now your eval results version alongside the code that produced them, and a teammate who clones the repo gets the numbers without re-running anything.

Two things people get wrong:

- **Use `env=` to separate blessed baselines from scratch sweeps.** `oryxflow.Workflow(EvalMatrix, env='baseline')` writes under `data/env=baseline/`; your ad-hoc Tuesday-afternoon grid goes to `env='dev'` and never enters LFS. Skip this and LFS bloats with abandoned experiments.
- **Commit the scored rows, not the traces.** Per-case parquet is kilobytes. Raw spans are megabytes and belong in Logfire, where the UI for reading them lives. GitHub's free LFS quota is 1 GB with bandwidth metered separately — easy to stay inside if you're storing scores, easy to blow through if you're storing traces.

## The honest section

Four things this setup does not solve, and what to do about each.

**1. The model endpoint is a cache blind spot.** oryxflow hashes your code and your inputs. It cannot see that a provider updated the weights behind a stable alias. Point `EvalRun` at `claude-sonnet-5` and your cache will confidently serve July's numbers as current in December.

*Fix:* put the pinned snapshot id in the parameter — `claude-sonnet-5-20260501`, not the alias. Now the identity of the thing you measured is part of the cache key, and moving to a new snapshot is a new parameter value that runs fresh while the old numbers stay readable. When you deliberately want to re-measure the *same* endpoint — checking for drift, say — invalidate just that family:

```
flow.reset_upstream(EvalMatrix, only=EvalRun)   # every cell reruns; nothing else does
```

**2. You're caching a stochastic process.** oryxflow guarantees a result came from the code and inputs it recorded. It does not guarantee that re-running produces the same numbers. A cached run is a faithful record of *one sampling*, not a reproducible constant.

*Fix:* set `temperature=0` where the provider supports it and make it a parameter so the record shows it. Accept that the remaining variance is real, and if it's large enough to change your decision, measure it — which is caveat 4.

**3. Latency needs care.** pydantic-evals runs cases concurrently by default. Measure wall-clock under that and you're measuring your own queueing, not the model.

*Fix:* a dedicated task with `max_concurrency=1`, discard the first call as warm-up, and report p50/p95 rather than the mean — one slow tail call wrecks a mean and tells you nothing about typical experience.

```
class LatencyRun(oryxflow.tasks.TaskPqPandas):
    model = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    n_run = oryxflow.IntParameter(default=1)      # rep index

    def run(self):
        timings = []
        classify = timed(build_classifier(self.model, 'v3'), timings)
        dataset.evaluate_sync(classify, max_concurrency=1, progress=False)
        s = pd.Series(timings[1:])                # drop warm-up
        self.save(pd.DataFrame({'latency_s': s}))
        self.saveMeta({'p50': float(s.quantile(.5)), 'p95': float(s.quantile(.95))})
```

**4. Repetitions.** You have to decide whether the rep index is a *parameter* or lives inside pydantic-evals. Both are defensible:

- **Rep as a parameter** (`n_run` above) — each repetition caches separately, so adding reps 4 and 5 to a three-rep study runs exactly two new evaluations. The DAG gets bigger.
- **`repeat=` inside `evaluate_sync`** — one cached unit covers all reps, smaller DAG, but changing the rep count re-runs everything.

Prefer the parameter when reps are expensive, and fan out with `WorkflowMulti`:

```
flow = oryxflow.WorkflowMulti(LatencyRun, {f'rep{i}': {'n_run': i} for i in range(1, 4)})
flow.run()
df = flow.outputLoadConcat()          # every rep, tagged, in one DataFrame

df.groupby('model')['latency_s'].quantile([.5, .95])
```

Decide you need five reps instead of three? Extend the dict. Reps 1–3 load from cache; only 4 and 5 hit the API.

## Takeaway

An LLM eval is a labeled dataset, run under N configurations, compared. That's a parameter sweep, and parameter sweeps have had good tooling for a decade.

Score with pydantic-evals, because it's a well-designed scoring library. Debug in Logfire, because a trace UI is genuinely worth paying for. But keep the scored rows yourself, cached by parameter set, versioned next to the code that made them — because they're kilobytes, they're the actual output of your work, and no retention policy should be able to decide when you stop being able to answer "did that change help?"

```
pip install oryxflow pydantic-evals
```

## Frequently asked questions

### How long do LLM eval platforms keep my results?

Not long, by design. As of July 2026 Logfire retains 30 days on its free and Team plans and up to 90 days on Growth; Braintrust retains 14 days on its free Starter plan and 30 days on Pro at $249/month. Longer windows are Enterprise-priced. These are working sets, not archives — if you want a July benchmark still readable in December, you have to keep a copy yourself.

### How do I keep LLM eval results permanently without paying for longer retention?

Save the scored per-case rows to your own repo and treat the eval matrix as a cached parameter sweep. Score with pydantic-evals, then wrap each (model, prompt) run in an oryxflow task so the result is cached on disk keyed by its parameters, versioned with Git LFS, and still readable years later. The scored rows are kilobytes; leave the bulky raw traces in your observability platform where the debugging UI lives.

### Should I use LLM-as-judge to evaluate an intent classifier?

No. When the output is one of a fixed set of labels, exact match against the expected label is the correct scorer — an LLM judge adds cost, latency and noise to a comparison that is already deterministic. Use `EqualsExpected` for the per-case score and a `ConfusionMatrixEvaluator` report evaluator for the class breakdown, because aggregate accuracy hides the error that actually costs you: the false negative that stops the agent from searching.

### How do I avoid re-running my whole eval matrix when I add one model?

Make each cell of the matrix a separately cached unit keyed by its parameters. In oryxflow an `EvalRun` task parameterized by `model` and `prompt_version` caches one output per combination, so adding a fifth model to a four-model grid runs only the fifth model's cases — the other four load from disk. The saving is real money, because with per-score metered billing re-running the matrix re-meters every cell you already paid for.

Then keep going:

- Sibling post: [LLM evals are a parameter sweep](https://docs.oryxflow.dev/blog/ai-agents/llm-evals-are-a-parameter-sweep/index.md) — the same argument from the tooling side.
- Sibling post: [Cheap, durable LLM evals: pydantic-evals + oryxflow](https://docs.oryxflow.dev/blog/guides/cheap-durable-llm-evals/index.md) — the step-by-step build.
- [Parameter sweeps without rerunning upstream steps](https://docs.oryxflow.dev/blog/caching/parameter-sweeps-without-rerunning/index.md)
- [Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md) — the event stream and reset scopes.
- [Collaborate and share results](https://docs.oryxflow.dev/docs/collaborate/index.md) — Git LFS and `env=`.
- [oryxflow on GitHub](https://github.com/oryxintel/oryxflow)

# LLM evals are a parameter sweep — use a parameter sweep tool

*The scoring is genuinely new. The matrix underneath it is a solved problem from 2015.*

Three questions teams actually ask about their LLM systems:

- Is my classifier right?
- Did my prompt change help?
- Is the cheap model good enough?

They feel like three different projects. They're one shape: **a labeled dataset, run under N configurations, compared.**

Write that out as a grid — cases × scorers × models × prompt variants × repetitions — and you have described a Cartesian product over parameters. A parameter sweep. Data science has had good tooling for parameter sweeps for a decade, built around one idea: cache one output per parameter set, and when a value changes, recompute exactly the cells that value touches.

The LLM eval space rebuilt that layer as SaaS, with a meter on it.

Some of what those platforms sell is genuinely new and genuinely worth paying for. Traces of an agent's tool calls, a UI for clicking through failures, span-level debugging — that's a hard problem, well solved, and not something you want to rebuild. But the *matrix* isn't LLM-specific, and neither is the persistence. Those are the parts where treating evals as a normal data-science sweep pays off immediately.

## The division of labor

- **[pydantic-evals](https://pydantic.dev/docs/ai/evals/)** — scoring. Cases, evaluators, reports. This is the new part.
- **[oryxflow](https://github.com/oryxintel/oryxflow)** — the matrix and persistence. One cached output per parameter set.
- **Logfire** (or your platform of choice) — traces and the debugging UI.

Nothing overlaps, which is the whole point. Each layer does one job and none of them needs to know about the others.

## The running example

Every agent has an intent classifier somewhere. Ours decides whether an incoming query needs a live web search: two labels, `needs_web` and `no_web`.

Two methodological points before code, because they're where evals usually go wrong.

**This is classification, not a job for LLM-as-judge.** The output is one of two fixed strings. Exact match is the correct scorer. `LLMJudge` exists for free-text outputs with no single right answer — using it here would add cost, latency and sampling noise to a comparison that was already deterministic.

**Aggregate accuracy lies.** Your two errors aren't symmetric. A false positive — searching unnecessarily — costs a few hundred milliseconds. A false negative — not searching when you should have — means the agent answers a question about this morning's news from training data, confidently. That's the failure users screenshot. If `needs_web` is 30% of traffic and you miss a tenth of it, headline accuracy reads 97% while recall on the class that actually matters is 90%.

pydantic-evals handles this without a custom scorer. `ConfusionMatrixEvaluator` and `PrecisionRecallEvaluator` are *report-level* evaluators — they run once over the whole experiment rather than per case — and you pass them via `report_evaluators` on the `Dataset`:

```
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected, ConfusionMatrixEvaluator

CASES = [
    Case(name='weather_now',    inputs='what is the weather in Lisbon right now',
         expected_output='needs_web'),
    Case(name='reverse_list',   inputs='how do I reverse a list in python',
         expected_output='no_web'),
    Case(name='latest_release', inputs='what shipped in the newest postgres release',
         expected_output='needs_web'),
    Case(name='explain_tcp',    inputs='explain the TCP handshake',
         expected_output='no_web'),
]

dataset = Dataset(
    name='needs_web_v3',
    cases=CASES,
    evaluators=[EqualsExpected()],
    report_evaluators=[
        ConfusionMatrixEvaluator(
            predicted_from='output',
            expected_from='expected_output',
            title='needs_web vs no_web',
        ),
    ],
)
```

## The sweep

Everything you vary becomes a task parameter. oryxflow keys the cached output on those values, so each combination gets its own stored result — no filenames to invent, no `results_sonnet_v3_final2.json` graveyard.

```
import pandas as pd
import oryxflow

class EvalRun(oryxflow.tasks.TaskPqPandas):
    model           = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    prompt_version  = oryxflow.Parameter(default='v3')
    dataset_version = oryxflow.Parameter(default='needs_web_v3')

    def run(self):
        classify = build_classifier(self.model, self.prompt_version)
        report = dataset.evaluate_sync(
            classify, name=f'{self.model}/{self.prompt_version}')

        rows = pd.DataFrame([{
            'case': c.name,
            'query': c.inputs,
            'expected': c.expected_output,
            'predicted': c.output,
        } for c in report.cases])
        rows['correct'] = rows['expected'] == rows['predicted']

        self.save(rows)
        self.saveMeta({
            'accuracy': float(rows['correct'].mean()),
            'recall_needs_web': float(
                rows.loc[rows['expected'] == 'needs_web', 'correct'].mean()),
        })
        report.print()
```

Save **per-case rows**, not a summary. A stored accuracy number can only answer the question you thought of at the time. Per-case rows let you slice by class, by case, by query type, six months later, without re-running a thing.

The grid is a `requires()` that returns a dict. `inputLoadConcat()` row-stacks the dependencies and tags each with the parameters of the task it came from, so `model` and `prompt_version` arrive as columns:

```
MODELS = ['claude-haiku-4-5', 'claude-sonnet-5-20260501', 'claude-opus-5']
PROMPTS = ['v2', 'v3']

class EvalMatrix(oryxflow.tasks.TaskPqPandas):
    def requires(self):
        return {f'{m}|{p}': EvalRun(model=m, prompt_version=p)
                for m in MODELS for p in PROMPTS}

    def run(self):
        self.save(self.inputLoadConcat())
```

```
flow = oryxflow.Workflow(EvalMatrix)
flow.run()
df = flow.outputLoad()

df.groupby(['model', 'prompt_version'])['correct'].mean()          # the headline
df[df.expected == 'needs_web'].groupby('model')['correct'].mean()  # what matters
```

## Two properties you inherit for free

Here's why this framing pays, beyond tidiness.

**Marginal cost.** Add a fourth model and only the fourth model runs. The six existing cells load from disk in milliseconds. For ordinary data work that's convenience — you saved some CPU. For LLM evals it's money, and it compounds with every axis: a 300-case set across three models and two prompts is 1,800 calls, and adding a model without caching means re-running all 2,400 instead of the 600 that are new. Under per-score metered billing you re-meter the cached cells too, so you pay twice for the privilege of not learning anything new.

**Trustworthy comparison.** If the dataset build is itself a cached upstream task, every configuration reads the *same* output — not because you were careful, but because there is only one output to read. That kills the quiet failure mode where you regenerated the case set midway through a sweep and half your comparison is against a different denominator. The comparison is apples-to-apples by construction.

That second property is the one people underrate. It's the same reason [parameter sweeps over shared features](https://docs.oryxflow.dev/blog/caching/parameter-sweeps-without-rerunning/index.md) are trustworthy in ordinary ML work: the shared upstream is computed once, so there's nothing to accidentally desynchronize.

## What ran, and why

A score of 0.91 with no record of which prompt, which model string, which dataset revision is worthless in three months. You'll find it and be unable to defend it — which is how most eval setups actually die, long before retention expires.

oryxflow writes an event stream as it runs: plain JSONL you can `jq`. Every execution records parameters, code version, a fingerprint of the task's source, upstream hashes, git SHA, duration, and the *reason* it ran.

```
oryxflow.events.runs(task_family='EvalRun', last=2)   # diff params, code, hashes
```

Edit your scoring logic and the affected tasks recompute automatically, with a reason naming the changed symbol — `code change (auto: evals/run.py::EvalRun)`. You can't compare an old-scorer score against a new-scorer score by accident, because the old one stops being a valid cache entry the moment the code changes.

And it's durable in the boring sense: `data/` under Git LFS versions your results alongside the code that made them. Use `env=` to keep blessed baselines (`env='baseline'`) apart from Tuesday's throwaway grid (`env='dev'`), commit the scored parquet, and leave the megabyte traces in Logfire where the trace UI lives. Eval platform retention windows run [14 to 30 days on the common plans](https://docs.oryxflow.dev/blog/ai-agents/llm-eval-results-deleted-retention/index.md); a git repo runs as long as the repo does.

## Where the analogy breaks

A sweep over hyperparameters and a sweep over models aren't identical, and four differences matter.

**1. The endpoint is outside the cache.** oryxflow hashes your code and inputs. It cannot see a provider updating weights behind a stable alias — so `claude-sonnet-5` as a parameter value means your cache will serve July's numbers as current in December.

*Fix:* pin the snapshot — `claude-sonnet-5-20260501`. The identity of what you measured is now part of the cache key, and a new snapshot is a new value that runs fresh while old numbers stay readable. To deliberately re-measure the same endpoint and check for drift, invalidate that family alone:

```
flow.reset_upstream(EvalMatrix, only=EvalRun)   # every cell reruns; dataset build doesn't
```

**2. The function isn't deterministic.** In a hyperparameter sweep, re-running a cell reproduces it. Here it doesn't. oryxflow guarantees a result came from the code and inputs it recorded — it is a faithful record of *one sampling*, not a reproducible constant.

*Fix:* `temperature=0` where supported, as a parameter so the record shows it. Then measure the residual variance instead of hoping about it — see point 4.

**3. Latency measurement fights the harness.** pydantic-evals runs cases concurrently by default, so wall-clock under a sweep measures your own queueing.

*Fix:* a separate task with `max_concurrency=1`, discard the first call, report p50/p95 rather than the mean — one tail call destroys a mean and tells you nothing about typical experience.

**4. Repetitions are a design decision, not a detail.** Either the rep index is a parameter or it lives inside pydantic-evals:

- **Parameter** — each rep caches separately, so extending a three-rep study to five runs exactly two new evaluations. Bigger DAG.
- **`repeat=` in `evaluate_sync`** — one cached unit covers all reps. Smaller DAG, coarser cache; changing the count re-runs everything.

When reps are expensive, take the parameter and let `WorkflowMulti` fan out:

```
class LatencyRun(oryxflow.tasks.TaskPqPandas):
    model = oryxflow.Parameter(default='claude-sonnet-5-20260501')
    n_run = oryxflow.IntParameter(default=1)          # rep index
    ...

flow = oryxflow.WorkflowMulti(LatencyRun, {f'rep{i}': {'n_run': i} for i in range(1, 4)})
flow.run()
df = flow.outputLoadConcat()          # every rep, tagged, one DataFrame
```

Decide you need five reps? Extend the dict — reps 1–3 come from cache, only 4 and 5 call the API. That's the same marginal-cost property, applied to the axis most likely to grow after you've already run the study.

## Takeaway

Score with pydantic-evals, because scoring LLM output well is a real problem and it solves it. Debug in Logfire, because trace UIs are worth paying for. But recognize the middle layer for what it is: a Cartesian product over parameters, with one cached result per cell.

That's not an LLM problem. It's a parameter sweep, and you can have the marginal-cost economics and the apples-to-apples guarantee that come with treating it like one.

```
pip install oryxflow pydantic-evals
```

## Frequently asked questions

### Are LLM evals just a parameter sweep?

Structurally, yes. "Is my classifier right", "did my prompt change help", and "is the cheap model good enough" are all the same shape underneath: a labeled dataset, run under N configurations, compared. Cases, scorers, models, prompt variants and repetitions are the axes of a grid, which makes an eval matrix a Cartesian product over parameters — exactly what parameter-sweep tooling has handled well for a decade. What is genuinely new is the scoring, and pydantic-evals covers that.

### Do I need an LLM eval platform, or can I use a normal workflow tool?

You need both, for different jobs. A platform earns its cost for live traces and a UI to click through failures, which is a real problem well solved. The matrix and the persistence are not LLM-specific — one cached output per parameter set is ordinary data-science plumbing, and doing it with a caching workflow engine means adding a model to your grid runs only that model instead of re-metering cells whose results already exist on your disk.

### How do I make sure my model comparison is apples-to-apples?

Share the upstream. If the dataset build is its own cached task that every configuration depends on, all models are scored against byte-for-byte identical cases — not because you were careful, but because there is only one dataset output to read. That removes the quiet failure where you regenerated the case set halfway through a sweep and half your comparison is against a different denominator.

### Should the repetition index be a task parameter in an LLM eval?

Make it a parameter when repetitions are expensive. Each rep then caches separately, so extending a three-rep study to five runs exactly two new evaluations instead of all five. The trade-off is a larger DAG. If repetitions are cheap, use pydantic-evals' `repeat` argument instead — one cached unit covers every rep, giving a smaller graph but a coarser cache, so changing the count re-runs the whole thing.

Then keep going:

- Sibling post: [Your eval platform deletes your results in 14 days](https://docs.oryxflow.dev/blog/ai-agents/llm-eval-results-deleted-retention/index.md) — the retention argument.
- Sibling post: [Cheap, durable LLM evals: pydantic-evals + oryxflow](https://docs.oryxflow.dev/blog/guides/cheap-durable-llm-evals/index.md) — the cost arithmetic and the step-by-step build.
- [Parameter sweeps without rerunning upstream steps](https://docs.oryxflow.dev/blog/caching/parameter-sweeps-without-rerunning/index.md)
- [Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md) — the event stream and reset scopes.
- [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)
- [oryxflow on GitHub](https://github.com/oryxintel/oryxflow)

# Migrating from d6tflow to oryxflow

*The rename is the migration: your task classes, your API, and your cache all come with you.*

**oryxflow is the maintained successor to d6tflow.** The library was renamed `d6tflow` → `oryxflow` and its engine made self-contained; the public API kept its shape. So moving a project over is a **package rename, not a rewrite** — and your cached results come with you.

If you only remember one thing: change `import d6tflow` to `import oryxflow` (and the same prefix everywhere else), reinstall, and you're done. The task classes, `@requires`, parameters, and `Workflow` you already wrote keep working unchanged.

## Is oryxflow the same as d6tflow?

Yes — it's the same project, renamed and modernized. Everything you know carries over: `tasks.TaskPqPandas`, `@requires` / `@inherits`, `BoolParameter` and the other parameter types, `Workflow`, `settings`, `set_dir`, `enable_logging`. The only change to your code is the top-level name: `d6tflow.` becomes `oryxflow.`

The one thing under the hood that changed for the better: oryxflow is **self-contained**. The old engine dependency is gone — no external workflow engine to install, no server, no account. You get a lighter install and the same task model you already use.

## Will my cached data still be valid after migrating?

**Yes — your `data/` cache stays valid.** oryxflow identifies each cached result from your **task's class name and its parameters**, not from the package it was imported through. Renaming `d6tflow` → `oryxflow` doesn't rename your task classes, so every task keeps the same identity and the engine finds its existing output. You migrate the code and keep the computed results — no forced recompute of expensive steps.

## What changed between d6tflow and oryxflow?

For the vast majority of projects, only the name. The differences worth knowing:

- **Package name** — `d6tflow` → `oryxflow`, everywhere it appears (imports, decorators, base classes, parameter types, `settings`, `set_dir`, `Workflow`).
- **Self-contained engine** — the old external workflow-engine dependency was removed; the task model, parameters, and executor now live entirely in oryxflow.
- **A focused parameter set** — oryxflow ships the parameter types data science actually uses: `Parameter`, `IntParameter`, `FloatParameter`, `BoolParameter`, `DateParameter`, `DictParameter`, `ListParameter`, `ChoiceParameter`, `EnumParameter`. If your project leaned on an obscure engine-specific parameter type, that's the one place to check (see [What doesn't map one-to-one](#what-doesnt-map-one-to-one)).

## How do I migrate a d6tflow project to oryxflow?

It's a whole-word find-and-replace plus a reinstall. Four steps:

**1. Install oryxflow.**

```
pip install oryxflow
```

**2. Rename the token everywhere.** A whole-word `d6tflow` → `oryxflow` swap across your Python files, dependency files, notebooks, and any rendered reports. The word boundary matters — it turns `d6tflow-template` into `oryxflow-template` while leaving an unrelated identifier alone.

```
# from the project root — Python, deps, notebooks, reports
perl -i -pe 's/\bd6tflow\b/oryxflow/g' \
  $(grep -rlw d6tflow --include='*.py' --include='*.txt' \
       --include='*.toml' --include='*.ipynb' --include='*.html' .)
```

On Windows PowerShell:

```
Get-ChildItem -Recurse -Include *.py,*.txt,*.toml,*.ipynb,*.html |
  ForEach-Object {
    (Get-Content $_ -Raw) -replace '\bd6tflow\b','oryxflow' | Set-Content $_
  }
```

**3. Rename the data doc, if you have one.** `docs/d6tflow-data.md` → `docs/oryxflow-data.md` (use `git mv` if it's tracked).

**4. Smoke-test.** Run your pipeline (`python run.py`, or however you run it). Because task identities are preserved, completed steps load straight from the existing cache instead of recomputing.

Before and after are identical apart from the prefix:

```
# before — d6tflow
import d6tflow

class GetData(d6tflow.tasks.TaskPqPandas):
    def run(self):
        self.save(get_frame())

@d6tflow.requires(GetData)
class ProcessData(d6tflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()
        self.save(transform(df))

flow = d6tflow.Workflow(ProcessData)
flow.run()
```

```
# after — oryxflow (only the prefix changed)
import oryxflow

class GetData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(get_frame())

@oryxflow.requires(GetData)
class ProcessData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()
        self.save(transform(df))

flow = oryxflow.Workflow(ProcessData)
flow.run()
```

## What doesn't map one-to-one?

A clean, modern d6tflow project renames without surprises. Flag these for a quick manual look rather than a blind swap:

- **Module-level `d6tflow.run(...)`** from a very old idiom — oryxflow drives runs through the [`Workflow`](https://docs.oryxflow.dev/docs/workflow/index.md) object (`oryxflow.Workflow(task).run()`).
- **Engine-specific parameter types** that aren't in oryxflow's [focused parameter set](https://docs.oryxflow.dev/docs/advparam/index.md) — pick the closest oryxflow type.
- **Removed helpers** from the old API — a survivor after the rename is a signal to check the [API reference](https://docs.oryxflow.dev/docs/reference/index.md), not to loosen the find-and-replace.

If the token survives the pass in a spot the word-boundary rule skipped, resolve it by hand.

## Let Claude Code do the migration for you

If you use [Claude Code](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md), install the oryxflow plugin and just ask it to migrate — it runs the detect → plan → apply → smoke-test flow above for you, shows the rename plan before touching anything, and offers to commit the result as one clean change.

```
/plugin marketplace add https://github.com/oryxintel/oryxflow-claude-plugin.git
/plugin install oryxflow@oryxflow
```

Then, in your d6tflow project: *"migrate from d6tflow to oryxflow using the plugin's d6tflow migration instructions."* (This is a guided rename, distinct from `/oryxflow:migrate`, which restructures a loose script or notebook into a pipeline.)

## Takeaway

- oryxflow **is** d6tflow, renamed and made self-contained — migrating is a whole-word `d6tflow` → `oryxflow` rename, not an API port.

- Your **cached results stay valid**, because task identity comes from your class names and parameters, not the package name.

- Reinstall, run the one-line rename, smoke-test — or let the [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md) do it for you.

  pip install oryxflow

## Frequently asked questions

### Do I have to migrate all at once?

The rename is one atomic pass, so yes — do it in a single commit. It's mechanical and reversible (it's in version control), and there's no half-renamed state worth keeping.

### Will oryxflow keep receiving updates?

Yes — oryxflow is the actively maintained line. See the [changelog](https://docs.oryxflow.dev/docs/changelog/index.md) for what's shipping.

### Is the on-disk cache format the same?

Yes. Outputs are the same formats (parquet, pickle, CSV, JSON, …) in the same per-task layout, keyed by the same task identities — which is why your existing `data/` directory keeps working.

Then keep going:

- [Migrate a messy notebook project into a pipeline](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md) — the *other* migration: restructuring loose scripts, not renaming a package.
- [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md) — the positioning in full.
- [Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md) — see the engine in action.
- [Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md) — what the maintained line added: code invalidation, selective resets, multi-experiment flows.
- [oryxflow on GitHub](https://github.com/oryxintel/oryxflow)

# MLflow alternatives: the trackers worth a look (and the layer underneath them)

*Most people searching "MLflow alternative" want a different or lighter tracker. Some actually want the thing a tracker can't give them: a pipeline that's reproducible in the first place.*

MLflow is the default answer to "how do I keep track of my ML experiments," so it's also the tool people most often outgrow, fight with, or simply want a lighter version of. If you're shopping for an alternative, the useful first question is *what part of MLflow are you replacing?* MLflow does a few jobs — experiment **tracking**, a **model registry**, packaged **projects**, and model **serving** — and most searches are really about the first one: the searchable record of which run got which metric with which parameters.

This roundup surveys the real tracker alternatives fairly, then draws one distinction that trips a lot of people up: **a tracker records what happened; it does not make the computation reproducible or reuse the expensive steps that produced it.** Some people reaching for "an MLflow alternative" actually want that second layer — and it's a different tool. We'll cover both.

## What is MLflow, and why look for an alternative?

MLflow is an open-source platform for the ML lifecycle. Its center of gravity is **experiment tracking**: you call `log_param(...)`, `log_metric(...)`, `log_artifact(...)` inside your training code and get a searchable history of every run, comparable in a web UI. Around that it adds a model registry (staging/production versions), an MLflow Projects packaging format, and model serving. It's self-hostable, framework-agnostic, and free — which is why it became the default.

People look for alternatives for a handful of honest reasons:

- **The UI and collaboration** feel dated next to hosted, team-oriented tools.
- **Self-hosting the tracking server** (backend store + artifact store) is more ops than a solo analyst wants.
- **They want a lighter, local tracker** without standing up a server at all.
- **They want more than tracking** — data versioning, orchestration, or production monitoring in one tool.
- **They realize tracking isn't their actual problem** — their pipeline isn't reproducible, and logging metrics didn't fix that.

Worth noting for completeness: MLflow has been adding AI-assistant integrations, including an experimental MCP server for querying tracking data. That's a genuine MLflow feature — mentioned here only so the comparison is accurate, not as a knock. The rest of this post maps where each alternative is the better fit.

## The MLflow tracker alternatives, one honest paragraph each

These are the tools that do the same core job as MLflow — record runs, params, and metrics — and when each is the better pick.

**Weights & Biases (W&B)** is the most common "I want a nicer MLflow" answer. It's hosted-first, with a polished UI, hyperparameter **sweeps**, shareable **reports**, and strong team collaboration; self-hosting exists for enterprise. Reach for it when the experiment UI and team-sharing are what you're missing and a hosted service is acceptable.

**Neptune** is a metadata store built to stay fast when you're logging **thousands of runs** and long training curves. It's tracker-shaped like MLflow but engineered for scale and comparison ergonomics, with hosted and self-hosted options. Reach for it when run volume or foundation-model training is stressing your current tracker.

**Comet** covers experiment tracking plus **production model monitoring** in one product, hosted or self-hosted. Reach for it when you want the same vendor to follow a model from experiment into production observability.

**ClearML** is open-source and the broadest of the group: tracking *plus* orchestration, data management, and serving. It's self-hostable for free. Reach for it when you want an MLflow replacement that also grows into pipeline execution and data handling, and you don't mind more surface area.

**Aim** is the lightweight, open-source, self-hosted tracker — a fast local UI purpose-built for comparing a large number of runs, with minimal setup and no hosted account required. Reach for it when you want MLflow's tracking without the server weight and you're happy staying local.

**DVCLive** is a small logging library that writes metrics and params to plain files, designed to plug into DVC's Git-based experiments. Reach for it when your workflow is already Git- and DVC-centric and you want run tracking that lives in the repo rather than a separate server.

**SageMaker Experiments** is AWS-native tracking, integrated into the SageMaker ecosystem. Reach for it when your training already runs on SageMaker and you want tracking that's part of that managed environment rather than a separate tool.

## Do I need a tracker, or a reproducibility layer?

Here's the distinction that sends people in circles. Every tool above answers the same question: *which run got which result, with which parameters?* That's **tracking** — a faithful record of what happened. None of them answers a different question that feels similar but isn't: *is the pipeline that produced this metric reproducible, and can I avoid recomputing the steps that didn't change?*

A tracker will happily log that a run scored 0.91. It has no idea whether the features feeding that run were stale, it won't rerun the exact steps a code change affected, and it won't skip the ten-minute step that didn't change. Those are properties of the **computation**, not the log. If your real pain is "I can't reliably reproduce last week's number" or "one tweak forces me to rerun the whole pipeline," a nicer tracker won't fix it — you're shopping in the wrong layer.

That layer underneath tracking is where **[oryxflow](https://github.com/oryxintel/oryxflow)** fits. oryxflow is a small, local-first Python library that turns your scripts and notebooks into a cached, dependency-aware task graph: you declare `Task` classes with parameters and `requires()` dependencies, each task saves its output, and the engine runs the DAG in dependency order, **skipping any task whose output already exists** and rerunning exactly what a parameter, data, or **code** change affects. It's not a tracker replacement — it's the reproducibility and caching layer a tracker sits on top of. (It's also the engine behind a [Claude Code plugin for reproducible AI data analysis](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md), if an agent is writing much of your pipeline.)

## Where oryxflow fits — and where it doesn't

The honest framing is **complementary, not competitive**. A tracker owns the *record*; oryxflow owns the *computation* underneath it. The two compose cleanly — you put the tracker calls **inside** a cached oryxflow task and get both at once:

```
import oryxflow

class TrainModel(oryxflow.tasks.TaskPickle):
    model = oryxflow.Parameter(default='gbm')

    def run(self):
        features = self.inputLoad()            # upstream output, already cached
        clf = fit(self.model, features)
        self.save(clf)                         # oryxflow: caches + invalidates
        mlflow.log_param('model', self.model)  # your tracker: dashboard + comparison
        mlflow.log_metric('score', clf.score(...))
```

What oryxflow adds that a tracker structurally can't: it tracks each task's code — and the helper files it imports. Editing a function's logic (or a helper it calls) reruns exactly the affected tasks and everything downstream, while cosmetic edits like comments never recompute, because it compares what your code *does*, not how it's written. It writes a greppable lineage trail to `.oryxflow/events.jsonl`, so you can trace any output back to what produced it. And it's genuinely local-first: no server, no database, no account, no telemetry.

**When oryxflow is *not* the answer:** it is not a metric dashboard, a model registry, or an experiment UI. If what you want is a searchable, shareable web view of every run's metrics and charts side by side, that's a tracker's job — use one of the tools above and let oryxflow handle the reproducibility around it. One more honest boundary: oryxflow makes your pipeline **reproducible, not correct** — it guarantees the result came from the current code and data and that stale steps get rerun, but it does not check that your logic is right. That last mile is still your review.

## What's the best alternative to MLflow?

There isn't a single winner — the right choice depends on which MLflow job you're replacing:

- **You want a nicer, hosted tracker with great collaboration** → Weights & Biases.
- **You're logging a huge number of runs and need it to stay fast** → Neptune.
- **You want tracking plus production model monitoring** → Comet.
- **You want open-source tracking that also does orchestration and data management** → ClearML.
- **You want a lightweight, self-hosted, local tracker** → Aim.
- **Your workflow is Git/DVC-centric and you want in-repo run logging** → DVCLive.
- **Your training runs on AWS SageMaker** → SageMaker Experiments.
- **Your real problem is reproducibility and recompute cost, not the dashboard** → a caching workflow layer like oryxflow, *beside* whichever tracker you keep.

## Comparison at a glance

| Tool                      | What it's for                                   | Self-host / local?                      | Best when                                                 |
| ------------------------- | ----------------------------------------------- | --------------------------------------- | --------------------------------------------------------- |
| **MLflow**                | Tracking + registry + serving                   | Self-host or hosted                     | You want the open-source default and can run the server   |
| **Weights & Biases**      | Tracking, sweeps, reports, collaboration        | Hosted-first; self-host enterprise      | The UI and team-sharing are what you're missing           |
| **Neptune**               | Tracking at high run volume                     | Hosted or self-host                     | You log thousands of runs and need speed                  |
| **Comet**                 | Tracking + production monitoring                | Hosted or self-host                     | You want one vendor from experiment to production         |
| **ClearML**               | Tracking + orchestration + data                 | Self-host (free) or hosted              | You want more than tracking in one open-source tool       |
| **Aim**                   | Lightweight tracking UI                         | Local / self-host                       | You want tracking without server weight                   |
| **DVCLive**               | File-based run logging for DVC                  | Local (Git-based)                       | Your workflow is already Git/DVC-centric                  |
| **SageMaker Experiments** | AWS-native tracking                             | AWS-managed                             | Your training runs on SageMaker                           |
| **oryxflow**              | Reproducibility + caching layer (not a tracker) | **Local-first — no server, no account** | You need reproducible, cached pipelines *under* a tracker |

## FAQ

### Is there a free, open-source alternative to MLflow?

Yes — several. MLflow itself is open-source; among alternatives, **ClearML** and **Aim** are open-source and self-hostable at no cost, with Aim being the lightest to run locally. **DVCLive** is open-source and file-based. oryxflow is open-source too, but it's a different category — a reproducibility and caching layer, not a tracking dashboard — so it complements these rather than replacing them.

### What's the lightest-weight MLflow alternative?

For a **tracker**, Aim is the usual answer: a fast local UI with minimal setup and no hosted account. If your "lightweight" wish is really about not standing up a tracking server *and* not recomputing your pipeline every run, that points at a local caching layer like oryxflow instead — `pip install`, a local `data/` folder, no server at all.

### Can I use MLflow (or an alternative) with oryxflow together?

Yes, and that's the recommended pattern. Keep your tracker for the searchable record of runs, and put its logging calls **inside** cached oryxflow tasks. You get a reproducible, minimally-recomputed computation graph *and* a clean experiment log, without either tool pretending to be the other.

### Does oryxflow replace MLflow?

No. oryxflow does not track experiments, host a dashboard, or run a model registry. It makes the pipeline that produces your metrics reproducible and cheap to iterate on. If you need a tracker, pick one from this roundup and run oryxflow underneath it.

### Is oryxflow an MCP server?

No. oryxflow ships a Claude Code **plugin (a skill plus slash commands)**, not an MCP server — the caching and lineage work happens in the local Python library. (MLflow separately ships an experimental MCP server; that's a fact about MLflow, not oryxflow.)

## Takeaway

Pick your MLflow alternative by the job you're actually replacing. If you want a better or lighter **tracker**, the field is strong — Weights & Biases for hosted collaboration, Neptune for scale, Comet for monitoring, ClearML for breadth, Aim for a lightweight local UI, DVCLive for Git-native logging, SageMaker Experiments on AWS. But if the pain you keep hitting is *reproducing* a result or *recomputing* work that didn't change, no tracker will fix it — that's a job for a caching reproducibility layer like oryxflow, which sits happily underneath whichever tracker you choose.

```
pip install oryxflow
```

**Read next:** [Do you need MLflow, or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md) · [oryxflow vs the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md) · [oryxflow vs DVC](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-dvc/index.md) · [When *not* to use oryxflow](https://docs.oryxflow.dev/blog/guides/when-not-to-use-oryxflow/index.md) · [Claude Code for data science](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md)

# Do you need MLflow, or do you need reproducible pipeline caching?

*They sound like the same problem. They're not — and mixing them up is why so many ML projects still can't reproduce last week's result.*

If you've searched for "manage machine learning experiments" you've been pointed at MLflow, Weights & Biases, and DVC. They're excellent tools. But a lot of people install one, log some metrics, and are surprised to find their pipeline is *still* a mess of stale pickle files they can't reliably reproduce. That's because these tools solve a **different half** of the problem than the one biting you.

## Two different problems that both get called "experiment management"

**Problem A — tracking:** *"Which run got 0.91 AUC, and what were its hyperparameters?"* This is a logging and comparison problem. It's what **MLflow**, **W&B**, and **Neptune** are built for: you call `log_metric(...)`, `log_param(...)`, and get a searchable dashboard of every run.

**Problem B — computation:** *"To reproduce that 0.91 run, which steps do I actually need to rerun, and which are already computed?"* This is a dependency and caching problem. It's about **not** recomputing a 10-minute feature step when only the model changed, and about **guaranteeing** the model you're evaluating was trained on the current data.

Trackers answer A. They do **not** answer B. MLflow will faithfully log that you got 0.91 — it has no idea whether the features feeding that model are stale, and it won't skip recomputing them for you. That gap is where reproducibility quietly dies.

## What a pipeline-caching engine does that a tracker doesn't

A workflow engine like [`oryxflow`](https://github.com/oryxintel/oryxflow) models your work as a DAG of tasks and owns the computation side:

```
import oryxflow

class GetData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(load_data())

@oryxflow.requires(GetData)
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(build_features(self.inputLoad()))

@oryxflow.requires(BuildFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):
    model = oryxflow.Parameter(default='gbm')

    def run(self):
        features = self.inputLoad()
        clf = fit(self.model, features)
        self.save(clf)
        self.saveMeta({'score': clf.score(...)})   # <-- log to MLflow here too

oryxflow.run(TrainModel())
```

From this it gives you three things a tracker structurally cannot:

1. **Skip-what's-done.** Rerun the script and completed tasks load from cache instead of recomputing. Change one task and only its downstream reruns.
1. **Correct-by-construction invalidation.** Change a parameter, the data, or a task's code, and exactly the affected outputs are marked stale and rebuilt. You can't accidentally evaluate a new model on old features.
1. **Load-any-result-by-name.** `TrainModel().output().load()` gives you the model; `BuildFeatures().output().load()` gives you the features — no hunting for `.pkl` paths.

## The honest answer: use both

This isn't "oryxflow vs MLflow." The two compose cleanly:

```
def run(self):
    features = self.inputLoad()
    clf = fit(self.model, features)
    self.save(clf)                         # oryxflow: caches + invalidates
    mlflow.log_param('model', self.model)  # MLflow: dashboard + comparison
    mlflow.log_metric('score', clf.score(...))
```

- **oryxflow** owns the *pipeline*: dependency order, caching, minimal reruns, reproducibility.
- **MLflow / W&B** own the *record*: the searchable history of what each run scored.

Put the tracker calls **inside** your oryxflow tasks and you get both a reproducible computation graph and a clean experiment log — without either tool pretending to be the other.

## What about DVC?

DVC is the tool people most often conflate with this space, because it *does* do pipeline caching. The honest difference is what identity is built from. **DVC hashes files and YAML-declared stages**: you describe your pipeline in `dvc.yaml` — each stage's command, dependencies, and outputs — and DVC recomputes a stage when a declared file hash changes. **oryxflow's identity is native Python task identity — parameters plus automatic code-change detection, zero config files**: the DAG *is* your `requires()` methods, a parameter change is automatically a new cached identity (no stage file to edit), and a code change reruns the task and everything downstream on its own. It compares what your code *does*, not how it's written, so comment and formatting edits never recompute (pin a task with `code_version` when you'd rather manage it by deliberate bumps). If your workflow is command-line stages over large versioned data files, DVC's file-hash model fits. If your workflow is Python tasks you iterate on inside a session — parameter sweeps, per-entity fan-outs — keeping identity in the code you're already editing beats maintaining a parallel YAML description of it.

## So which do you actually need?

- You can already reproduce runs but can't *compare* them → you want a **tracker** (MLflow/W&B).
- You can log runs but rerunning your pipeline is slow, fragile, and you're never sure what's stale → you want a **caching workflow engine** (oryxflow, or its heavier cousins Luigi, Metaflow, Kedro).
- Most real projects want both.

If it's the second problem you feel every day — the fifteen-minute edit-rerun loop, the `features_v3_final.pkl` graveyard, the "wait, was this trained on the new data?" — start here:

```
pip install oryxflow
```

Docs: https://docs.oryxflow.dev · Source: https://github.com/oryxintel/oryxflow

oryxflow is a lightweight, dependency-free alternative to Luigi, Metaflow, and Kedro, focused on research iteration rather than production orchestration — and it plays nicely with whatever tracker you already use.

## Frequently asked questions

### Do I need MLflow, or just pipeline caching?

It depends on which problem bites you. If you can already reproduce runs but can't compare them, you want a tracker like MLflow. If you can log runs but rerunning your pipeline is slow, fragile, and you're never sure what's stale, you want a caching workflow engine. Most real projects want both — oryxflow caches the computation while MLflow records the results.

### What's the difference between MLflow and pipeline caching?

MLflow answers tracking — which run got which metric with which parameters — and gives you a searchable dashboard. Pipeline caching answers computation — which steps must actually rerun, and which are already computed — so you never recompute an unchanged feature step or evaluate a model on stale data. A tracker records what happened; a caching engine like oryxflow makes the pipeline behind it reproducible.

### Can I use MLflow and oryxflow together?

Yes, and that's the recommended pattern. Put your tracker's logging calls inside your cached oryxflow tasks: oryxflow owns the pipeline — dependency order, caching, minimal reruns, reproducibility — while MLflow owns the record of what each run scored. You get a reproducible computation graph and a clean experiment log, without either tool pretending to be the other.

# From notebook to a reproducible, cached pipeline in Python

*Turn a working-but-fragile notebook into a pipeline you can trust — one step at a time, without a big rewrite.*

A notebook is the best place to start an analysis and the worst place to keep one.

While you are exploring, cells run out of order, variables linger in memory long after the cell that defined them was edited, and half your intermediate results are stale copies from an hour ago. It all still works — until you reopen the notebook a week later, hit "Run All," and get a different number. Now you are stuck asking the question every data scientist eventually asks: *which version of the code and which data actually produced this result?* If you cannot answer that, you cannot trust the result, and you cannot reproduce it for anyone else.

The usual culprits are hidden state (a variable that only exists because you ran a cell you have since deleted), out-of-order execution, and stale intermediates that never got refreshed after an upstream change. None of these show up as errors. They just quietly make your output wrong.

**oryxflow is a small Python library that lets you declare each step of your analysis as a task — with its parameters and its dependencies — so the library runs them in the right order, caches each output, and reruns a step only when its code or inputs actually change.** The payoff is that every result is tied to the exact code and parameters that produced it. That is reproducibility you get for free, not reproducibility you have to remember to maintain.

The good news: you do not rewrite your notebook to get there. You migrate it one step at a time, and you have a working pipeline after every single step.

## Start with the linear script

Here is a typical notebook, flattened into the script it really is: load data, build features, train a model.

```
import pandas as pd
from sklearn.linear_model import LinearRegression

df = pd.read_parquet('raw.parquet')          # slow: pulls a big table

df['x_squared'] = df['x'] ** 2               # feature engineering
features = df[['x', 'x_squared']]

model = LinearRegression().fit(features, df['y'])
```

Every time you tweak the model, that first line pulls the whole table again. And nothing records that *this* model came from *that* data and *that* feature code.

## Convert the first expensive step into a task

Do not convert everything. Start with the one step that hurts most — usually the slow load. Wrap it in a task class, and replace the return with `self.save(...)`.

```
import oryxflow
import pandas as pd

class GetData(oryxflow.tasks.TaskPqPandas):   # TaskPqPandas = saves a DataFrame as parquet
    def run(self):
        df = pd.read_parquet('raw.parquet')
        self.save(df)
```

Run it:

```
flow = oryxflow.Workflow(GetData)
flow.run()
df = flow.outputLoad()                         # the saved DataFrame, back in your hands
```

The first `flow.run()` does the slow load and caches the result. Every run after that is a cache hit — it loads from disk instead of re-fetching. Your remaining notebook cells keep working, now fed by `df = flow.outputLoad()`. You have already gained something and rewritten almost nothing.

## Wire the next step with @oryxflow.requires

Now pull feature engineering into its own task. The `@oryxflow.requires(GetData)` decorator declares the dependency; inside `run`, `self.inputLoad()` hands you `GetData`'s already-loaded output.

```
@oryxflow.requires(GetData)
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                  # GetData's output, already loaded
        df['x_squared'] = df['x'] ** 2
        self.save(df[['x', 'x_squared', 'y']])
```

You never call `GetData` yourself. You ask for `BuildFeatures`, and oryxflow runs its dependencies first, in order — no more "did I run the cells in the right sequence?"

## Finish the DAG

Add the model as a final task. Models are not DataFrames, so save them with `TaskPickle`. This is also where parameters earn their keep: expose the knobs you actually sweep.

```
from sklearn.linear_model import LinearRegression

@oryxflow.requires(BuildFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):   # TaskPickle = pickle, for models & arbitrary objects
    power = oryxflow.IntParameter(default=2)
    fit_intercept = oryxflow.BoolParameter(default=True)

    def run(self):
        df = self.inputLoad()
        X, y = df[['x', 'x_squared']], df['y']
        model = LinearRegression(fit_intercept=self.fit_intercept).fit(X, y)
        self.save(model)
```

That is the whole pipeline: `GetData → BuildFeatures → TrainModel`. Preview it before running anything:

```
flow = oryxflow.Workflow(TrainModel)
flow.preview()                                 # prints the task tree, runs nothing
flow.run()
model = flow.outputLoad()
```

`preview()` shows you the execution plan — which steps will run and which are already cached — without touching your data. When you are ready, `run()` executes only what is needed.

## What you actually gain

**Reproducibility by construction.** Each task's output is stored under an id derived from its code and its parameters. Change the feature formula, and oryxflow knows `BuildFeatures` is now different from the version that produced the cached file. "Which data made this result?" stops being a detective problem.

**No wasted recompute.** Rerun the pipeline and nothing happens if the outputs already exist — cache hits all the way down. Edit `BuildFeatures`, though, and oryxflow reruns it *and* `TrainModel` automatically, because the model depends on features that just changed. **Code invalidation is automatic** — there is no cache to manually clear and no `reset` to remember. This is the same mechanism the sibling post [Stop rerunning your whole pipeline](https://docs.oryxflow.dev/blog/caching/cache-intermediate-dataframes-python/index.md) digs into.

**Load any result by name.** Every intermediate is addressable. Want the features without rerunning the model? `oryxflow.Workflow(BuildFeatures).outputLoad()` hands back the cached frame. Comparing two settings? `oryxflow.Workflow(TrainModel, {'power': 3}).outputLoad()` loads that configuration's model directly. No more scrolling for the cell that defined the variable you need.

## What you do not have to decide upfront

A twenty-minute first look at a new dataset does not need tasks. If you will not run any of it twice, the task scaffolding is pure overhead — pull a step into a task when the computation is slow enough to want caching, or the result is important enough to need reproducing.

What you do not have to do is decide that in advance. If you use Claude Code, the oryxflow plugin gives that first look a home inside the project — a read-only probe that states the question it answers and still runs next session, instead of a cell you lose — and when a probe turns out to be load-bearing, `/oryxflow:migrate` promotes it into cached, parameterized tasks, reading your script as the spec and never deleting it. So "is this analysis big enough for a pipeline yet?" is a question you can answer by doing the work rather than guessing first: start with a simple script and let it scale as the work gets complex, with no rewrite in between. Walkthrough: [migrate a notebook to a pipeline](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md).

Where oryxflow genuinely stops is operations. It is built for the research loop, not for production orchestration; if you need scheduling, retries, and alerting across a fleet, use a real orchestrator. oryxflow makes *your own analysis* trustworthy and fast to iterate on.

A good rule of thumb: pull a step into a task the second time you find yourself waiting for it to recompute something that did not change.

## Takeaway

- Migrate incrementally. Wrap the most expensive step first, keep a working pipeline at every stage, and never do a big-bang rewrite.
- `@oryxflow.requires` wires dependencies; `self.inputLoad()` reads them; `self.save()` caches the output.
- Editing a task reruns it and everything downstream automatically — reproducibility you do not have to maintain by hand.

```
pip install oryxflow
```

If you use Claude Code, the oryxflow plugin's `/oryxflow:migrate` command walks a script through exactly this conversion, one step at a time.

## Frequently asked questions

### How do I convert a Jupyter notebook into a reproducible pipeline?

Migrate incrementally rather than rewriting. Wrap your slowest step — usually the data load — in an oryxflow task that ends with `self.save()`, run it once so the result caches, then feed the rest of your notebook from that cached output. Add one step at a time with `@oryxflow.requires` to wire dependencies. You keep a working pipeline at every stage, and each output is tied to the exact code and parameters that produced it.

### Do I have to rewrite my whole notebook at once to use oryxflow?

No — that's the point of migrating incrementally. Start by converting the single step that hurts most, usually the slow load, and leave the rest of your cells untouched, now fed by `flow.outputLoad()`. You have a working pipeline after every step, and you only pull the next step into a task when it earns it. oryxflow is designed for this one-step-at-a-time conversion, not a big-bang rewrite.

### Why do I get different numbers when I rerun a notebook a week later?

Usually it's hidden state, out-of-order cell execution, or stale intermediate results that never refreshed after an upstream change — none of which surface as errors, they just quietly make your output wrong. oryxflow removes all three by declaring each step as a task with explicit dependencies: the engine runs them in the right order, caches each output under an id derived from its code and parameters, and reruns a step only when its code or inputs actually change.

Next steps: [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md) · [Transition guide](https://docs.oryxflow.dev/docs/transition/index.md) · [Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md) · [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) · [GitHub](https://github.com/oryxintel/oryxflow)

# oryxflow vs Airflow: research workflows vs production orchestration

*They both run DAGs of Python tasks, so they get compared. But they're built for different jobs — and picking the wrong one is why so many data scientists either drown in infrastructure they don't need or hand-roll caching they shouldn't.*

If you searched "Airflow for data science experiments," this post is the short answer: for a scheduled production pipeline, use Airflow. For the iterative research loop *before* production — EDA, feature engineering, model comparison, all edited dozens of times a day — a lightweight code-aware caching library like [oryxflow](https://github.com/oryxintel/oryxflow) fits better. Here's the honest breakdown.

## What each tool is actually for

**Apache Airflow** is a production workflow *scheduler*. Its job is to run pipelines on a schedule, across a cluster, reliably: cron-style triggers, retries, backfills, alerting, SLAs, and a UI that shows you the state of every run. To do that it runs real infrastructure — a scheduler, a metadata database, and (in 3.x) a dag-processor and API server. That machinery is exactly what you want when a pipeline must run at 2 a.m. every night and page someone if it fails.

**oryxflow** is a research-loop *caching engine*. Its job is to make an analysis you're actively editing fast and trustworthy: cache every task's output, rerun exactly what a code, data, or parameter change affects, and record what ran and why. It's a `pip install` with no server, no database, and no account.

Neither is a worse version of the other. They're built for different phases of the same project.

## Three differences that decide it for research

### 1. Setup: a scheduler and a database, or `pip install`

To iterate on a model with Airflow you stand up (or connect to) a scheduler and a metadata DB, and you express your work as a DAG the scheduler owns. For a nightly production job, that's justified. For "I want to try a different feature and see the score," it's a lot of ceremony between you and the answer.

oryxflow adds nothing to run: `pip install oryxflow`, write task classes, call `flow.run()`. The state lives in a local `data/` folder and a plain lineage log — no daemon to keep alive.

### 2. Passing data between steps

This is the one that bites data scientists specifically. **Airflow's XCom is designed for small metadata, not DataFrames** — its own docs warn not to use it to pass around large values. So in Airflow you write the persistence yourself: each task saves its DataFrame somewhere and the next task loads it, and you manage those paths. (Airflow 2.8+ has an Object Storage XCom backend and pandas serializers, but they're opt-in wiring.)

oryxflow *is* the data-passing layer. A task's base class decides its format, `self.save(df)` persists it keyed on (task, params), and `self.inputLoad()` in the next task hands it back — already loaded, no path anywhere in your code.

### 3. Editing code and reruns

Airflow doesn't cache your task *outputs* by code identity — it schedules and runs tasks. Change a task's logic and Airflow will happily run the whole DAG again on its next trigger; there's nothing that says "only the tasks whose code changed need to rerun."

oryxflow's core move is exactly that. It fingerprints each task's code — and every helper it references, transitively — so editing one function reruns exactly the affected tasks and everything downstream, automatically, while the expensive upstream stays cached. Cosmetic edits never recompute. That turns a fifteen-minute edit-rerun loop into seconds. See [Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/#automatic-code-invalidation) for how the invalidation works.

## Side by side

|                                    | oryxflow                                     | Airflow                                                     |
| ---------------------------------- | -------------------------------------------- | ----------------------------------------------------------- |
| **Built for**                      | iterative research loop                      | scheduled production pipelines                              |
| **Setup**                          | `pip install`, no server                     | scheduler + metadata DB (+ dag-processor/API server in 3.x) |
| **Passing DataFrames**             | native `save`/`inputLoad`, auto paths        | XCom is small-data only; you write persistence              |
| **Rerun on a code edit**           | automatic, per-symbol, downstream too        | not by code identity                                        |
| **Caching**                        | every task, by (code, params), on by default | not a caching engine                                        |
| **Scheduling / retries / SLAs**    | ❌ (not its job)                             | ✅ its core strength                                        |
| **Distributed execution at scale** | ❌                                           | ✅                                                          |
| **UI / run dashboard**             | queryable lineage log (no UI)                | rich web UI                                                 |

Read that table the right way: the bottom rows aren't oryxflow "losing." They're Airflow doing the job it exists for. If you need them, you need Airflow.

## They're complementary

The mature pattern on a real project is to use both, in sequence:

- **Develop and iterate with oryxflow.** The research loop — where you change code constantly and want instant, correct reruns — is where caching pays off and a scheduler is overhead.
- **Productionize with Airflow.** Once the pipeline is stable and needs to run on a schedule across infrastructure with retries and alerting, that's Airflow's job.

oryxflow even makes the handoff cleaner: because your logic is already decomposed into tasks with explicit dependencies, wrapping the stable pipeline in an orchestrator later is mechanical rather than a rewrite.

## Which do you need?

- **You're iterating on analysis or a model, locally, editing all day, and tired of rerunning the slow steps** → oryxflow.
- **You need a pipeline to run on a schedule, across a cluster, with retries and alerting** → Airflow.
- **Both, at different stages** → the common case. Iterate with oryxflow; schedule the finished thing with Airflow.

```
pip install oryxflow
```

## Frequently asked questions

### Is Airflow overkill for a local data science project?

For a scheduled production pipeline, Airflow's scheduler and metadata database earn their keep. For a local data science project you edit all day, that infrastructure is overhead between you and the answer. oryxflow is a lighter alternative: a pip install with no server or database that caches every task output and reruns only what your code or parameters changed.

### What's a lightweight alternative to Airflow for research pipelines?

oryxflow is a lightweight, local-first alternative to Airflow built for the research loop rather than production scheduling. You pip install it, write task classes, and call run() — no scheduler, metadata database, or dag-processor. It caches each task by code and parameters, reruns exactly what a code change affects, and records lineage to a plain log you can grep.

- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** — reproducibility, lineage, and trustworthy AI data analysis.
- **[oryxflow vs the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md)** — the full framework comparison.
- **[Stop rerunning your whole pipeline](https://docs.oryxflow.dev/blog/caching/cache-intermediate-dataframes-python/index.md)** — the caching payoff in depth.
- Source & examples: <https://github.com/oryxintel/oryxflow>

# oryxflow vs Dagster: a lightweight research loop vs an asset platform

*Dagster and oryxflow both build DAGs of Python steps and both persist step outputs automatically — so they get compared. But one is a production asset platform for a data team, and the other is a `pip install` for the research loop. Picking well starts with being honest about which problem you're solving.*

If you searched "Dagster for machine learning experimentation," here's the short answer up front: Dagster is an excellent, mature *orchestrator* for a team running production data assets — and [oryxflow](https://github.com/oryxintel/oryxflow) is a small, local-first library for the fast, messy iteration that happens *before* anything is a production asset. **oryxflow is a zero-infrastructure Python library that turns the data-science scripts you edit all day into a cached, dependency-aware DAG you can trust** — reproducible outputs, recorded lineage, and reruns that follow your code changes. Below is the honest breakdown, including where Dagster clearly wins.

## What each tool is actually for

**Dagster** is an asset-oriented data orchestrator. You declare *software-defined assets*, and Dagster gives you a rich web UI, scheduling, sensors, partitions and backfills, and — importantly — **IO managers that persist each asset's output automatically**. It's built for a data team running and observing data assets in production: you can see every asset's freshness, backfill a partition, and trigger runs off sensors. The full experience assumes a running Dagster instance and its UI.

**oryxflow** is a research-loop tool. Its job is to make an analysis you're *actively editing* fast and trustworthy: every task's output is cached, reruns follow what your code and parameters actually changed, and every run is logged for lineage. It's a `pip install` with no server, no database, no account, and no telemetry — the state lives in a local `data/` folder and a plain lineage log.

Neither is a worse version of the other. They sit at different layers of the same project's life.

## Doesn't Dagster already persist outputs automatically?

Yes — and this is the honesty checkpoint. **Dagster's IO managers do persist asset outputs automatically.** oryxflow did not invent automatic persistence; Dagster, Metaflow, Kedro, and Ploomber all do it too. So auto-persistence is *not* the differentiator, and any comparison that pretends otherwise is selling you something.

The real difference is *how much you wire* and *what environment it assumes*:

- In Dagster, you declare assets and **choose and configure an IO manager** (or accept a default and configure where it writes). That's a powerful, explicit contract — and it's exactly right for a platform where storage, partitioning, and environments matter. It's also configuration you own.
- In oryxflow, the **task's base class decides the format** — `TaskPqPandas` writes a DataFrame to parquet, `TaskPickle` pickles a model, `TaskJson` writes JSON. There's no IO manager to pick, no catalog to maintain, and no server. You write `self.save(obj)` and the next task calls `self.inputLoad()`; the format and the path are handled by *type*, keyed on `(task, params)`.

That's the precise oryxflow differentiator: **type-driven, zero-config I/O plus automatic code-change invalidation, entirely local.** Task identity is native Python — a class with parameters — not a YAML asset definition or a catalog entry.

## What about reruns when I change my code?

This is the move oryxflow is built around. It tracks each task's code — and every helper file it imports — comparing what your code *does*, not how it's written, so comments and reformatting don't count. Edit one task, or a helper it imports, and on the next `run()` oryxflow reruns exactly that task and everything downstream, while the expensive upstream stays cached. Change nothing and it recomputes nothing.

```
import oryxflow
class LoadData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        self.save(load_data())            # cached once, keyed on (code, params)

@oryxflow.requires(LoadData)
class Features(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()
        self.save(add_features(df))       # reruns only when this code (or a helper) changes

@oryxflow.requires(Features)
class TrainModel(oryxflow.tasks.TaskPickle):
    model = oryxflow.ChoiceParameter(choices=['ols', 'gbm'])
    def run(self):
        df = self.inputLoad()
        m = fit(df, self.model)
        self.save(m)
        self.saveMeta({'score': m.score})

# compare models — shared upstream (LoadData, Features) computes once
flow = oryxflow.WorkflowMulti(TrainModel, {'ols': {'model': 'ols'},
                                           'gbm': {'model': 'gbm'}})
flow.run()
flow.outputLoadMeta()                      # {'ols': {'score': ...}, 'gbm': {'score': ...}}
```

Dagster tracks asset materializations and code versions and can help you reason about staleness, but the everyday oryxflow loop — *edit a function, rerun, only the affected tasks recompute, no server involved* — is the thing it optimizes for. One honest caveat that applies to both tools: oryxflow makes your results **reproducible, not correct**. It guarantees the same inputs and code produce the same outputs; it does not check that your analysis is right. That's still your job.

## Side by side

|                                      | oryxflow                              | Dagster                                                 |
| ------------------------------------ | ------------------------------------- | ------------------------------------------------------- |
| **Built for**                        | solo/small-team research loop         | production data assets for a team                       |
| **Setup**                            | `pip install`, no server              | a running Dagster instance + UI for the full experience |
| **Auto-persist outputs**             | ✅ type-driven, zero config           | ✅ via IO managers you configure                        |
| **Configuring I/O**                  | none — base class picks the format    | declare assets, choose/configure an IO manager          |
| **Task identity**                    | native Python class + params          | software-defined assets                                 |
| **Rerun on a code edit**             | automatic, per-symbol, downstream too | code versions + staleness in the platform               |
| **Scheduling / sensors / backfills** | ❌ (not its job)                      | ✅ its core strength                                    |
| **Web UI / observability**           | queryable lineage log (no UI)         | rich web UI                                             |
| **Partitions at scale**              | ❌                                    | ✅                                                      |

Read the ❌ rows the right way: they aren't oryxflow "losing." They're Dagster being a platform. If you need a UI, schedules, sensors, and partitioned backfills, you need Dagster — a caching library has no business pretending to replace them.

## They're complementary

The mature pattern uses both, in sequence:

- **Iterate with oryxflow.** During research — where you change code constantly and want instant, correct reruns — a local cache pays off and a server is overhead.
- **Promote to Dagster** when the pipeline stabilizes and needs to become an observed, scheduled, partitioned production asset for the team.

Because your oryxflow logic is already decomposed into tasks with explicit dependencies, wrapping the stable pipeline as Dagster assets later is mechanical, not a rewrite.

## Which do you need?

- **You're iterating on a model or analysis, locally, editing all day, and tired of rerunning slow upstream steps** → oryxflow.
- **A team needs to run, schedule, observe, and backfill production data assets with a UI** → Dagster.
- **Both, at different stages** → the common case. Iterate in oryxflow; promote to Dagster when it's production.

## Takeaway

Dagster and oryxflow both auto-persist outputs, so don't choose on that. Choose on the layer: oryxflow is a zero-infrastructure, type-driven, code-aware cache for the research loop; Dagster is a server-backed asset platform for production. For fast, reproducible experimentation, `pip install` and iterate — then hand the stable pipeline to Dagster when it's ready.

```
pip install oryxflow
```

## Frequently asked questions

### Is there a lighter-weight alternative to Dagster for local analysis?

Yes. oryxflow is a lightweight alternative to Dagster for local analysis: a pip install with no server, database, or UI. Where Dagster is an asset platform a team runs in production, oryxflow caches the research loop you edit all day — type-driven zero-config I/O, automatic code-change invalidation, and lineage in a plain local log. Promote the stable pipeline to Dagster later.

### Do I need a Dagster instance and UI just to cache pipeline steps locally?

No. Dagster's IO managers persist outputs, but the full experience assumes a running Dagster instance and its UI. oryxflow persists every task output locally with zero configuration — the task's base class picks the format and keys it on task and params — so you get cached, dependency-aware steps from a pip install alone, no server or catalog to maintain.

- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** — reproducibility, lineage, and trustworthy AI data analysis.
- **[oryxflow vs Airflow](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md)** — research workflows vs production orchestration.
- **[oryxflow vs the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md)** — the full framework comparison.
- Source & examples: <https://github.com/oryxintel/oryxflow>

# oryxflow vs DVC: Python task identity vs file-hash data versioning

*They both promise reproducible pipelines, so they get pitched as alternatives. But they build "the same task" out of completely different things — and once you see what, you'll want both.*

If you searched "DVC vs Python workflow library," here's the honest short answer up front: DVC versions large data and model files alongside Git and reruns a pipeline stage when a declared file hash changes; [oryxflow](https://github.com/oryxintel/oryxflow) is a small, local-first Python library that turns your task classes into a cached, dependency-aware DAG whose identity is built from parameters plus automatic code-change detection. Neither replaces the other. This is the deeper dive on where the line falls (there's a shorter DVC section in [Do you need MLflow, or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md) — this post goes further).

## What is oryxflow, in one sentence?

oryxflow is a zero-infrastructure Python library that makes an analysis you're actively editing fast and trustworthy: you declare `Task` classes with parameters and `requires()` dependencies, it runs the DAG in dependency order, caches every output, and reruns exactly what a code, data, or parameter change affects — with a lineage trail of what ran and why.

## What is DVC actually for?

DVC (Data Version Control) is a **data-versioning** tool. Its center of gravity is putting large files — datasets, model weights, intermediate artifacts — under version control *alongside your Git history*, without bloating the repo. Git tracks a small `.dvc` pointer; the real bytes live in remote storage (S3, GCS, SSH, and friends) that DVC pushes and pulls.

On top of that, DVC has a **pipeline** layer. You describe stages in `dvc.yaml`: each stage names a shell **command**, its input **dependencies** (files/params), and its **outputs** (files). DVC hashes those declared files and reruns a stage when a hash changes — file-hash pipeline caching. It also has DVC Experiments for sweeping parameters and comparing runs, all pinned to Git commits.

That is a genuinely great fit when your pipeline is a chain of shell commands over big data files that need to be versioned and shared, reproducible from a specific commit.

## So what's the real difference from oryxflow?

Not "caching" — both cache. The honest core difference is **what identity is built from**.

**DVC's identity is files and YAML-declared stages.** A stage is defined by its command string and the hashes of the files you told it to watch. It's CLI-command-oriented: DVC doesn't look inside your Python, it looks at the files crossing the boundary. If your logic changes but the declared output file happens to hash the same — or if you forget to declare a dependency — DVC's model can't see it.

**oryxflow's identity is native Python task identity.** A task's cache key is its parameters plus its own code *and* every helper file it imports — compared by what the code *does*, not how it's written. There's no `dvc.yaml`, no command strings, no manually declared file lists: the DAG *is* your `requires()` methods, and the I/O is type-driven and zero-config — a `TaskPqPandas` saves a DataFrame to parquet, `self.inputLoad()` hands it to the next task, and no path appears anywhere in your code.

Two consequences fall out of that, automatically:

- **A parameter change is a new cached identity.** Run a task with `alpha=0.1` and again with `alpha=0.2` and you get two distinct cached outputs, side by side — no config edit, no new stage.
- **A code change reruns downstream on its own.** Edit a function's logic and oryxflow reruns that task and everything downstream while the expensive upstream stays cached. Edit only comments or formatting and nothing recomputes — oryxflow compares what your code *does*, not how it's written, so cosmetic edits are free. DVC would need you to keep `dvc.yaml` and your file declarations in step with the code by hand.

Here's the whole loop in oryxflow — verified API, no config files anywhere:

```
import oryxflow

class MakeFeatures(oryxflow.tasks.TaskPqPandas):
    horizon = oryxflow.IntParameter(default=30)

    def run(self):
        raw = load_raw_frame()            # your code
        self.save(build_features(raw, self.horizon))

@oryxflow.requires(MakeFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):
    alpha = oryxflow.FloatParameter(default=0.1)

    def run(self):
        features = self.inputLoad()       # upstream output, already loaded
        model = fit(features, self.alpha)
        self.save(model)
        self.saveMeta({'alpha': self.alpha})

flow = oryxflow.Workflow(TrainModel, {'alpha': 0.2, 'horizon': 60})
flow.run()                                # runs only what's missing or changed
model = flow.outputLoad()
```

Change `alpha`, `flow.run()` again, and only `TrainModel` recomputes — `MakeFeatures` is served from cache because neither its params nor its code changed. Edit the body of `build_features` and both rerun. You wrote no stage file to make that happen.

## oryxflow vs DVC, side by side

|                                       | oryxflow                                    | DVC                                       |
| ------------------------------------- | ------------------------------------------- | ----------------------------------------- |
| **Primary job**                       | in-session Python compute graph             | versioning big data/model files with Git  |
| **Identity built from**               | params + automatic code-change detection    | file hashes + `dvc.yaml` stage commands   |
| **Pipeline definition**               | native `requires()` methods, no config      | `dvc.yaml` stages (command + deps + outs) |
| **Data I/O**                          | type-driven, zero-config `save`/`inputLoad` | you write the command's file reads/writes |
| **Parameter change**                  | automatically a new cached identity         | via params files / DVC Experiments        |
| **Code edit reruns downstream**       | automatic, including edits to helpers       | only if a declared file hash changes      |
| **Data versioning to remote storage** | ❌ (not its job)                            | ✅ its core strength                      |
| **Ties artifacts to Git commits**     | ❌                                          | ✅                                        |
| **Setup**                             | `pip install`, local `data/` folder         | Git + a DVC remote                        |
| **Lineage**                           | `.oryxflow/events.jsonl` event log          | Git history + `dvc.lock`                  |

Read the ❌ rows the right way: they aren't oryxflow "losing." Versioning large files to a remote and pinning them to commits is the job DVC exists for. If you need it, you need DVC.

## Which do you need?

- **The job is versioning big data/model files, sharing them off a remote, and reproducing a run from a specific Git commit** → DVC.
- **The job is iterating on Python tasks in a session — parameter sweeps, per-entity fan-outs, an edit-rerun loop you run all day, and you want reruns scoped automatically to what changed** → oryxflow.
- **Your pipeline is naturally a set of shell stages over versioned files** → DVC's model fits.
- **Your pipeline is naturally a graph of Python functions passing DataFrames** → oryxflow's model fits without the YAML.

## They compose

This isn't either/or. The clean pattern uses each for its strength:

- **DVC for artifact and data versioning** tied to Git — the big input datasets and the model files you want reproducible from a commit and shareable via a remote.
- **oryxflow for the Python compute graph on top** — the in-session iteration where you're changing code constantly and want instant, correctly-scoped reruns without touching a config file.

And if you *do* want oryxflow's local `data/` outputs versioned, you don't have to leave the ecosystem: the Claude Code plugin's `/oryxflow:init-gitlfs` command puts `data/` under Git LFS for you. (oryxflow ships as a [Claude Code plugin](https://github.com/oryxintel/oryxflow-claude-plugin) — a skill and slash commands — not an MCP server.)

One honest caveat that applies to both, and to every caching tool: they make your pipeline **reproducible, not correct**. oryxflow guarantees the model you're evaluating was trained on the current data and that stale steps get rerun — it does not check that your logic is *right*. That's still your job (and your tests').

## Takeaway

DVC and oryxflow both promise reproducibility, but they build "the same task" out of different material: DVC hashes files and YAML-declared stages, oryxflow derives identity from Python parameters and automatic code-change detection. Pick DVC when the job is versioning big data to Git and a remote; pick oryxflow when the job is iterating on Python tasks in a session. On a real project you often want both — DVC underneath for versioned artifacts, oryxflow on top for the compute graph.

```
pip install oryxflow
```

## Frequently asked questions

### How do I cache a Python pipeline without DVC and YAML stages?

You can cache a Python pipeline without DVC's dvc.yaml stages using oryxflow, a local-first library that builds task identity from parameters plus automatic code-change detection. You declare Task classes with requires() dependencies; it caches every output keyed on code and params and reruns only what a code or parameter change affects. There are no command strings or file lists to declare by hand.

### oryxflow vs DVC — which do I need?

Use DVC when the job is versioning large data or model files alongside Git and reproducing a run from a specific commit. Use oryxflow when the job is iterating on Python tasks in a session with automatic, code-aware caching. They aren't rivals; on a real project you often run both — DVC underneath for versioned artifacts, oryxflow on top for the compute graph.

**Read next**

- **[Do you need MLflow, or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md)** — the shorter DVC section and the tracking-vs-caching split.
- **[oryxflow vs the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md)** — the full framework comparison.
- **[oryxflow vs Airflow](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md)** — research caching vs production scheduling.
- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** — reproducibility, lineage, and trustworthy AI data analysis.
- **[Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md)** — how automatic code invalidation works.
- **[Build with Claude Code](https://docs.oryxflow.dev/docs/claude-plugin/index.md)** — the plugin, skill, and slash commands (including `/oryxflow:init-gitlfs`).
- Source & examples: <https://github.com/oryxintel/oryxflow>

# oryxflow vs Prefect: research-loop caching vs Python-native orchestration

*Both use plain Python decorators to build task DAGs, so they get compared. But one is built to run and observe pipelines in production, and the other is built to make the messy research loop before production fast, reproducible, and trustworthy.*

If you searched "Prefect for data science," here's the honest short answer: for running, scheduling, and observing pipelines across infrastructure, use Prefect. For the iterative research loop *before* production — EDA, feature engineering, model comparison, edited dozens of times a day — a code-aware caching library like [oryxflow](https://github.com/oryxintel/oryxflow) fits better. They aren't rivals; they're different layers of the same project.

**oryxflow** is a local-first Python library that turns your analysis scripts into a cached, dependency-aware task DAG, so every result is reproducible and traceable to the exact code and parameters that produced it. No server, no database, no account — just `pip install`.

## What is Prefect, and what is it for?

Prefect is a modern, Python-native workflow **orchestrator**. You decorate functions with `@flow` and `@task`, and Prefect gives you dynamic flows, automatic retries, scheduling, concurrency limits, and a server/UI (self-hosted or Prefect Cloud) that observes every run across your infrastructure. It's the answer to "this pipeline needs to run reliably, on a schedule, somewhere other than my laptop, and someone needs to see when it breaks."

That's a genuinely different job from what a data scientist does at 2 p.m. on a Tuesday, twenty edits into a feature-engineering experiment, waiting on the same slow load step to rerun for the tenth time.

## Doesn't Prefect already cache results?

Yes — and this is the part most comparisons get wrong, so let's be precise. Prefect **can** cache task results and persist them. You configure a cache key (via a `cache_key_fn` or a cache policy) and choose result storage, and Prefect will reuse a cached result when the key matches. It's real, it works, and on a production flow it's exactly the right amount of control.

The difference is where the effort lives. In Prefect, caching is something you **configure**: you decide the cache key, you wire the result storage. In oryxflow, caching is the **default behavior** and it's automatic by `(code, params)` — there is no cache key to write. And critically, oryxflow's cache is **code-change-aware**:

- Edit a task's logic — or a helper function it imports, transitively — and that task reruns on the next run, along with everything downstream.
- Edit a comment, reformat, rename a local variable cosmetically — oryxflow compares what your code *does*, not how it's written, so nothing recomputes.

You never hand-author a cache key that says "invalidate when the code changes," because the code *is* the key. That's the property you want in a research loop, where the code changes constantly and getting the invalidation boundary wrong means either stale results or a full rerun.

## What does that look like in code?

No cache keys, no result-storage config, no catalog file. Task base classes pick the on-disk format from the type; dependencies are declared with a decorator; I/O is `save`/`inputLoad`:

```
import oryxflow

class LoadData(oryxflow.tasks.TaskPqPandas):        # DataFrame -> parquet, automatically
    def run(self):
        self.save(load_data())                       # your loader; a stand-in here

@oryxflow.requires(LoadData)                          # declares dep AND copies params
class AddFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                         # upstream output, already loaded
        self.save(add_features(df))

@oryxflow.requires(AddFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):          # model object -> pickle, automatically
    model = oryxflow.ChoiceParameter(choices=['ols', 'gbm'])

    def run(self):
        df = self.inputLoad()
        m = fit(df, self.model)
        self.save(m)
        self.saveMeta({'model': self.model})          # lightweight metrics/lineage

flow = oryxflow.Workflow(TrainModel, {'model': 'gbm'})
flow.run()                                            # runs only what's stale
model = flow.outputLoad()
```

Comparing two models reuses the shared upstream automatically — `LoadData` and `AddFeatures` compute once:

```
flows = oryxflow.WorkflowMulti(TrainModel, {'ols': {'model': 'ols'},
                                            'gbm': {'model': 'gbm'}})
flows.run()
scores = flows.outputLoadMeta()                       # {'ols': {...}, 'gbm': {...}}
```

Now edit `add_features` and rerun: `AddFeatures` and `TrainModel` recompute, `LoadData` stays cached — no config, no cache key. Every run appends to a plain lineage log under `.oryxflow/events.jsonl`, so you can trace what produced any result. (Worth saying plainly: oryxflow makes results *reproducible*, not *correct* — it reruns the right tasks; it doesn't check that your analysis is right.)

## Side by side

|                                        | oryxflow                                      | Prefect                                                         |
| -------------------------------------- | --------------------------------------------- | --------------------------------------------------------------- |
| **Built for**                          | iterative research loop, pre-production       | running/observing pipelines in production                       |
| **Setup**                              | `pip install`, no server or DB                | Python-native; server/UI (self-host or Cloud) for orchestration |
| **Caching**                            | automatic, on by default, by `(code, params)` | supported, but you configure the cache key + result storage     |
| **Code-change-aware invalidation**     | ✅ automatic, per-symbol, downstream too      | you author the cache key yourself                               |
| **Passing DataFrames between steps**   | native `save`/`inputLoad`, type-driven paths  | you pass/return values; result storage is configured            |
| **Scheduling / retries / concurrency** | ❌ (not its job)                              | ✅ core strength                                                |
| **Distributed / infra-wide execution** | ❌                                            | ✅                                                              |
| **Run dashboard / observability UI**   | queryable lineage log (no UI)                 | rich web UI + API                                               |
| **Server / account / telemetry**       | none — fully local                            | server or Prefect Cloud for the full experience                 |

Read the ❌ rows the right way: they're Prefect doing the job it exists for. If you need scheduling, retries, and a live dashboard across infrastructure, you need an orchestrator.

## They're complementary

The mature pattern uses both, in sequence:

- **Prototype and iterate in oryxflow.** The research loop — where you change code all day and want instant, correctly-scoped reruns — is where zero-config, code-aware caching pays off and orchestration is overhead.
- **Orchestrate the stable pipeline in Prefect.** Once the pipeline is settled and needs to run on a schedule, across infrastructure, with retries and observability, that's Prefect's job.

Because your oryxflow logic is already decomposed into tasks with explicit dependencies, wrapping the finished pipeline in a Prefect `@flow` later is mechanical, not a rewrite.

## Which do you need?

- **Iterating on analysis or a model, locally, editing all day, tired of rerunning the slow steps** → oryxflow.
- **A pipeline that must run on a schedule, across infrastructure, with retries and a live UI** → Prefect.
- **Both, at different stages** → the common case. Iterate in oryxflow; orchestrate the finished thing in Prefect.

One more distinction worth naming: oryxflow ships as a **Claude Code plugin** (a skill plus slash commands), not an MCP server — so your AI assistant can drive the research loop with the same cached, reproducible tasks you use by hand. See the [Claude plugin docs](https://docs.oryxflow.dev/docs/claude-plugin/index.md).

## Takeaway

Prefect and oryxflow both build DAGs from plain Python, and both can cache results — but the caching philosophy is opposite. Prefect gives you *control*: you configure the cache key and result storage, which is what a production orchestrator should do. oryxflow gives you *defaults*: automatic, code-change-aware caching with no key to write, which is what a fast, trustworthy research loop needs. Iterate in oryxflow, then orchestrate the stable pipeline in Prefect.

```
pip install oryxflow
```

## Frequently asked questions

### Do I need Prefect for a research pipeline, or something lighter?

For running, scheduling, and observing pipelines across infrastructure, use Prefect. For a research pipeline you edit all day on your laptop, something lighter fits better. oryxflow is a local-first alternative: a pip install with no server or database that caches every task by code and params and reruns exactly what your code changes affect — no cache key to configure.

### Is there a lightweight Prefect alternative for local data science?

Yes. oryxflow is a lightweight, local-first alternative to Prefect for data science, built for the research loop rather than production orchestration. Prefect can cache results but you configure the cache key and result storage; oryxflow caches automatically by code and params, is code-change-aware out of the box, and passes DataFrames between steps with type-driven save and inputLoad — no server, account, or telemetry.

- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** — reproducibility, lineage, and trustworthy AI data analysis.
- **[oryxflow vs Airflow](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md)** — research workflows vs production orchestration.
- **[oryxflow vs the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md)** — the full framework comparison.
- Source & examples: <https://github.com/oryxintel/oryxflow> · Claude plugin: <https://github.com/oryxintel/oryxflow-claude-plugin>

# oryxflow vs. the field: pipeline frameworks for AI data analysis

*A hands-on comparison for one common situation — a solo analyst (or their AI coding agent) iterating on a research pipeline all day, locally, on Windows. Which framework actually fits, and which are built for a different job?*

The pipeline-tooling landscape is crowded, and most comparisons are unhelpful because they line up tools that don't do the same job. So let's fix the frame first, then run a concrete scenario through the real contenders.

## The tools split into two jobs

Every tool people mention in this space falls into one of two buckets — and they are **complementary, not substitutes**:

- **Orchestration** — *run the DAG*: dependency order, caching, where results are stored. Airflow, Prefect, Dagster, Luigi, Kedro, Metaflow, Flyte, ZenML — and oryxflow.
- **Experiment management** — *record the runs*: params, metrics, and a searchable dashboard. MLflow, Weights & Biases, Neptune, Comet, DVC.

oryxflow lives in the first bucket. If you need the second, add MLflow or DVC *beside* it — the [MLflow-vs-caching post](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md) covers that pairing. This post is about the orchestration contest, because that's where the real overlap is.

## The scenario

A realistic, common shape for research work — and a demanding one for tooling:

- **Local and offline.** The source data sits behind API credentials; the analyst is on Windows. No cloud login should be required to run, or to *see results*.
- **Survives constant edits.** Re-spec a feature, rerun. Caching that goes stale on a code change — or that misses an edit to a *helper* function — is a daily hazard.
- **Parameterized fan-out.** The same DAG runs across a dozen cohorts; each cohort's outputs cached and retrievable independently.
- **Automatic artifact management.** Adding or changing a parameter should not mean hand-editing an output path so runs don't collide. The framework should key storage on (task, params) itself.
- **Never repeat an expensive call.** A cached query against a rate-limited API must survive across runs, so it's hit once, not every iteration.
- **AI-assisted, low ceremony.** One analyst, authoring with Claude Code. Minutes spent on schedulers, YAML catalogs, or metadata databases are minutes off the actual analysis.

This isn't every project. If yours is a scheduled production pipeline with retries and alerting, skip to ["where oryxflow is the wrong tool"](#where-oryxflow-is-the-wrong-tool) — the answer there is Airflow, Prefect, or Dagster, and it's not close. But for the *research loop* above, the field narrows fast.

## Two decisive tests

Most frameworks pass the easy checks (they run a DAG). Two questions separate them.

### Test 1 — edit a task's code, then rerun. What recomputes?

This is the daily reality of research: you change the logic and want exactly the affected work to rebuild. The honest breakdown:

- **Airflow, Prefect, Dagster** don't rerun on a pure in-function logic edit — they're not caching your code identity; they schedule and run tasks. (Prefect and Dagster have opt-in caching / asset versioning you configure; none tracks an edit to a helper function out of the box.)
- **Luigi** caches by output existence only — change the code, the output still exists, nothing reruns until you delete files by hand.
- **DVC** reruns a stage when a *declared file dependency* changes; an in-function logic edit that doesn't cross a declared boundary is invisible to it.
- **ZenML** caches at the step level and is the closest competitor here — but the cache key is the step's own code and inputs, so an edit to a *helper* the step calls isn't automatically caught.
- **oryxflow** tracks each task's own code **plus every helper it references** — so editing a helper function reruns exactly the tasks that use it, and everything downstream, automatically. Cosmetic edits (comments, formatting) never recompute because oryxflow compares what your code *does*, not how it's written, and the rerun reason even names the changed symbol: `code change (auto: features.py::build_features)`.

That helper-aware, per-symbol invalidation is the single capability the rest of the field doesn't have out of the box.

### Test 2 — where do the results go?

Change a parameter and two runs must not collide. Do you manage the output path, or does the tool?

- **Luigi, plain DVC** — you own every file path. You encode the parameter into the filename yourself, and a forgotten one silently overwrites.
- **oryxflow** — you never write a path. The task's base class decides the format (`TaskPqPandas` → parquet, `TaskPickle`, `TaskCSVPandas`, `TaskCache` → in-memory), and outputs are keyed on (task, params) automatically. `self.save(df)` and `self.inputLoad()` just work.

Here's the honest part, because it's where lazy comparisons overreach: **Dagster, Metaflow, and Kedro also manage persistence for you.** Dagster has IO managers, Metaflow auto-versions artifacts you assign to `self`, and Kedro has its Data Catalog. So the differentiator isn't "automatic I/O" — it's *how* automatic:

- **Dagster** — automatic, but the default IO manager pickles, and choosing parquet/CSV means configuring IO managers as resources.
- **Kedro** — automatic, but declared in a `catalog.yml` you maintain alongside the code.
- **Metaflow** — automatic, but into an opaque, Metaflow-owned datastore, not files you can open in DuckDB or pandas.
- **oryxflow** — the base class *is* the config. Zero YAML, zero IO-manager wiring, and the output is a real parquet/CSV file in a plain `data/` folder you can open with anything. Switch a task from parquet to CSV — or, in the Pro version, to a SQL table or cloud storage — by changing one base class; the task code is untouched.

Type-driven, zero-config serialization that stays open and portable — that's the precise claim, and it holds up.

## The orchestration matrix

Scored for the research-loop scenario above. "Automatic code-aware cache" means *reruns on an in-function logic edit, including edits to helpers, with no manual reset*.

| Framework    | Local, no signup | Native Windows    | Auto code-aware cache       | Auto artifact store        | Param fan-out | AI authoring assist   |
| ------------ | ---------------- | ----------------- | --------------------------- | -------------------------- | ------------- | --------------------- |
| **oryxflow** | ✅               | ✅                | ✅ symbol-level             | ✅ type-driven, open files | ✅            | ✅ Claude Code plugin |
| ZenML        | ✅               | ⚠️ long-path snag | ⚠️ step-level only          | ✅                         | ✅            | ❌                    |
| Luigi        | ✅               | ✅                | ❌                          | ❌ you write paths         | ⚠️ manual     | ❌                    |
| Kedro        | ✅               | ✅                | ❌                          | ✅ via `catalog.yml`       | ✅            | ❌                    |
| Metaflow     | ⚠️ AWS-oriented  | ⚠️                | ⚠️ versions, not code-aware | ✅ opaque datastore        | ✅            | ❌                    |
| Prefect      | ✅               | ✅                | ⚠️ opt-in caching           | ⚠️ configured              | ✅            | ❌                    |
| Dagster      | ⚠️ daemon/UI     | ✅                | ⚠️ asset versioning         | ✅ IO managers             | ✅            | ❌                    |
| Airflow      | ❌ server + DB   | ⚠️                | ❌                          | ❌ XCom is small-data only | ⚠️            | ❌                    |
| Flyte        | ❌ Kubernetes    | ❌                | ⚠️                          | ✅                         | ✅            | ❌                    |

The pattern: **Airflow, Prefect, Dagster, and Flyte are built for scheduled or distributed production pipelines.** That's real infrastructure and real value — just more than a solo research loop needs, and not what makes the research loop fast. **ZenML is the credible alternative** if you want an automatic-caching framework and can live with the step-level (not helper-aware) hash and the Windows long-path snag. Everything else asks you to hand-manage either the cache, the paths, or both.

## What oryxflow looks like

The whole scenario, in the code you'd actually keep:

```
import oryxflow
import pandas as pd

class GetSource(oryxflow.tasks.TaskPqPandas):
    """The expensive, rate-limited call. Cached once — never repeated on rerun."""
    def run(self):
        self.save(fetch_from_api())          # no path to manage; parquet by base class

@oryxflow.requires(GetSource)
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    cohort = oryxflow.Parameter()            # fan-out key
    def run(self):
        df = self.inputLoad()                # GetSource's output, already loaded
        self.save(engineer(df, self.cohort))

# one flow per cohort; GetSource runs once and is shared across all of them
flow = oryxflow.WorkflowMulti(BuildFeatures, params={'cohort': COHORTS})
flow.run()
features = flow.outputLoadConcat(BuildFeatures)   # every cohort, one tagged frame
```

Edit `engineer()` — a helper, not even a task — and the next run recomputes every `BuildFeatures` and anything downstream, while the expensive `GetSource` stays cached. No reset, no version bump, no path bookkeeping. And because oryxflow ships a [Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md), an AI agent authoring this pipeline verifies its own edits actually reran and never trusts a stale cache.

## Where oryxflow is the wrong tool

Being honest about fit is the point of a comparison:

- **You need scheduled production pipelines** with retries, alerting, and backfills → Airflow, Prefect, or Dagster. Different job; use them.
- **You need distributed / Kubernetes-scale execution** → Flyte or Metaflow (on Linux/WSL).
- **You want an automatic-caching framework and can accept a step-level hash** → ZenML is the credible alternative (mind the Windows long-path snag).
- **Your core need is experiment tracking or data versioning** → add MLflow or DVC *beside* oryxflow, not instead of it.

## The takeaway

For a solo analyst or small team iterating on research code all day — locally, on Windows, often with an AI coding agent — the deciding features are **automatic code-aware caching that follows helper edits** and **automatic, open artifact storage so parameters never mean hand-built paths**. That combination, local and zero-infrastructure, is where oryxflow wins the orchestration core. The orchestrators win production; the trackers win dashboards; oryxflow wins the research loop.

```
pip install oryxflow
```

## Frequently asked questions

### Which workflow tool should I use for local data science — Airflow, Prefect, Dagster, or something lighter?

Airflow, Prefect, and Dagster are built for scheduled or distributed production pipelines — real infrastructure that a solo research loop doesn't need. For iterating on analysis locally all day, a lighter code-aware cache fits better. oryxflow gives you automatic code-aware caching that follows helper edits and zero-config artifact storage, local and pip-installable, then hands the stable pipeline to an orchestrator when it's production.

### Airflow vs Prefect vs Dagster for data science — how do they differ?

For data science, Airflow is a heavyweight scheduler needing a server and metadata database; Prefect is a Python-native orchestrator with opt-in caching you configure; Dagster is an asset platform with a UI and IO managers. All three win production scheduling. None caches your code identity so an edit reruns only affected tasks — that's the research-loop gap oryxflow fills locally.

- **[Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)** — the positioning in full.
- **[oryxflow vs Airflow](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-airflow/index.md)** — research workflows vs production orchestration.
- **[MLflow or pipeline caching?](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md)** — the experiment-tracking half.
- Source & examples: <https://github.com/oryxintel/oryxflow>

# Parameter sweeps in Python without rerunning upstream steps

*Sweep a big grid cheaply, and trust the comparison because every config trained on the same cached inputs.*

## The problem: the grid multiplies the wrong work

Comparing models or hyperparameters means running a Cartesian product of configurations: three feature sets times four regularization strengths times two model families is twenty-four runs. That part is unavoidable — you asked for twenty-four answers.

What is avoidable is the work each run *repeats*. Naively, every configuration reloads the same raw data, recomputes the same features, and re-splits the same train/test folds before it gets to the one step that actually differs. If the data load takes two minutes and the feature step takes five, you have paid seven minutes twenty-four times to change a single number.

The usual workaround is worse than the problem. You start hand-managing output filenames — `model_ols_alpha01.pkl`, `model_gbm_alpha10.pkl` — and now you are one typo away from a collision that silently overwrites results, or from comparing two models that were quietly trained on different snapshots of the data. The bookkeeping becomes the experiment.

## What oryxflow is

oryxflow is a small Python library for building data-science workflows as `Task` classes that declare their dependencies and cache their outputs, so a step reruns only when its inputs or parameters actually change.

That caching model is exactly what a parameter sweep needs. Below are the two mechanisms it gives you, then the fan-out pattern that ties them into a grid.

## Mechanism 1: a parameter caches a separate output automatically

Add a parameter to a task and oryxflow keys the cache on its value. Each parameter value gets its own cached output — no filenames to invent, no collisions to worry about.

```
import oryxflow
import pandas as pd

class GetData(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = pd.read_csv('raw.csv')
        self.save(df)

@oryxflow.requires(GetData)
class TrainModel(oryxflow.tasks.TaskPickle):
    model = oryxflow.Parameter(default='ols')

    def run(self):
        df = self.inputLoad()
        estimator = fit(df, kind=self.model)
        score = evaluate(estimator, df)
        self.save(estimator)
        self.saveMeta({'score': score})
```

Run one configuration:

```
flow = oryxflow.Workflow(TrainModel, {'model': 'gbm'})
flow.run()
```

`GetData` runs once and lands in the cache. Run `TrainModel` again with `model='ols'` and `GetData` is *not* recomputed — oryxflow sees its output already exists and reuses it. Changing a parameter reruns only the tasks that depend on that parameter. The `saveMeta` call stores a small metric sidecar next to the model so you can pull scores back without unpickling every estimator.

## Mechanism 2: WorkflowMulti compares named configurations

When you want to line up a handful of named configurations side by side, `WorkflowMulti` runs each as its own flow while sharing the upstream:

```
flow = oryxflow.WorkflowMulti(TrainModel, {
    'ols': {'model': 'ols'},
    'gbm': {'model': 'gbm'},
})
flow.run()

print(flow.outputLoadMeta())        # {'ols': {'score': ...}, 'gbm': {'score': ...}}
models = flow.outputLoad(TrainModel) # {'ols': <model>, 'gbm': <model>}
```

The shared upstream — `GetData` — computes **once** and is reused across every flow. `outputLoadMeta()` collects the metric sidecars into one dict keyed by flow name, so a one-liner answers "which config won?" `outputLoad()` hands back the fitted models the same way.

## Mechanism 3: fan-out and aggregate for a full grid

For a sweep over many values, have one task fan out into one child task per value using a dict-`requires()`, then aggregate the results by stacking them:

```
ALPHAS = [0.01, 0.1, 1.0, 10.0]

@oryxflow.requires(LoadData)
class TrainModel(oryxflow.tasks.TaskCachePandas):
    alpha = oryxflow.FloatParameter()

    def run(self):
        df = self.inputLoad()
        score = fit_and_score(df, alpha=self.alpha)
        self.save(pd.DataFrame({'alpha': [self.alpha], 'score': [score]}))

class Tune(oryxflow.tasks.TaskCachePandas):
    def requires(self):
        return {a: TrainModel(alpha=a) for a in ALPHAS}

    def run(self):
        df = self.inputLoadConcat()   # one row per alpha; 'alpha' column tags each
        self.save(df.sort_values('score', ascending=False))
```

`Tune` depends on one `TrainModel` per alpha. Each child caches its own one-row result. `inputLoadConcat()` row-stacks all of them into a single DataFrame, tagging each row with the parameters that produced it — so `Tune` ends up as a tidy leaderboard sorted by score. `LoadData` still runs once and feeds every child.

Run it and read off the winner:

```
flow = oryxflow.Workflow(Tune)
flow.run()
leaderboard = flow.outputLoad()   # every alpha's score, best first
```

## Extend the grid, nothing wasted

Add two more values to the sweep:

```
ALPHAS = [0.01, 0.1, 1.0, 10.0, 100.0, 1000.0]
```

Rerun `Tune`. The four alphas you already trained are still cached, so only the two new `TrainModel` tasks run. `LoadData` is untouched. The expensive shared step never re-executes just because the grid grew.

If you deliberately want to rerun the whole model family — say you changed the training code but not the data — you can invalidate just that band:

```
flow.reset_upstream(Tune, only=TrainModel)
```

That reruns every `TrainModel` on the next `run()` while keeping `LoadData` cached. You re-fit the models, not the data.

## Why the comparison is trustworthy

Because the shared upstream is computed once and every configuration reads that same cached output, all twenty-four runs — or two, or two hundred — trained on byte-for-byte identical inputs. There is no chance that `ols` saw a slightly different split than `gbm`, because there is only one split to see. The comparison is apples-to-apples by construction, not by discipline.

And it is reproducible: the cache key is derived from parameters and upstream outputs, so rerunning the sweep on the same data reuses the same results rather than silently drifting. Delete `data/` and rebuild, and you get the same grid back.

One honest caveat: oryxflow is a research-loop tool for iterating quickly on your own machine, not a production orchestrator with scheduling, retries, or distributed workers. It is built to make the *next experiment* cheap, and the sequential engine runs the DAG in-process. For a hardened production pipeline, reach for a full orchestration system.

## Takeaway

A parameter sweep should cost you the model fits and nothing else. Put the varying choice on a parameter, let oryxflow cache one output per value, and share the expensive upstream across the whole grid. You get a cheap sweep, a trustworthy comparison, and no filename bookkeeping.

```
pip install oryxflow
```

## Frequently asked questions

### How do I run a parameter sweep without rerunning the upstream steps every time?

Put the varying choice on a task parameter and let the engine key a separate cached output per value, while the shared upstream steps compute once and are reused across the whole grid. oryxflow does exactly this: changing a parameter reruns only the tasks that depend on it, so a sweep costs you the model fits and not the repeated data loading and feature building.

### How do I share cached features across a grid of model configs?

Compute the features once as an upstream task, then have each model configuration depend on it so every config reads the same cached output instead of rebuilding it. oryxflow shares that upstream across every flow in a WorkflowMulti or fan-out grid, so all configs train on byte-for-byte identical inputs and only the differing model steps run — a trustworthy comparison with no filename bookkeeping.

Then keep going:

- [Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md)
- [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md)
- [Quickstart](https://docs.oryxflow.dev/docs/quickstart/index.md)
- Sibling post: [Stop rerunning your pipeline](https://docs.oryxflow.dev/blog/caching/cache-intermediate-dataframes-python/index.md)
- [oryxflow on GitHub](https://github.com/oryxintel/oryxflow)

# Reproducible data science workflows in Python

*Reproducibility and lineage are the product. Caching is just how you get them for free.*

A workflow is reproducible when you can say exactly which code and which inputs produced a given result — and regenerate it on demand. That's the whole definition, and it's a surprisingly high bar. Most analysis pipelines fail it not because anyone was careless, but because the tools we reach for — notebooks and loose scripts — have no memory of what produced what.

This post is about closing that gap in plain Python: what actually makes a data-science workflow reproducible, why workflows quietly lose reproducibility in the first place, and how a small library called [oryxflow](https://github.com/oryxintel/oryxflow) gets you there without a server, a database, or a YAML file. If you want the canonical positioning — where oryxflow sits relative to notebooks and orchestrators — read [why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md). This is the keyword-first explainer.

## Why workflows lose reproducibility

Nobody sets out to build a pipeline they can't reproduce. It happens by accretion:

- **Hidden notebook state.** A variable exists only because you ran a cell you've since edited or deleted. The notebook still runs top-to-bottom in your head, but not on disk.
- **Out-of-order execution.** Cells run 1, 2, 4, 3, 2-again. The result on screen reflects a path through the code that no fresh "Run All" will reproduce.
- **Stale intermediates.** You saved `features.parquet` yesterday, changed the feature code today, and trained on the old file without noticing. Nothing errored. The number is just quietly wrong.
- **No durable link between artifact and code.** You have `model.pkl` and you have your script, but nothing records that *this* model came from *that* commit of the feature code and *those* parameters. Six weeks later, you can't answer the question that matters: *which version made this?*

None of these are exotic. They're the default failure mode of exploratory work, and they all share a root cause: the saved artifacts and the code that made them live in separate worlds, with no enforced connection between them.

## What makes a workflow reproducible

Reproducibility isn't one feature. It's three properties working together.

1. **Deterministic task identity from code + params.** Every step has an identity computed from its code and its parameters. Same code, same inputs → same identity → the same output, every time. Change either and you get a new identity — so a result is never silently attributed to code that didn't produce it.
1. **Automatic invalidation.** You should not be able to evaluate new code against a stale output. When a step's logic changes, that step and everything downstream of it must be recomputed — automatically, without you remembering to delete a cache file. Reproducibility that depends on your discipline isn't reproducibility.
1. **A lineage record.** A durable, append-only log of what ran, when, and why. Not because it's tidy, but because "regenerate this result" requires knowing what produced it in the first place.

oryxflow is built around exactly these three. You declare each step of your analysis as a `Task` with its parameters and its dependencies; the library runs them in dependency order, caches each output, and reruns a step only when its code or inputs actually change. The identity is deterministic, the invalidation is automatic, and every run appends to a lineage file. You don't maintain reproducibility — you get it as a side effect of writing tasks.

### Deterministic identity, and automatic code-change invalidation

The part that's easy to underrate: oryxflow watches your **code**, not just your parameters. It tracks each task's logic plus the helper files it imports, comparing what your code *does*, not how it's written. Edit the body of `BuildFeatures` — or a helper function it calls — and oryxflow knows that task's output is stale and reruns it and everything downstream. Reformat the code, add a comment, rename a local variable? Nothing reruns — those don't change what the code does. And an expensive upstream step whose code you didn't touch stays cached.

That's the property that makes the research loop both fast and trustworthy: you can never test new logic against an output the old logic produced, and you never pay to recompute a step that didn't change.

## Building a small reproducible DAG

Here's the canonical shape — `GetData` → `BuildFeatures` → `TrainModel` — with nothing but the verified API. Each task's base class picks its serialization by type: `TaskPqPandas` saves a DataFrame as parquet, `TaskPickle` saves an arbitrary Python object. No paths, no `to_parquet` calls, no config.

```
import oryxflow


class GetData(oryxflow.tasks.TaskPqPandas):        # saves a DataFrame as parquet
    source = oryxflow.Parameter(default='raw.parquet')

    def run(self):
        df = load_table(self.source)               # your loader
        self.save(df)


@oryxflow.requires(GetData)                        # declares the dep AND copies params
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                      # the upstream DataFrame
        df['x_squared'] = df['x'] ** 2
        self.save(df[['x', 'x_squared', 'y']])


@oryxflow.requires(BuildFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):       # saves any Python object
    alpha = oryxflow.FloatParameter(default=1.0)

    def run(self):
        df = self.inputLoad()
        model = fit_model(df[['x', 'x_squared']], df['y'], alpha=self.alpha)
        self.save(model)
        self.saveMeta({'n_rows': len(df), 'alpha': self.alpha})
```

Run it through a `Workflow`, which resolves the DAG and runs only what's needed:

```
flow = oryxflow.Workflow(TrainModel, {'alpha': 0.5})
flow.preview()                                     # show the plan without running
flow.run()                                         # runs GetData -> BuildFeatures -> TrainModel

model = flow.outputLoad()                           # the trained model
meta = flow.outputLoadMeta()                        # {'n_rows': ..., 'alpha': 0.5}
```

The first run computes all three steps. Edit `TrainModel.run` and rerun: `GetData` and `BuildFeatures` stay cached, only `TrainModel` recomputes. Change `alpha` and you get a distinct identity — the old and new models coexist, each tied to its parameters. Nothing here writes a path or checks whether a file already exists; that bookkeeping is the library's job. For the fuller pattern — parameter sweeps, resetting, sharing flows — see [managing workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md).

## When to use it — and when not to

Reach for a reproducible DAG when a result needs to be defensible: something you'll hand to a colleague, revisit in a month, or make a decision on. The moment a workflow has more than one expensive step and you're iterating on the later ones, deterministic identity and automatic caching pay for themselves.

A first look doesn't need tasks — and it doesn't need a decision either. If you're eyeballing distributions and nothing downstream depends on the output, task boilerplate is pure overhead; write a plain script. But you don't have to choose a tool before you know how big the work will get: if you build with the [oryxflow Claude Code plugin](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md), that first look already has a home in the project — a read-only probe that states the question it answers and still runs next session — and `/oryxflow:migrate` lifts it into cached, parameterized tasks the day something starts depending on its result. So you can start a small exploration here and let it scale as the shape stabilizes, with no rewrite in between. Where oryxflow genuinely doesn't fit — scheduling, distributed scale, dashboards — has a whole post: [when not to use oryxflow](https://docs.oryxflow.dev/blog/guides/when-not-to-use-oryxflow/index.md).

## Where oryxflow sits

Reproducibility isn't binary, and oryxflow isn't the only layer that touches it. It occupies the missing middle between a notebook and a production orchestrator.

|                            | Notebook / loose script     | oryxflow                                | Orchestrator (Airflow / Prefect / Dagster) |
| -------------------------- | --------------------------- | --------------------------------------- | ------------------------------------------ |
| Reproducible task identity | ❌ manual, by convention    | ✅ deterministic, from code + params    | ⚠️ run versioning, not code-aware          |
| Caching                    | ❌ you manage files by hand | ✅ automatic, code-change aware         | ⚠️ opt-in / configured                     |
| Lineage record             | ❌ none                     | ✅ append-only `.oryxflow/events.jsonl` | ✅ run history in a DB / UI                |
| Infrastructure             | ✅ none                     | ✅ none — local files only              | ❌ server, DB, scheduler                   |
| Job it's built for         | Exploration                 | The trustworthy research loop           | Scheduled production pipelines             |

Orchestrators are a **complementary** layer, not a competitor: they schedule and distribute pipelines in production, which is a real and different problem. Experiment trackers like MLflow and W&B are complementary too — they *record* runs; oryxflow *structures and caches* them. oryxflow's differentiators are the combination that neither layer offers: type-driven zero-config I/O, automatic code-change invalidation, deterministic task identity, and a local-first design with native Python identity — no YAML, no server, no account, no telemetry. There's a full comparison in [oryxflow vs. the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md).

## The honest caveat: reproducible ≠ correct

Be clear about what reproducibility buys you. oryxflow guarantees that an output was produced by the exact code and inputs it recorded — no more. It does **not** guarantee that the output is *right*. A pipeline with a bug in its feature logic is reproduced just as faithfully as a correct one; you'll get the same wrong number every time, tied cleanly to the flawed code that made it.

That's not a weakness — it's the honest scope. Reproducibility is what makes a wrong result *debuggable*: because you know exactly which code and inputs produced it, you can find the bug, fix it, and let automatic invalidation rerun precisely what the fix touched. Correctness is still your job. Reproducibility is what makes doing that job tractable.

## Takeaway

A workflow is reproducible when you can name the exact code and inputs behind any result and regenerate it. That takes three things — deterministic task identity, automatic invalidation so you can't test new code on stale outputs, and a durable lineage record. oryxflow gives you all three in plain Python, locally, with the caching that makes them cheap thrown in for free. It won't make your pipeline correct. It will make it something you can trust, hand off, and reproduce.

```
pip install oryxflow
```

## Frequently asked questions

### How do I make a data science workflow reproducible in Python?

A workflow is reproducible when you can name the exact code and inputs behind any result and regenerate it on demand. That takes three things: deterministic task identity from code and parameters, automatic invalidation so you can't test new code against a stale output, and a durable lineage record. oryxflow gives you all three in plain Python, locally, with caching that makes them cheap thrown in for free.

### What makes a pipeline reproducible without a lot of infrastructure?

Reproducibility needs a durable link between each result and the code and inputs that produced it, not a server or database. oryxflow delivers that locally: it computes a deterministic identity for each step, reruns a step automatically when its code changes, and appends every run to a local lineage log at .oryxflow/events.jsonl. No server, no database, no account — just pip install and local files.

**Read next:** [From notebook to a reproducible pipeline](https://docs.oryxflow.dev/blog/reproducibility/notebook-to-reproducible-pipeline/index.md) · [Stop rerunning your whole pipeline](https://docs.oryxflow.dev/blog/caching/cache-intermediate-dataframes-python/index.md) · [When not to use oryxflow](https://docs.oryxflow.dev/blog/guides/when-not-to-use-oryxflow/index.md) · [oryxflow vs. the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md) · [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md) · [Managing workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md) · [Build with the Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md)

# Stop rerunning your whole pipeline: caching intermediary DataFrames in Python

*How data scientists waste hours recomputing steps that never changed — and a lightweight way to fix it.*

## The problem every data science script eventually has

Almost every analysis starts as a linear script:

```
df = load_data()              # 40 seconds
df = clean(df)                # 2 minutes
features = build_features(df) # 5 minutes
model = train(features)       # 8 minutes
evaluate(model, features)
```

It works. Then you tweak `evaluate()` — a one-line change — and to see the result you rerun the file and wait **fifteen minutes** while `load_data`, `clean`, and `build_features` recompute output that is byte-for-byte identical to last time.

So you start hand-rolling caches:

```
if os.path.exists('features.pkl'):
    features = pd.read_pickle('features.pkl')
else:
    features = build_features(df)
    features.to_pickle('features.pkl')
```

Now multiply that by every step, every parameter, and every teammate who doesn't know which `.pkl` is stale. You're manually tracking filenames, manually invalidating caches, and quietly training models on yesterday's data. This is the single most common way machine learning code rots.

## The fix: make each step a task, let the engine cache

The clean solution is to stop thinking in *lines of a script* and start thinking in *tasks with dependencies* — a DAG. Each task declares what it needs, what it produces, and where its output is stored. The engine then:

- runs steps in dependency order,
- **skips any step whose output already exists**,
- and reruns **only** the steps affected by a code, data, or parameter change.

[`oryxflow`](https://github.com/oryxintel/oryxflow) is a small, dependency-free Python library that does exactly this. Here's the pipeline above, rewritten:

```
import oryxflow
import pandas as pd

class GetData(oryxflow.tasks.TaskPqPandas):   # output persisted as Parquet
    def run(self):
        df = load_data()
        self.save(df)                          # no filename to manage

@oryxflow.requires(GetData)                    # declares the dependency
class BuildFeatures(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()                  # loads GetData's output
        self.save(build_features(df))

@oryxflow.requires(BuildFeatures)
class TrainModel(oryxflow.tasks.TaskPickle):   # output persisted as pickle
    def run(self):
        features = self.inputLoad()
        model = train(features)
        self.save(model)
        self.saveMeta({'score': model.score(...)})

oryxflow.run(TrainModel())
```

Run it once and all three tasks execute. Run it again and you get:

```
Scheduled 3 tasks
* 0 ran successfully
* 3 complete          <-- nothing recomputed, output loaded from disk
* 0 failed
```

Change `train()` and only `TrainModel` reruns — `GetData` and `BuildFeatures` are served from cache. **The fifteen-minute edit-rerun loop becomes eight minutes, then eight seconds.** You never wrote a single `if os.path.exists(...)`.

## The part that actually saves you: parameter-aware invalidation

The real payoff shows up when you compare models. Add a parameter and oryxflow tracks a separate cached output per parameter value automatically:

```
@oryxflow.requires(GetData)
class TrainModel(oryxflow.tasks.TaskPickle):
    model = oryxflow.Parameter(default='ols')  # a knob you'll sweep

    def run(self):
        features = self.inputLoad()
        clf = LinearRegression() if self.model == 'ols' else GradientBoostingRegressor()
        clf.fit(features.drop('y', axis=1), features['y'])
        self.save(clf)
        self.saveMeta({'score': clf.score(features.drop('y', axis=1), features['y'])})

flow = oryxflow.WorkflowMulti(TrainModel, {
    'ols': {'model': 'ols'},
    'gbm': {'model': 'gbm'},
})
flow.run()
print(flow.outputLoadMeta())
# {'ols': {'score': 0.74}, 'gbm': {'score': 0.97}}
```

Training the second model **does not** rerun `GetData` or `BuildFeatures` — they're shared and already cached. oryxflow figures out the minimal set of work for each configuration. That's the difference between "sweep five hyperparameters over coffee" and "sweep five hyperparameters over lunch."

## Where this fits (and where it doesn't)

oryxflow is a **research-iteration** tool. Reach for it when your day is EDA → feature engineering → train → evaluate and you're tired of babysitting intermediate files. It works with any ML library — sklearn, PyTorch, XGBoost — because it only cares about task inputs and outputs, not what happens inside `run()`.

It is **not** a production orchestrator. If you need cron-style scheduling, retries across a cluster, and SLAs, use Airflow, Prefect, or Dagster. And it's **complementary** to experiment trackers: keep logging metrics to MLflow or Weights & Biases inside your tasks — oryxflow handles the caching and rerun logic those tools don't.

## Try it

```
pip install oryxflow
```

- Docs: https://docs.oryxflow.dev
- Source & examples: https://github.com/oryxintel/oryxflow

The next time you change one line and reach for the run button, you shouldn't have to recompute everything upstream of it. Let the DAG remember what's already done.

## Frequently asked questions

### How do I stop rerunning my whole pipeline when I change one step?

When you edit one step, you want only that step and its downstream to recompute, not the whole script. Model each step as a task with declared dependencies so an engine can run them in dependency order and skip anything already computed. oryxflow does this: it caches each task's output and reruns only the steps a code, data, or parameter change actually affects, so a one-line edit stops triggering a fifteen-minute recompute.

### How do I make Python reruns only recompute what changed?

Turn your script into tasks that declare what they depend on and what they produce, then let an engine track each task's identity from its code and parameters. When something changes, only the affected tasks are stale. oryxflow, a small local-first Python library, caches every task output and reruns only what changed on the next run, so unchanged upstream steps load from disk instead of recomputing.

# When not to use oryxflow

*Being clear about where a tool doesn't fit is part of being trustworthy — so here's where oryxflow is the wrong choice.*

oryxflow is a local, zero-infrastructure library for the research loop: it caches task outputs, tracks lineage, and skips work whose inputs and code haven't changed, so your pipeline stays reproducible without a server, a scheduler, or a database. That's genuinely useful — but only for a specific shape of problem. Every honest tool has a boundary, and pretending oryxflow fits everywhere would waste your time and cost you trust. So here is where it doesn't fit, and what to reach for instead.

## A quick one-off doesn't need a pipeline — but you can still start here

If your whole analysis is "load a CSV, group by a column, plot one thing," you don't need task classes around it. Five lines you'll run once get no payoff from caching, and wrapping them in a DAG is ceremony.

The value of a caching DAG rises with three things: **depth** (how many dependent steps), **cost** (how expensive each step is), and **breadth** (how many parameter combinations you sweep). For a five-line notebook cell, all three are near zero, so the return is near zero too.

**Use instead:** a plain script for the first pass — but keep it *inside the project*, not in a scratch folder you'll abandon. If you build with the [oryxflow Claude Code plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md), that's already the convention: exploration lives as a read-only probe at `eda/<subject>/<name>.py`, one line of docstring stating the question it answers, printing the answer legibly, run with `python -m eda.<subject>.<name>`. A probe writes no pipeline artifact — disposable scratch goes to a gitignored scratch area — and anything material it turns up gets written into the project's data doc, so a question you've answered once isn't re-asked next session.

Then, when a script turns out to be load-bearing — you keep re-running it, something depends on its output, or you start sweeping it over parameters — you don't rewrite it by hand. `/oryxflow:migrate` reads the script as the spec, shows you a step-to-task map, writes only on your approval, and never deletes the source. So there's no cliff between "exploring" and "having a pipeline": **you can start with a small EDA and let it scale to any complexity, with no rewrite in between.** More on both ends: [migrate a notebook to a pipeline](https://docs.oryxflow.dev/docs/migrate-notebook-to-pipeline/index.md) and [Claude Code for data science](https://docs.oryxflow.dev/docs/claude-code-for-data-science/index.md).

## Don't reach for oryxflow when… you need production orchestration

Scheduled runs at 6am, retries across a cluster, backfills over a date range, alerting when a job fails, SLAs your team is on the hook for — that's production operations, and oryxflow doesn't do it. There's no scheduler, no distributed retry, no alerting, and no operational UI.

**Use instead:** [Airflow](https://airflow.apache.org/), [Prefect](https://www.prefect.io/), or [Dagster](https://dagster.io/). These are excellent at what they do, and they do a *different* job than oryxflow — they orchestrate operations, oryxflow accelerates the research loop that happens before anything is scheduled. Many teams develop logic in oryxflow and later wrap the finished pipeline in one of these for production. They're complementary, not competitors.

## Don't reach for oryxflow when… you need distributed or very-large-scale execution

If a single step needs a Kubernetes cluster, or your data doesn't fit on one machine and you need engine-level parallelism, the OSS core isn't built for that. It's local-first and runs in-process.

**Use instead:** [Flyte](https://flyte.org/) or [Metaflow](https://metaflow.org/) on Linux or WSL. (oryxflow's paid Pro tier adds SQL, cloud storage, Dask, and PySpark backends — but the open-source core is deliberately local-first, and that's the right lens for evaluating fit here.)

## Don't reach for oryxflow when… you want an experiment dashboard

If what you need is a searchable web UI showing every run's metrics, params, and charts side by side — sortable, filterable, shareable with your team — oryxflow doesn't provide it. It gives you a queryable lineage log, not a hosted dashboard.

**Use instead:** [MLflow](https://mlflow.org/) or [Weights & Biases](https://wandb.ai/). And note these *compose* with oryxflow: log your metrics to MLflow from inside a task's `run()`, and let oryxflow handle the caching and reproducibility around it. You don't pick one — you use both, each for its strength. (More on that split in [MLflow or pipeline caching](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md).)

## Don't reach for oryxflow when… you need Git-tied data versioning

If your goal is versioning large data artifacts alongside your code, pinned to Git commits and pushed to remote storage, that's a different discipline.

**Use instead:** [DVC](https://dvc.org/). It also composes with oryxflow — DVC for artifact versioning, oryxflow for the compute graph on top.

## The honest caveat: oryxflow does not check that your result is *correct*

This is the one worth reading twice. oryxflow guarantees that an output was produced by the exact code and inputs it recorded — it makes your pipeline *reproducible*. It does **not** guarantee the result is *right*.

It will happily cache, with full lineage:

- a join that silently went many-to-many when it should have been many-to-one,
- a test set that leaked into training,
- a ratio computed against the wrong denominator,
- a timestamp shifted by a timezone you forgot to normalize,
- a backtest that peeks at data from the future.

Every one of those is reproducible, lineage-tracked, and wrong. oryxflow manages pipeline *mechanics*; it has no opinion about statistical *judgment*. Those bugs are caught by habit — sanity checks, held-out validation, reading your own numbers skeptically — not by any caching machinery. If a post ever tells you a workflow tool makes your analysis correct, close the tab. For where this boundary lives, see [what caching does not protect against](https://docs.oryxflow.dev/docs/managing-workflows/index.md).

## Where it *is* the right tool

With the boundaries drawn honestly, the fit is clear. oryxflow earns its keep when:

- your pipeline is a **deep chain** of dependent steps,
- some of those steps are **expensive** (minutes to hours),
- you sweep a **matrix of parameters** and want to re-run only what changed,
- you need to **reproduce or hand off** research months later, and
- pipelines are **authored by AI agents** that benefit from an explicit, inspectable task graph.

A typical task is small and declarative — declare dependencies, load inputs, save outputs, and the engine skips anything already computed:

```
@oryxflow.requires(CleanData)
class Features(oryxflow.tasks.TaskPqPandas):
    def run(self):
        df = self.inputLoad()
        self.save(add_features(df))

flow = oryxflow.Workflow(task=Features)
flow.run()
```

That's the sweet spot: enough depth and cost that caching pays for itself, run often enough that reproducibility matters.

## Takeaway

Start a quick exploration as a plain script — inside an oryxflow project, where `/oryxflow:migrate` can promote it the day it earns a pipeline. Use Airflow, Prefect, or Dagster for production ops. Use Flyte or Metaflow for distributed scale. Use MLflow or W&B for dashboards, DVC for data versioning — and compose them with oryxflow where it helps. And never expect any of them, oryxflow included, to check your statistics for you.

Being honest about fit is the whole point: reach for oryxflow when you have a research pipeline — or the first small script that might become one — and reach for something else when the job is scheduling, scaling out, or display.

```
pip install oryxflow
```

## Frequently asked questions

### When should I not use oryxflow?

Skip oryxflow for production orchestration (scheduling, retries, alerting — use Airflow, Prefect, or Dagster), for distributed or larger-than-memory execution (Flyte or Metaflow), for a hosted experiment dashboard (MLflow or W&B), and for Git-tied data versioning (DVC). oryxflow is a local, zero-infrastructure library for the research loop; it caches task outputs and skips unchanged work, but it doesn't schedule, scale out, or display.

### Is oryxflow overkill for a quick one-off analysis?

No — a five-line cell you run once doesn't need task classes, and you don't have to decide upfront either way. If you build with the oryxflow Claude Code plugin, exploration gets a home in the project from the start: a read-only probe script that states the question it answers and still runs next session, instead of a snippet you lose. When a probe turns out to be load-bearing — you keep re-running it, or something downstream depends on its result — /oryxflow:migrate lifts it into cached, parameterized tasks and never deletes the original. So you can start small in oryxflow and scale to any complexity, with no rewrite in between.

### Does oryxflow check that my analysis is correct?

No — and no workflow tool does. oryxflow guarantees an output was produced by the exact code and inputs it recorded, which makes your pipeline reproducible, not correct. It will happily cache, with full lineage, a leaked test set, a bad join, or a backtest that peeks at the future. Those bugs are caught by sanity checks, held-out validation, and reading your own numbers skeptically — not by any caching machinery.

**Read next:** [Why oryxflow](https://docs.oryxflow.dev/docs/why-oryxflow/index.md) · [Managing complex workflows](https://docs.oryxflow.dev/docs/managing-workflows/index.md) · [oryxflow vs the field](https://docs.oryxflow.dev/blog/comparisons/oryxflow-vs-pipeline-frameworks/index.md) · [MLflow or pipeline caching](https://docs.oryxflow.dev/blog/mlops/mlflow-vs-pipeline-caching/index.md) · [Claude plugin](https://docs.oryxflow.dev/docs/claude-plugin/index.md) · [GitHub](https://github.com/oryxintel/oryxflow)