sequences of analysis operations

[jeremymanning]

we may want a way to support sequences of operations. for example, could we come up with an elegant way of supporting something like:

• reduce data to n1 dimensions using algorithm 1
• align data using hyperalignment
• reduce the aligned data to n2 dimensions using algorithm 2
• align the result using SRM
• project the (twice-)aligned data onto 3d using algorithm 3

if we add a preprocess function, we may want to apply multiple preprocessing steps in sequence (e.g. normalize, than binarize, etc.).

[jeremymanning]

another thought is to support a convenient syntax for something like:

hyp.plot(hyp.tools.reduce(x, model='TSNE', ndims=10), model='SpectralEmbedding')

Instead, we could support the following syntax:

hyp.plot(x, model=['TSNE', 'SpectralEmbedding'], ndims=[10, 3])

[jeremymanning]

Also noting (per discussion with @andrewheusser), we could consider supporting something like

hyp.plot(x, xform=[{'reduce': 'TSNE', 'params': {...}}, {'normalize': 'across'}, {'reduce': 'IncrementalPCA'}, {'normalize': 'within'}, {'align': 'hyperalignment'}, {'reduce': 'SpectralEmbedding'}])

In other words, allow an xform keyword argument that takes a list of dictionaries, each specifying a transformation to apply to the data (in sequence).

We'd also have to modify the DataGeometry.transform function to support this.

Upside: it's more flexible and allows the user to do whatever crazy stuff they want (argues for a democratic approach?)
Downside: it's more flexible and allows the user to do crazy stuff, even if it makes no sense (argues for a fascist approach?)

[jeremymanning]

Current status (1.0 update, PR #272): still not addressed. hypertools/tools/analyze.py is a fixed one-shot pipeline: aligner(reducer(normalizer(data, ...), ...), align=align) — exactly one normalize, one reduce, one align, in that order. No xform= keyword exists on analyze() or plot() (grepped hypertools/plot/plot.py for xform — only an internal local buffer variable of that name, not a public kwarg).

How this could be built on the 1.0 architecture:
The codebase already has a recent, relevant precedent: the manip dispatcher (hypertools/manip/manip.py) — a Manipulator base class + MANIPULATORS registry + unpack_model() resolver, wrapped in dw.decorate.funnel. The same shape generalizes cleanly to a sequence-of-steps feature:

STEPS = {'normalize': normalizer, 'reduce': reducer, 'align': aligner,
         'cluster': clusterer, 'manip': manip}

def analyze(data, xform=None, normalize=None, reduce=None, ndims=None, align=None, ...):
    if xform is not None:
        for step in xform:
            key, spec = next(iter(step.items()))
            data = STEPS[key](data, **(spec if isinstance(spec, dict) else {'model': spec}))
        return data
    return aligner(reducer(normalizer(data, ...), ...), align=align)  # unchanged legacy path

xform=None (default) preserves current behavior exactly, so this is additive/backward-compatible. plot() would thread xform= through to analyze() the same way it already threads reduce=/align=/normalize=.

Alternatives / prior art:

• scikit-learn's Pipeline is the standard general pattern for chaining steps, but hypertools' align/cluster operate over lists of arrays (not single arrays), so a bespoke sequential dispatcher (as above) fits better than wrapping in sklearn.pipeline.Pipeline directly.
• The manip dispatcher itself is the closest in-repo prior art for a composable-step registry.

Note: parts of this ask (composable single-array transforms) are already covered by the newer manip dispatcher (Normalize/ZScore/Smooth/Resample), but that dispatcher explicitly does not chain reduce/align/cluster steps — the core ask here (multi-step reduce→align→reduce→align pipelines) is still fully open.

Tagging as medium effort; leaving open for consideration in a future release.

[jeremymanning]

Let's implement this through the manip keyword-- add reduce/align/cluster to manip.

This also raises another important issue: we need to carefully document the implementation order for reduce, align, cluster, manip, etc. A nice flow chart diagram would be useful.