More text models

[jeremymanning]

Looks like we could relatively easily add support for a bunch more models via this package: https://radimrehurek.com/gensim/install.html

[jeremymanning]

Not addressed in the HyperTools 1.0 update (PR #272, #272): text2mat.py still only ships scikit-learn's CountVectorizer/TfidfVectorizer and LatentDirichletAllocation/NMF; no gensim integration or dependency exists.

How this could be built on the 1.0 architecture: text2mat.py's vectorizer_models/texts dicts are the same "string/dict/class/instance → resolve → registry lookup" shape used elsewhere in the 1.0 codebase (e.g. hypertools/reduce/reduce.py's models dict + _resolve_model, or hypertools/manip/manip.py's MANIPULATORS list + unpack_model). Since text2mat.py hasn't yet been ported to the newer Manipulator-style base-class+registry+funnel pattern, adding gensim support is a good opportunity to modernize it at the same time: add a small adapter layer (e.g. GensimDoc2Vec/GensimLSI wrapper classes) that bridges gensim's train/infer_vector methods to the fit_transform contract text2mat already documents for user-supplied models, then register them into the existing dicts (or a new embedding_models dict) the same way.

Alternatives / prior art:

• gensim — actively maintained (4.4.0, released 2025-10), confirmed importable/installable. However, gensim 4.x removed the old gensim.sklearn_api wrapper module (present in gensim 3.x, deprecated then dropped in 4.0) that used to make this "relatively easy" — hand-written adapters are now required, which is why this is medium rather than low effort.
• Users can already pass a custom class/instance as vectorizer=/semantic= today, so advanced users can hand-roll a gensim wrapper without any hypertools changes — this only closes the "built-in, no-glue-code" gap.
• Modern alternative worth considering instead of/alongside gensim: transformer-based sentence embeddings (sentence-transformers, HF embedding pipelines) are now the more common default for semantic text representations.

Note: parts of this ask are superseded by the shift toward transformer-based embeddings as the default for semantic similarity/topic work; gensim's classical word2vec/LDA-era models remain useful mainly for topic modeling (LSI/HDP) that sklearn doesn't offer.

Tagging as medium effort; leaving open for consideration in a future release.

[jeremymanning]

we're already supporting sentence transformers via HF embeddings (through the shift to wrapping the data-wrangler package).

adding wrappers for gensim seems like a good direction. for the API, let's parse it as:

• first look for the scikit-learn model
• then, if that doesn't exist, look for the gensim model
• then, if that doesn't exist, look for the huggingface model

So the main change is writing the wrappers for gensim so that no additional code is required for vectorize/semantic.