Skip to content

Add search pipelines - #145

Open
esmarkowski wants to merge 2 commits into
feature/add-ml-commons-agentsfrom
feature/add-search-pipelines
Open

esmarkowski wants to merge 2 commits into
feature/add-ml-commons-agentsfrom
feature/add-search-pipelines

Conversation

@esmarkowski

@esmarkowski esmarkowski commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Ingest and search pipelines are now separate classes that share one base and one processor DSL. Models can set a default search pipeline, and a relation can choose a pipeline for a single query.

class RagPipeline < Stretchy::SearchPipeline
  response_processor :retrieval_augmented_generation, model: :claude_model, context_field_list: [:body]
end

class Post < StretchyModel
  default_search_pipeline :rag_pipeline        # index.search.default_pipeline
end

Post.search_pipeline(:hybrid_normalization).where(title: "hello")   # ?search_pipeline=hybrid_normalization
  • SearchPipeline.create! issues PUT /_search/pipeline/<name>.
  • Search pipelines are OpenSearch-only.
  • The pipeline rake tasks manage both kinds and skip search pipelines on Elasticsearch.
  • The rake tasks select a single pipeline with PIPELINE instead of MODEL.
  • A processor's model: option accepts a model class, a class name or a model id.
  • Stretchy::Pipeline still works as a deprecated subclass of Stretchy::IngestPipeline. It warns through Stretchy.deprecator until 0.8.

Verified against OpenSearch 3.9.0:

  • A filter_query default search pipeline filtered results.
  • A per-query search_pipeline overrode the default.

Specs pass on OpenSearch 3.9.0, OpenSearch 2.19.7 and Elasticsearch 8.12.2.

Closes #138

Spencer Markowski and others added 2 commits November 16, 2026 17:25
Stretchy::IngestPipeline replaces Stretchy::Pipeline, and
Stretchy::SearchPipeline adds request, response and phase results
processors managed through PUT /_search/pipeline/<name>. Both share
Pipelines::Base and Pipelines::Processor, whose model: option now accepts a
model class, class name or id.

Stretchy::Pipeline still works as a deprecated subclass of IngestPipeline,
warning through Stretchy.deprecator until 0.8. The pipeline rake tasks manage
both kinds, skipping search pipelines on Elasticsearch, and select one with
PIPELINE instead of MODEL.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`default_search_pipeline :rag_pipeline` sets index.search.default_pipeline in
the index settings, and `Post.search_pipeline(:rag_pipeline)` sends the
search_pipeline request parameter for one query. Both accept a SearchPipeline
class, its class name or a pipeline name.

routing and search_pipeline share one definition for single value search
options.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@esmarkowski
esmarkowski added this pull request to stack #142 November 16, 2026 22:31
@esmarkowski esmarkowski changed the title feature/add search pipelines Add search pipelines Nov 16, 2026
@esmarkowski
esmarkowski marked this pull request as ready for review November 16, 2026 22:31

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add search pipelines

1 participant