Elasticsearch to OpenSearch Migration Plan

Elasticsearch to OpenSearch Migration Plan

Purpose

Document the feasibility and plan for migrating from Elasticsearch 8.x to OpenSearch 2.x. This migration would reduce licensing concerns and costs while maintaining feature parity.

Current State

Elasticsearch Usage in Codebase

Feature	How We Use It	Files
Full-text search	Course/consultation search	search.service.ts
KNN/Vector search	Product embeddings, user embeddings	recommendation.service.ts, user-embedding.service.ts
Painless scripts	Scoring, field calculations	Various query builders
PIT pagination	Deep pagination for large result sets	search.service.ts
Aggregations	Faceted search, analytics	search.service.ts

Current Client

• Package: @elastic/elasticsearch v8.x
• Connection: Via ElasticsearchService from @nestjs/elasticsearch

Compatibility Assessment

What Works the Same

Feature	Compatibility	Notes
Basic CRUD	Full	Index, get, delete, bulk operations identical
Full-text queries	Full	match, multi_match, bool queries unchanged
Aggregations	Full	Terms, range, nested aggs work the same
PIT pagination	Full	Point-in-time API available in OpenSearch
Painless scripts	High	Core Painless syntax compatible

What Requires Changes

Feature	Change Required	Effort
Client library	@elastic/elasticsearch → @opensearch-project/opensearch	Low
Vector field type	dense_vector → knn_vector	Medium
KNN query syntax	Different query structure	Medium
Index mappings	Update vector field definitions	Low

Migration Steps

Phase 1: Client Swap (Day 1)

1. Install OpenSearch client

   npm uninstall @elastic/elasticsearch @nestjs/elasticsearch
   npm install @opensearch-project/opensearch

2. Create OpenSearch module wrapper

   • Create opensearch.module.ts to replace ElasticsearchModule
   • Maintain same service interface

3. Update imports across codebase

   • Find/replace ElasticsearchService references

Phase 2: Vector Search Migration (Day 1-2)

1. Update index mappings for vector fields

   Before (Elasticsearch):

   {
     "product_embedding": {
       "type": "dense_vector",
       "dims": 1536,
       "index": true,
       "similarity": "cosine"
     }
   }

   After (OpenSearch):

   {
     "product_embedding": {
       "type": "knn_vector",
       "dimension": 1536,
       "method": {
         "name": "hnsw",
         "space_type": "cosinesimil",
         "engine": "nmslib"
       }
     }
   }

2. Update KNN query syntax

   Before (Elasticsearch 8.x):

   {
     knn: {
       field: 'product_embedding',
       query_vector: userEmbedding,
       k: 10,
       num_candidates: 50
     }
   }

   After (OpenSearch):

   {
     query: {
       knn: {
         product_embedding: {
           vector: userEmbedding,
           k: 10
         }
       }
     }
   }

3. Reindex data with new mappings

Phase 3: Testing & Validation (Day 2-3)

1. Unit tests - Verify all search functions work
2. Integration tests - Test against OpenSearch instance
3. Performance comparison - Benchmark KNN search latency
4. Data validation - Ensure search results quality unchanged

Phase 4: Deployment

1. Set up OpenSearch cluster (or use managed service)
2. Migrate data from Elasticsearch
3. Update environment configurations
4. Deploy updated application
5. Monitor for issues

Files to Modify

File	Changes
server/src/elastic/elastic.module.ts	Replace with OpenSearch module
server/src/elastic/elastic.service.ts	Update client usage
server/src/elastic/indexes/*.ts	Update vector field mappings
server/src/recommendation/services/*.ts	Update KNN query syntax
server/src/search/search.service.ts	Update query builders
package.json	Swap client packages

Estimated Effort

Phase	Time
Client swap	0.5 day
Vector search migration	1 day
Testing & validation	1 day
Total	2-3 days

Risks & Mitigations

Risk	Mitigation
Query syntax differences we missed	Comprehensive integration tests
Performance regression on KNN	Benchmark before/after, tune HNSW parameters
Painless script edge cases	Review all script_score queries
Data migration issues	Run parallel environments during transition

Decision

Status: Planned for future (not urgent)

Rationale: Current Elasticsearch setup works well. Migration provides cost savings but requires dedicated effort. Plan preserved here for when the time is right.

Resources

• OpenSearch Documentation
• OpenSearch KNN Plugin
• ES to OpenSearch Migration Guide

Last Updated

2025-01-27