Upgrading Search¶

Starting version 4.1.0, CrafterCMS now uses OpenSearch. This section describes how to upgrade search for CrafterCMS installed on a server.

Please read through the upgrade instructions here first. The steps for upgrading to OpenSearch follows almost exactly the same steps as listed in that document.

Important

Remember to manually shut down and backup CrafterCMS before beginning your upgrades!

Upgrading 3.1.x -> 4.2.0 (from ES 6.x)¶

To upgrade your 3.1.x installation, we’ll be running the upgrade scripts from a new binary archive. We’ll use the upgrade-search.sh script, which will update the data in place. This script tells the search engine to re-index internally to the new format and should only be run on CrafterCMS 3.1.x installs. Please backup your data directory before running the script.

Here is the upgrade-search script params:

 -h,--help                                  Show usage information
    --port <port>                           Elasticsearch port to use for
                                            upgrade ES temporary instance
    --status-retries <max status retries>   How many times to try to get a
                                            yellow status from the ES
                                            cluster (waiting 10s between
                                            retries)
    --status-timeout <seconds>              Timeout in seconds for the
                                            status check of the ES cluster
    --stay-alive                            Set to true to keep the
                                            process alive after reindexing
                                            is complete. This allows to
                                            query the ES server and
                                            review.
e.g:

# Run in a different port
./upgrade-search.sh --port 9206 /path/of/install/to/be/upgraded --stay-aliveUpgrading 3.1.x -> 4.1.0 (from ES 6.x)

Here are the steps to upgrade your CrafterCMS 3.1.x install:

Download the CrafterCMS version you’d like to upgrade to, and extract the files.
Configure the root directory of the new bundle to use Java version 17 (This is required to run the upgrade-search.sh script since ES 7 won’t run with Java 21)

Run the upgrade-search.sh script from your newly extracted files.

Run the upgrade-search script¶

./bin/upgrade/upgrade-search.sh /path/of/install/to/be/upgraded --stay-alive
========================================================================
Search upgrade started
========================================================================
...
End process. Stop Elasticsearch
Move indexes from 'data/indexes-es' to 'indexes'
========================================================================
Search upgrade completed
========================================================================

Configure the root directory back to Java version 21

Upgrade using the upgrade-target.sh script. We’ll need to upgrade the target without backing up the bin folder. This is because the MariaDB version is now 11 for 4.2.0, and we cannot start the old MariaDB from the new bundle. Please manually back it up before running the upgrade-target.sh script. Remember to enter No when prompted by the script Backup the bin folder before upgrade? [(Y)es/(N)o]:

Run the upgrade-target script¶

./bin/upgrade/upgrade-target.sh /path/of/install/to/be/upgraded
...
> Backup the bin folder before upgrade? [(Y)es/(N)o]:n
...
========================================================================
Upgrade completed
========================================================================
!!! Please read the release notes and make any necessary manual changes, then run the post upgrade script: /path/of/install/to/be/upgraded/bin/upgrade/post-upgrade.sh !!!

Run the post-upgrade.sh script from the install that’s being upgraded. This will start CrafterCMS and ask for a signal to continue, then recreate search indexes. Remember to read the release notes or any relevant upgrade articles and make any necessary manual changes before running the post-upgrade.sh script. You’ll need to configure the installation root directory to use Java version 21 before running the script.

Run the post-upgrade script¶

cd /path/of/install/to/be/upgraded/bin/upgrade/
./post-upgrade.sh
========================================================================
Post-upgrade 3.1.30 -> 4.2.0
========================================================================
...
Please make sure Crafter has started successfully before continuing
> Continue? [(Y)es/(N)o]: y
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Re-creating Search indexes for sites
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
WARNING: This will delete the current Search site indexes and recreate them.
This is necessary because of a major Search upgrade. Don't proceed
if you can't have any search downtime.
> Proceed? [(Y)es/(N)o]: y
...
========================================================================
Post-upgrade completed
========================================================================
Crafter has already been started, you can use the system again

The script will prompt you to check that CrafterCMS has started successfully before proceeding as noted above. To do this, monitor the tomcat logs and check for the line like below to make sure CrafterCMS has started (this could take a while because of the upgrade manager (UM) updates):

27-Jun-2024 08:14:11.119 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [127790] milliseconds

Once the post-upgrade script is done, all indices should be now available in OpenSearch and CrafterCMS is now ready for use.

Upgrading 4.0.x -> 4.2.0 (from ES 7.15)¶

When upgrading from 4.0.x (running ES 7) the indices are not compatible at all, so the content needs to be reprocessed and indices rebuilt completely. The rebuilding of the indices is handled by the post-upgrade.sh script. Remember that the upgrade-search.sh script should NOT be run when upgrading your CrafterCMS 4.0.x install.

To upgrade your 4.0.x installation, we’ll be running the upgrade scripts from a new binary archive. Here are the steps:

Download the CrafterCMS version you’d like to upgrade to, and extract the files.
Upgrade using the upgrade-target.sh script from your newly extracted files:
Run the upgrade-target script¶
```
./upgrade-target.sh /path/of/install/to/be/upgraded
```
Run the post-upgrade.sh script. You’ll need to configure the installation root directory to use Java version 21 before running the script. This will:
- Remove old data/indexes-es directory (old indexes are not usable by OpenSearch)
- Start CrafterCMS and ask for signal to continue
- Once started and CrafterCMS is up (including UM execution), let the post-upgrade continue by typing Y:
```
Please make sure Crafter has started successfully before continuing
> Continue? [(Y)es/(N)o]:
```
- Post-upgrade will continue to trigger the reindex of all targets by calling the Deployer API /api/1/target/deploy-all

Monitor the Deployer logs and wait for the reindex to be completed. You should see a message like the following:

2024-04-20 14:36:46.050  INFO 376430 --- [deployment-1] org.craftercms.deployer.impl.TargetImpl  : Deployment for editorial110-authoring finished in 9.953 secs

Upgrading 4.1.x -> 4.2.0¶

There are no extra steps required for upgrading your 4.1.x install to 4.2.0, simply follow the instructions here. Remember that the upgrade-search.sh script should NOT be run when upgrading your CrafterCMS 4.1.x install.

Manual Updates for Search¶

The Upgrade Manager (UM) performs most of the updates required to upgrade your project to OpenSearch, such as the import updates in your classes. There are some instances where manual updates may need to be performed like below:

Updating Search Beans¶

If you have an application context that injects Elasticsearch like below, it will need to be updated to inject OpenSearch:

Application context that injects Elasticsearch¶

<beans xmlns="http://www.springframework.org/schema/beans"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

    <bean id="demoProfileService" class="com.demo.services.ProfileService" />

    <bean id="demoSearchService" class="com.demo.services.SearchService">
        <property name="elasticsearch" ref="crafter.elasticsearchService" />
        <property name="urlTransformationService" ref="crafter.urlTransformationService" />
    </bean>
</beans>

To update to OpenSearch, in the example above, the property is called elasticsearch and will need to be renamed. In the example below, the property has been renamed to searchClient:

Application context injection updated to OpenSearch¶

<beans xmlns="http://www.springframework.org/schema/beans"
             xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
             xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

    <bean id="demoProfileService" class="com.demo.services.ProfileService" />

    <bean id="demoSearchService" class="com.demo.services.SearchService">
        <property name="searchClient" ref="crafter.searchClient" />
        <property name="urlTransformationService" ref="crafter.urlTransformationService" />
    </bean>
</beans>

Here’s a sample Groovy script that executes a search query in a project created using the Website Editorial blueprint:

Sample search query Groovy script.

package org.craftercms.sites.editorial

import org.opensearch.client.opensearch._types.query_dsl.BoolQuery
import org.opensearch.client.opensearch._types.query_dsl.Query
import org.opensearch.client.opensearch._types.query_dsl.TextQueryType
import org.opensearch.client.opensearch.core.SearchRequest
import org.apache.commons.lang3.StringUtils
import org.craftercms.engine.service.UrlTransformationService
import org.craftercms.search.opensearch.client.OpenSearchClientWrapper

class SearchService {

    static final String ARTICLE_CONTENT_TYPE = "/page/article"
    static final List<String> ARTICLE_SEARCH_FIELDS = [
            'subject_t^1.5',
            'sections_o.item.section_html^1.0'
    ]

    OpenSearchClientWrapper searchClient
    UrlTransformationService urlTransformationService

    SearchService(OpenSearchClientWrapper searchClient, UrlTransformationService urlTransformationService) {
        this.searchClient = searchClient
        this.urlTransformationService = urlTransformationService
    }

    def search(userTerm) {
        def query = new BoolQuery.Builder()

        // Filter by content-type
        query.filter(q -> q
                .match(m -> m
                        .field("content-type")
                        .query(v -> v
                                .stringValue(ARTICLE_CONTENT_TYPE)
                        )
                )
        )

        if (userTerm) {
            // Check if the user is requesting an exact match with quotes
            def matcher = userTerm =~ /.*("([^"]+)").*/
            if (matcher.matches()) {
                // Using must excludes any doc that doesn't match with the input from the user
                query.must(q -> q
                        .multiMatch(m -> m
                                .query(matcher.group(2))
                                .fields(ARTICLE_SEARCH_FIELDS)
                                .fuzzyTranspositions(false)
                                .autoGenerateSynonymsPhraseQuery(false)
                        )
                )

                // Remove the exact match to continue processing the user input
                userTerm = StringUtils.remove(userTerm, matcher.group(1))
            } else {
                // Exclude docs that do not have any optional matches
                query.minimumShouldMatch("1")
            }

            if (userTerm) {
                // Using should makes it optional and each additional match will increase the score of the doc
                query
                // Search for phrase matches including a wildcard at the end and increase the score for this match
                        .should(q -> q
                                .multiMatch(m -> m
                                        .query(userTerm)
                                        .fields(ARTICLE_SEARCH_FIELDS)
                                        .type(TextQueryType.PhrasePrefix)
                                        .boost(1.5f)
                                )
                        )
                // Search for matches on individual terms
                        .should(q -> q
                                .multiMatch(m -> m
                                        .query(userTerm)
                                        .fields(ARTICLE_SEARCH_FIELDS)
                                )
                        )
            }
        }

        SearchRequest request = SearchRequest.of(r -> r
                .query(query.build()._toQuery())
        )

        def result = searchClient.search(request, Map)

        if (result) {
            return processUserSearchResults(result)
        } else {
            return []
        }
    }

    private def processUserSearchResults(result) {
        def articles = []
        def hits = result.hits().hits()

        if (hits) {
            hits.each {hit ->
                def doc = hit.source()
                def article = [:]
                article.id = doc.objectId
                article.objectId = doc.objectId
                article.path = doc.localId
                article.title = doc.title_t
                article.url = urlTransformationService.transform("storeUrlToRenderUrl", doc.localId)

                articles << article
            }
        }

        return articles
    }

}

Search Methods/Groovy Code¶

You might encounter the following error in your project, which indicates your search methods in your groovy code needs to be updated for OpenSearch:

Error message in logs indicating groovy code needs to be updated¶

Caused by: org.craftercms.engine.exception.ScriptException: No signature of method: org.craftercms.engine.search.SiteAwareOpenSearchClient.search() is applicable for
    argument types: (org.opensearch.action.search.SearchRequest) values: [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[],
    indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false,
    allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], routing='null', preference='null', requestCache=null,
    scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null,
    getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"from":0,"size":6,"query":{"query_string":{"query":"content-type:\"/page/blogpost\" AND ( (NOT
    (_exists_:unlisted_b)) OR unlisted_b:false) ","fields":[],"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,
    "fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,
    "boost":1.0}},"sort":[{"publishedDate_dt":{"order":"desc"}}]}, cancelAfterTimeInterval=null, pipeline=null}]
Possible solutions: search(org.opensearch.client.opensearch.core.SearchRequest, java.lang.Class, java.util.Map), each(groovy.lang.Closure), macro(groovy.lang.Closure)