Populating a Store-specific external search

Initial Setup

This section explains which steps need to be completed to set up the integration between commercetools Composable Commerce, a message queuing service, and an external search service. The first four steps are specific to the service providers and therefore not explained here in detail.

1. Set up Store-specific search indices.
2. Set up a message queue for notifications about changes on Products.
3. Set up a message queue for notifications about changes on Product Selections.
4. Set up a message queue for notifications about changes on Stores.

Once this is complete, you need to set up three subscriptions to get notified upon changes on product data related to a Store.

Set up Subscriptions

The following subscriptions need to be set up on commercetools Composable Commerce:

Subscription for Changes on Products

Create a Subscription to listen to any updates on Product with the following changes field:

{
  ...
  "changes": [
    {
      "resourceTypeId": "product"
    }
  ]
}

Subscription for Changes on Product Selections

By using Message Subscriptions on Product Selection Messages, you get notified whenever a Product was added or removed to/from a Product Selection or there have been a change on a Variant Selection.

Subscription for Changes on Stores

By using Message Subscriptions on Store Messages, you get notified whenever a Product Selection was added or removed to/from a Store or whenever a Product Selection is activated or deactivated for a Store.

If the Stores' configuration does not change, you can skip this step since there is no need to subscribe to those changes.

Initial product data feed

You have now the search indices and message queues in place and you have also set up the Subscriptions to ensure you will not miss any update on Product data from now on. What is left to do, is to fill the search indices with Product data since the indices are still empty.

For each search index you have to find all Products available for the respective Store:

1. Given a Store, iterate over all Product Selection Assignments to get Product References. You can sort and filter on the product.id field for that purpose. Keep in mind that the response will include duplicate Products whenever more than one active Product Selection of the given Store includes a Product, so you might want to keep them distinct on your side.
2. For each Product, fetch the respective Product Projection in the Store and add it to your search index.

Keeping the search indices up to date

Feeding the search indices with initial product data is now complete. Over time it is likely that the Product data or the Product Selection will change. You need to reflect these changes in the search indices to keep them up to date in the store front, too.

Reacting to changes on Products

Any change on a Product triggers a Message to be sent to the Product queue you have set up before. You react upon such messages in the following way:

Checking which Stores are impacted

In case you maintain a preconfigured list of Stores, you can avoid this step by going straight to Updating the search index. Otherwise you need to fetch all the Stores in which the updated Product is available by:

1. finding out which Product Selections the Product belongs to.
2. iterating over all returned Product Selections and checking which Stores are impacted.

In the second step you use following where query to filter for Stores that contain activated Product Selections with the changed Product:

productSelections(active=true and productSelection(id in :productSelectionIds))

and you pass productSelectionIds as input variables.

Updating the search index

After you found out in which Store(s) the updated Product is available, you can now update the respective search indices with the changed product information.

For each impacted Store,

• if the Product was deleted, it can be removed from the search index of the corresponding Store.
• if the Product was updated, fetch the projected Product in the Store.
  • If the Product is available in the Store, put the returned data into the search index of the corresponding Store.
  • If a 404 Not Found error is returned, the Product is not available in this Store and you delete it from the search index of the corresponding Store.

As an optimization, we recommend comparing the version of the updated Product with the version stored in the search index, and to only update in case the version number from the received message is higher than the number in the search index.

Reacting to changes on Product Selections

In the section before you handled changes on the individual Products, but you also want to update our Store's search index whenever a new Product was added or removed to/from the Store's Product Selections. This case is not covered by changes on Products but by messages on the Product Selection queue you set up earlier like this:

Checking which Stores are impacted

In case you maintain a preconfigured list of Stores, you can avoid this step by going straight to Updating the search index. Otherwise you need to fetch all the Stores impacted by the change.

From the ProductSelectionProductAdded Message, the ProductSelectionProductRemoved Message, or the ProductSelectionVariantSelectionChanged Message, you get the ID of Product Selection.

You can check which Stores are impacted by using the following where query to filter for Stores that contain the updated Product Selection:

productSelections(active=true and productSelection(id = :productSelectionId))

and you pass productSelectionId as input variables.

Updating the search index

If a Product was added to a Product Selection,

1. fetch the Product Projection in the Store and
2. put the returned data into the search index of the corresponding Store.

If the Variant Selection of a Product to a Product Selection was changed,

1. fetch the Product Projection in the Store and
2. update the search index of the corresponding Store with the returned data.

If a Product was removed from a Product Selection, delete it from the search index of the corresponding Store.

Reacting to changes on Stores

You also want to update our Store's search index whenever a Product Selection was added or removed to/from a Store or whenever a Product Selection was activated or deactivated in a Store. You handle this by processing messages from the Store queue you set up earlier like this:

Checking which Stores are impacted

You get the ID of the impacted Store from the Store Messages.

Updating the search index

In case a Store was created, feed all product data into the new search index as described in initial product data feed.

In case a Store was deleted, you can remove the corresponding search index.

In case the Product Selection configuration of a Store was changed, you have two options:

• You ignore the details from the StoreProductSelectionsChanged Message and completely reindex all data for this Store as described in reindex all data from time to time.
• Alternatively, you are more specific with our updates and reindex only with those Products that are affected by the configuration change:
  1. For each Product Selection that was added, removed, activated or deactivated, you can iterate on all Products' References.
  2. For each Product, you fetch the projected product in the Store.
  • If the Product is not found you remove it from the search index.
  • Otherwise you put this data into the Store's search index.

Best practices

Detect unnecessary updates

We recommend you to store the version of the Products in the search indices, and to use this information to detect if some product data is already up to date and does not need any further updates.

Reindex all data from time to time

A search index can become out of date in case an update was missed, or due to changes on a Store's configuration. To keep such inconsistencies to a minimum, we recommend reindexing all product data in a search index, not only on demand, but also from time to time.

The following steps suggest a way how to do that without impacting your live traffic for the search index of one Store:

1. Create a second search index for this Store.
2. Let the Product updates now apply to both search indices: the live index and the new index.
3. Feed all product data into the new search index as described in initial product data feed.
4. Once the new search index is ready, move your live traffic over to it.
5. Delete the old live index that is not used anymore.