Integrate product data

Key questions to ask before planning your integration

Before designing an integration, take some time to explore the Integrations Marketplace for existing solutions. Using a pre-existing integration can save significant time and effort.

However, if you decide to create an integration from scratch, there are some key things to bear in mind. One of the main points to consider is the origin and handling of your product data:

• What is the primary source for product data? Define what you consider to be the source (or sources) of truth for product data.
• How many data sources are there? Determine the number of data sources and if they should all send data to Composable Commerce through a single integration.
• Where can product data be modified?: Decide if product data can be modified within Composable Commerce.
• What is the direction of the data flow? Evaluate if the integration is bi-directional (meaning some data needs to be sent back to the source system), what data is transferred between systems, and how data consistency is maintained.

Diving deeper into the sources of product data, let's explore scenarios that can influence your integration design.

Single source of product data

The most straightforward scenario is having a single source for product data, like a product information management (PIM) system, where no product data is modified in Composable Commerce. In this case, an integration process pulls the data from the PIM system and stores it in Composable Commerce.

Managing multiple sources of product data

Handling multiple sources of product data adds layers of complexity. For an integration where Composable Commerce can also modify product data, define the following:

1. The source of truth for each attribute:
   • Determine which product attributes come from your primary product data source, such as an ERP or PIM system.
   • To avoid unintentional changes, configure these attributes as read-only within Composable Commerce. You can achieve this by placing them in a restricted Attribute Group, ensuring no user edits are possible.
2. The process for creating Products in Composable Commerce:
   • Ensure the initial version of a product is sent to Composable Commerce just once when it is first created.
   • Restrict any subsequent updates to only include attributes that can be changed in Composable Commerce.
3. The number of integrations required:

• Choose between:
  • A single integration that consolidates product data from all sources into Composable Commerce. This is often preferable if the dependencies and requirements allow.
  • Multiple processes that populate Composable Commerce with each providing a different piece of the product information. Here, a primary process first creates the Product in Composable Commerce, ensuring that any subsequent processes feed additional data only after Product creation.

In addition, for cases where the integration is bi-directional—meaning Composable Commerce needs to send back product data to the source system—define the following:

4. The process for synchronizing attributes managed by Composable Commerce:
   • Identify the attributes for which Composable Commerce is the source of truth, and ensure that only these attributes are sent back to the source system or other integrated systems.

Choosing your integration approach

After understanding the requirements around single or multiple data sources, you are now ready to integrate the data. The method to choose depends on the frequency of updates, the volume of data, and the capabilities of your data sources. Let's explore the main integration approaches to determine the best fit for your scenario.

Full versus incremental upload

When integrating product data, you have two primary methods to choose from: full uploads and incremental uploads.

Full uploads involve sending all product data from the source to Composable Commerce. This approach takes longer due to the extensive amount of data involved. As such, the data in Composable Commerce remains in a partially updated state for longer. Handling this requires a detailed approach, especially when other processes depend on all products being updated.

Incremental uploads focus on products that have been newly created or modified since the last upload. This method is faster than full uploads, but not all data sources support it. If your data source does not support incremental updates, it's a good idea to use a middleware equipped with a database to track the changes.

One of the benefits of the incremental method is its ability to address time-critical updates. For example, if a product description contains an error, you can fix it in the primary data source, such as a PIM system, and run the integration. This responsiveness is a distinct advantage over full uploads, which might not be suitable for ad-hoc updates.

Event-driven versus bulk approach

Based on your requirements, you can choose between event-driven and bulk integrations.

Event-driven integrations handle data changes as they occur and provide near real-time updates. As such, they require frequent requests. Use this approach when data must be updated as soon as possible or available, as the real-time update of this data is crucial for operations.

Bulk integrations can be more efficient, especially when only updating specific changes. They are appropriate for scenarios with substantial data volumes where immediate updates are not essential. These integrations usually run at set intervals, with nightly updates being common.

Data mapping

Once you decide on your integration approach, we can move to another pivotal aspect of the process: data mapping. This step aligns data from your primary sources with Composable Commerce, ensuring consistency and relevance. Let's explore some key principles that can simplify the process:

• Only transfer essential product data: not all details from your PIM system or other data sources might be necessary in Composable Commerce. Ensure that you only map and transfer the data that are directly relevant to commerce.
• Ensure structure compatibility: the data structure in Composable Commerce, as highlighted in the Product Modeling Guide, should be tailored for commerce utility. Conversely, PIM data is optimized for product enrichment. It's essential to understand that these structures can differ, and you may need to make adjustments when mapping between the two.
• Avoid a 1-to-1 relationship between product types: linking Product Types in Composable Commerce directly to those in the originating data source might not always be the best approach. Such a setup would necessitate regular checks for changes in structure to apply any updates accordingly. Adopting a less specific product type setup with versatile attributes can simplify this process.

As you proceed with data mapping, prioritize essential attributes, especially those crucial for search functionalities or specific features. Establish a clear mapping strategy for these. Any supplementary information can be consolidated, for instance, as a JSON text attribute.

Managing price and inventory information

Price and inventory updates are more frequent and time-critical than updates to core product information. We strongly recommend implementing them as separate, event-based integrations. Even if data for pricing and inventory are managed within the same system, separating the processes ensures fast and accurate updates.

Data integration tools in Composable Commerce

Composable Commerce offers a range of tools and APIs to facilitate an efficient data integration process:

• Subscriptions API: for event-driven integrations, you can subscribe to Messages for changes to resources in Composable Commerce. To learn more on subscribing to events, visit the Subscriptions API documentation.

• Import API or Java Sync for bulk integrations: when using the Import API, monitor the container summary and operations for error detection and to confirm the process completion, especially before starting dependent processes. Alternatively, consider using tools like the Java Sync Library.

• Products, Inventory, and Prices APIs: when using the Composable Commerce APIs for integrations, design each API Client with the necessary permissions for its specific role. By having dedicated API Clients for each process, you can enhance security and prevent unnecessary permissions from being granted.

Conclusion

Effective data integration in Composable Commerce relies on thorough planning and a detailed understanding of the requirements and technical possibilities. By navigating the options and best practices outlined in this guide, you can plan a data integration that ensures efficient operations.