Use the Import API

Practice working with the Import API and SDK to create and update a Customer.

After completing this page, you should be able to:

Use the Import API to create and update a Customer in Composable Commerce.

On this page, you will work through an example use case to get a better feel for using the Import API. We will use an SDK (you can use either the Java or TypeScript SDK) to import new Customers into your Composable Commerce Project with the Import API.

The example use case will be similar to the Customer example given at the beginning of this module, except you will start with importing just one Customer before attempting to import 100,000 of them! However, we will practice making both a create and an update request, so you can see how both work.

We will attempt to execute these two main tasks in our example:

Import the Customer knowing that it does not yet exist. Then we will check if the Customer has been created or not.
Modify the Customer. We will now re-run our code and hence, re-use the Import API to import the Customer knowing that it now exists. Then we will check if the Customer has been updated or not.

Here is the Customer data for our exercise:

{
  "key": "imported-customer-01",
  "firstName": "Sam",
  "lastName": "Davies",
  "email": "sdavies@example.com",
  "password": "secret123",
  "addresses": [{ "key": "imported-customer-01-address", "country": "DE" }]
}

Remember that the Import API uses key for identifying and referencing resources. That’s why the example above not only has a key for the Customer but also for each address.

Prepare an Import API Client

If you’ve been working through the Developer Essentials learning path sequentially, you would have already prepared your Composable Commerce Project and environment. If not, you should do the following before continuing with the practical exercise:

Follow the getting started guide to set up your Project (remember to create a Project using the sample data).
Prepare your work environment, including setting up your IDE. You only need to choose one of the programming languages.

Create an API Client in the Merchant Center

For this exercise, you should create a new API Client with the following scopes: manage_import_containers, manage_customers, and manage_customer_groups. It is best practice to not give API Clients more scopes than necessary, as it reduces the chance of accidents and misuse. Remember to copy or download the API Client credentials, as you will be unable to access them once you leave the API Client details page.

Create an import client in your SDK

Now, let's create an import client for the SDK. The following Java and TypeScript code will create a client that uses the Import API. Notice that it is very similar to what we have already done in the Prepare your work environment module, the main difference here is that the Import API domain is slightly different:

Java

import com.commercetools.importapi.client.ProjectApiRoot;
import com.commercetools.importapi.defaultconfig.ImportApiRootBuilder;
import com.commercetools.importapi.defaultconfig.ServiceRegion;
import io.vrap.rmf.base.client.oauth2.*;

public class ImportClientBuilder {
  public static ProjectApiRoot createImportApiClient() {
    final ProjectApiRoot importApiRoot = ImportApiRootBuilder.of()
      .defaultClient(ClientCredentials.of()
        .withClientId("{clientId}")
        .withClientSecret("{clientSecret}")
        .withScopes("{scopes}")
      .build(),ServiceRegion.GCP_EUROPE_WEST1)
    .build("{projectKey}");

    return importApiRoot;
  }
}

TypeScript

You will also need to add the importHost to your environment variables, as the base URL used to access the Import API is different from the HTTP API. The variable name should be CTP_IMPORT_API_URL and the value should be the host that matches your project Region. See Hosts and authorization for further information.

import fetch from 'node-fetch';
import { ClientBuilder } from '@commercetools/ts-client';
import 'dotenv/config';
import { createApiBuilderFromCtpClient } from '@commercetools/importapi-sdk';

// --- Configuration ---
const projectKey = process.env.CTP_PROJECT_KEY;
const clientId = process.env.CTP_CLIENT_ID;
const clientSecret = process.env.CTP_CLIENT_SECRET;
const authUrl = process.env.CTP_AUTH_URL;
const importApiUrl = process.env.CTP_IMPORT_API_URL;
const scopes = [];

// --- Middleware Functions ---

// Function for custom header middleware
function createCustomHeaderMiddleware() {
  return (next) => (request) => {
    return next({
      ...request,
      headers: {
        ...request.headers,
        'accept-language': 'en-AU',
      },
    });
  };
}

// Function for custom logger middleware
const customLoggerMiddleware = {
  logLevel: 'debug',
  httpMethods: ['POST', 'GET'],
  maskSensitiveData: true,
  logger: (method, ...args) => {
    console.log(`[CUSTOM LOGGER] ${method}`, ...args);
  },
};

// --- Middleware Options ---

// Auth Middleware Options
const authMiddlewareOptions = {
  host: authUrl,
  projectKey: projectKey,
  credentials: { clientId, clientSecret },
  scopes: scopes,
  httpClient: fetch,
};

// Http Middleware Options
const httpMiddlewareOptions = {
  host: importApiUrl,
  includeResponseHeaders: true,
  maskSensitiveHeaderData: true,
  includeOriginalRequest: true,
  includeRequestInErrorResponse: true,
  enableRetry: true,
  retryConfig: {
    maxRetries: 3,
    retryDelay: 200,
    backoff: false,
    retryCodes: [500, 503],
  },
  httpClient: fetch,
};

// Correlation ID Middleware Options
const correlationIdMiddlewareOptions = {
  // Replace with your own UUID, ULID, or a generator function. Do not use this example in production!
  generate: () => 'cd260fc9-c575-4ba3-8789-cc4c9980ee4e',
};

// Concurrent Modification Middleware Options
const concurrentModificationMiddlewareOptions = {
  concurrentModificationHandlerFn: (version, request) => {
    console.log(`Concurrent modification error, retry with version ${version}`);
    request.body.version = version;
    return JSON.stringify(request.body);
  },
};

// --- Optional Telemetry Middleware (Commented Out) ---
/*
const telemetryOptions = {
  createTelemetryMiddleware,
  apm: () => typeof require('newrelic'),
  tracer: () => typeof require('/absolute-path-to-a-tracer-module'),
};
*/

// --- Client Creation ---
const client = new ClientBuilder()
  .withProjectKey(projectKey)
  .withClientCredentialsFlow(authMiddlewareOptions)
  .withLoggerMiddleware(customLoggerMiddleware)
  .withCorrelationIdMiddleware(correlationIdMiddlewareOptions)
  .withMiddleware(createCustomHeaderMiddleware())
  .withHttpMiddleware(httpMiddlewareOptions)
  .withConcurrentModificationMiddleware(concurrentModificationMiddlewareOptions)
  // .withTelemetryMiddleware(telemetryOptions)
  .build();

// --- API Root Creation ---
const apiRootImport = createApiBuilderFromCtpClient(client).withProjectKeyValue(
  { projectKey }
);

export { apiRootImport };

Now we have a client for importing data, and we are ready to import our Customer. As defined in our scopes we can use it to create and monitor containers, import Customers, and import Customer Groups. This will allow us to complete our following tasks.

We will follow the Import API workflow as we continue, so it would be a good idea to have it at hand for easy referral.

Workflow step 1: Create an Import Container

We will first create an Import Container that can process the import of our Customer. This is a simple create operation. As you know, Composable Commerce requires you to POST an ImportContainerDraft. For the key, it is good practice to use something descriptive, and we will use myProject-customer-container-learn in the following examples.

Here’s our code that allows us to do this:

ProjectApiRoot importApiRoot = ImportClientBuilder.createImportApiClient();

String containerKey = "myProject-customer-container-learn";

importApiRoot
  .importContainers()
  .post(
    ImportContainerDraftBuilder
      .of()
      .key(containerKey)
      .resourceType(ImportResourceType.CUSTOMER)
      .build()
  )
  .executeBlocking();

importApiRoot.close();

Nice work! Let's now verify that the Import Container exists:

String containerKey = "myProject-customer-container-learn";

List<ImportContainer> containerList = importApiRoot
  .importContainers()
  .get()
  .executeBlocking()
  .getBody()
  .getResults();

for (ImportContainer ic : containerList) {
  if (ic.getKey().equals(containerKey)) System.out.println(
    "Our container (" + containerKey + ") is ready!"
  );
}

If successful, you will get a verification response such as the following:

You can also view the Import Container in the Merchant Center by going to Operations > Import > Import logs.

Great job! Let’s move on to the next step.

Workflow Step 2: Import the Customer (CREATE)

Remember the Customer that was mentioned in our use case? Now that our Import Container is ready, we can finally import our Customer into the Project.

After taking another glance at the workflow, our next step is to create and submit a CustomerImportRequest:

importApiRoot
  .customers()
  .importContainers()
  .withImportContainerKeyValue(containerKey)
  .post(
    CustomerImportRequestBuilder
      .of()
      .resources(
        CustomerImportBuilder
          .of()
          .key("imported-customer-01")
          .firstName("Sam")
          .lastName("Davis")
          .email("sdavis@example.com")
          .password("secret123")
          .addresses(
            CustomerAddressBuilder
              .of()
              .key("imported-customer-01-address")
              .country("DE")
              .build()
          )
          .build()
      )
      .build()
  )
  .executeBlocking();
importApiRoot.close();

After submitting the Import Request, you can view the status of your Import Container and Import Operations in the Merchant Center by going to Operations > Import > Import logs.

If the operation is successful, the Status will change to Successfully completed.

We can also see our new Customer by going to Customers > Customer list:

Workflow Step 3: Import the Customer (UPDATE)

Our new Customer is looking good, but it looks like we have made a small mistake. If you look again at the JSON data for our Customer at the beginning of our use case, you might notice that our Customer’s last name is actually “Davies” and not “Davis”. We need to correct that mistake as soon as possible! As mentioned previously, we don’t have to use different requests for creating and updating resources.

In that case all we have to do is update the code we just used to create the Customer with the accurate information:

"lastName": "Davies",
"email": "sdavies@example.com",

Then, we can run the same code from Step 2 again. No other change is required!

After a few moments we should see our Customer updated in the Merchant Center:

Workflow Step 4: Import Status Monitoring

Of course the Merchant Center is not the only way of checking the status of your Import Containers and operations. Let’s see how we can use the SDK programmatically to automate the monitoring.

After submitting an ImportRequest, you can query the Import Container to receive a list of its Import Operations and their statuses. The code would look like this:

List<ImportOperation> importOperations = importApiRoot
  .importContainers()
  .withImportContainerKeyValue(containerKey)
  .importOperations()
  .get()
  .executeBlocking()
  .getBody()
  .getResults();
for (ImportOperation io : importOperations) {
  System.out.println(
    "Resource: " + io.getResourceKey() + " State: " + io.getState()
  );
}
apiRoot.close();

The following is an example response to the request we have just made, with the state of each operation highlighted:

{
  "limit": 20,
  "offset": 0,
  "count": 3,
  "total": 3,
  "results": [
    {
      "version": 2,
      "importContainerKey": "myProject-customer-container-learn",
      "resourceKey": "imported-customer-01",
      "id": "23d3bd12-47d5-44af-aa17-774f02f0cbb7",
      "state": "imported",
      "resourceVersion": 5,
      "createdAt": "2023-12-07T14:28:25.159Z",
      "lastModifiedAt": "2023-12-07T14:29:05.487Z",
      "expiresAt": "2023-12-09T14:28:25.159Z"
    },
    {
      "version": 1,
      "importContainerKey": "myProject-customer-container-learn",
      "resourceKey": "imported-customer-02",
      "id": "3be2c856-3278-41ac-ac47-a7e1bcfff03d",
      "state": "processing",
      "resourceVersion": 5,
      "createdAt": "2023-12-07T14:28:26.143Z",
      "lastModifiedAt": "2023-12-07T14:29:05.421Z",
      "expiresAt": "2023-12-09T14:28:25.159Z"
    },
    {
      "version": 2,
      "importContainerKey": "myProject-customer-container-learn",
      "resourceKey": "imported-customer-03",
      "id": "2a528594-4c01-48cb-9ef9-6cc3620978d1",
      "state": "unresolved",
      "resourceVersion": 5,
      "createdAt": "2023-12-07T14:28:25.111Z",
      "lastModifiedAt": "2023-12-07T14:29:05.487Z",
      "expiresAt": "2023-12-09T14:28:25.112Z"
    }
  ]
}

Processing states

In the result you should see that your Import Operations are either in the processing or imported state. More processing states are available that you should be aware of, like unresolved, as seen in the example.

What does unresolved mean and how can it be fixed? Let’s find out in the next example where we will create another Customer using the Import API that triggers the unresolved state:

importApiRoot
  .customers()
  .importContainers()
  .withImportContainerKeyValue(containerKey)
  .post(
    CustomerImportRequestBuilder
      .of()
      .resources(
        CustomerImportBuilder
          .of()
          .key("imported-customer-02")
          .firstName("Kate")
          .lastName("Holmes")
          .email("kholmes@example.com")
          .password("secret123")
          .customerGroup(
            CustomerGroupKeyReferenceBuilder
              .of()
              .key("imported-customers")
              .build()
          )
          .addresses(
            CustomerAddressBuilder
              .of()
              .key("imported-customer-02-address")
              .country("DE")
              .build()
          )
          .build()
      )
      .build()
  )
  .executeBlocking();

After running this code and waiting for the import request to be processed, run the script to get the Container status or check the Container status in the Merchant Center. You should see the status of unresolved with typeId: "customer-group" and key: "imported-customers".

This means that the Import Request contains key references to resources which do not exist. In this particular case the Project is missing a Customer Group with a key of imported-customers.

To rectify this, we need to create a Customer Group with the key imported-customers or the Import Operation will be deleted 48 hours after the time of its creation.

Checking the status of Import Containers

What if you would like to see an overall view of how many Import Operations (and in which state) your Import Container has? Well, the Import Summary does just that!

An Import Summary describes the status of an Import Container by the number of resources in each processing state. It can be used to monitor the progress of import per Import Container. You can use the following code to monitor the status of the Import Container:

OperationStates operationStates = importApiRoot
.importContainers()
.withImportContainerKeyValue(containerKey)
.importSummaries()
.get()
.executeBlocking()
.getBody()
.getStates();


System.out.println(
"Processing: " + operationStates.getProcessing() +
"\nValidation failed: " + operationStates.getValidationFailed() +
"\nUnresolved: " + operationStates.getUnresolved() +
"\nWaiting for MasterVariant: " + operationStates.getWaitForMasterVariant() +
"\nImported: " + operationStates.getImported() +
"\nRejected: " + operationStates.getRejected() +
);

Best practices

As you have learned in this module, the Import API is a very powerful and flexible tool for bulk adding and updating resources. However, to best use the Import API it is important to be aware of limitations and best practices.

To get the most out of the Import API it’s recommended that you familiarize yourself with the Import API best practices. Pay particular attention to:

How to organize your Import Containers so that you can monitor individual behavior and learn for adjusting later imports.
How to adjust the size of your import operations per container.